blog | news | projects | notebooks | politics | github

mutt

Mutt is a text-based email client. it is ideal if: - You want to be able to check your email from a terminal - You want a lightweight email client - You want to encrypt or digitally sign your mail

The configuration process for mutt and the email tools that are often used with it can be confusing and time-sucking. These are the tools I chose and the steps I took.

I receive all of my mail through a google account on a custom domain (dcalacci.net). All of my other mail is forwarded to that google accounts' inbox. I like to be able to send mail from any one of the address I own.

I keep local copies of all my mail by using offlineimap. Mutt reads my mail from local directories on the machine I'm using.

I send mail using msmtp.

Authentication for accessing and sending mail uses gpg-agent, a daemon that I use to manage my gpg-encrypted passwords.

The instructions here are directed at Arch Linux users, but much of it applies to any UNIX-based OS, including OSX.

Local Mail

I like to store copies of my mail locally. That way, I can read, respond, search, and analyze my mail without being connected to the internet.

Offlineimap

offlineimap is a python utility that syncs mail to and from IMAP servers. You can get the package from the community repo.

Configured properly and run at a regular interval, offlineimap will keep your local mail directory and your remote IMAP server in sync.

Configuration

The offlineimap configuration file is usually located at ~/.offlineimaprc.

[general]
ui         = Noninteractive.Quiet
accounts   = dcalacci.net
fsync      = False
pythonfile = ~/bin/offlineimap.py

This configuration specifies that offlineimap should use the Noninteractive.Quiet interface. It also declares a list of accounts; I only read mail from one account, so I only have one listed. Using fsync can sometimes make offlineimap much slower, so I disable it here.

The pythonfile is a file that offlineimap will parse into python code as it reads from the configuration file. This lets you use python functions in your configuration. I use it to help keep my passwords secure. I'll describe in detail what my offlineimap.py does in a later section.

[Repository gmail-local]
type = Maildir
localfolders = ~/.mail/dan-dcalacci.net
[Repository gmail-local]
type = Maildir
localfolders = ~/.mail/dan.calacci-gmail.com
nametrans = lambda foldername:
                           re.sub ('inbox',     '[Gmail].Inbox',
                           re.sub ('sent',      '[Gmail].Sent Mail',
                           re.sub ('drafts',    '[Gmail].Drafts',
                           re.sub ('flagged',   '[Gmail].Starred',
                           re.sub ('important', '[Gmail].Important',
                           re.sub ('archive',   '[Gmail].All Mail',
                           re.sub ('spam',      '[Gmail].Spam',
                           re.sub ('trash',     '[Gmail].Trash',
                           re.sub ('-',         ' ',
                           foldername)))))))))

This specifies the configuration settings for a local mail repository, which means that it's the repository that handles all the mail that we store locally on disk. This specific repository is named gmail-local. You can choose different types of mail repositories to store your local mail in. I chose the maildir type maildir works well with many email clients and search utilities. It's also easy to manually navigate. This way, if I change mail clients in the future, I can keep my local mail copies and not have to migrate my mail repository to anything new. Knowing details about the structure of maildir isn't necessary to set up local mail.

I store my mail for all accounts in the folder ~/.mail. I also specify subfolders, where the @ is replaced by a -, for each individual account, like this:

~/.mail
|--dan.calacci-gmail.com/
|--dcalacci-ccs.neu/

The nametrans specification is the most complicated piece of storing mail offline, and is highly dependent on the configuration of your remote repository, so I will address it after introducing the remote configuration.

[Repository gmail-remote]
maxconnections = 1
type = Gmail
remoteuser = dan@dcalacci.net
realdelete = no
nametrans = lambda foldername: re.sub ('^\[gmail\].', '',
                               re.sub ('sent-mail', 'sent',
                               re.sub ('all-mail', 'archive',
                               re.sub ('starred', 'flagged',
                               re.sub (' ', '-',
                               foldername.lower())))))
folderfilter = lambda folder: not re.search('(Mailbox)', folder)
remotepasseval = mailpasswd("gmail")
sslcacertfile = /etc/ssl/certs/ca-certificates.crt

This repository is a remote repository; it's the configuration for the remote IMAP server. I limit the number of connections to 1, since I should be the only one accessing it, from one machine, and I specified the type as Gmail: offlineimap actually has some built-in settings that compensate for Gmail's unusual behavior with IMAP. I also specify my username, using remoteuser.

I don't delete any of my mail. I just archive it. Setting realdelete = no ensures that when I delete mail from a folder, it doesn't actually get deleted (i.e., it doesn't get moved to the trash folder), so it will stay in All_Mail. If a message is deleted from the All Mail and Trash folders, however, that message will be deleted.

sslcacertfile specifies the ssl certificate file to use with the IMAP server. Gmail requires this, and arch by default uses /etc/ssl/certs/ca-certificates.crt. If you wish to use a different certificate, simply specify it here.

Nametrans

On a normal IMAP server, a user has folders, like inbox, sent, or any custom folders they have created. Usually a message exists only in one folder. If you move a message from inbox to the custom folder travel, the message is moved from inbox, and placed in travel.

Gmail IMAP actions work a little differently than normal IMAP, because Gmail has a notion of 'labels'. Each label in Gmail has a corresponding IMAP folder, and each message may have multiple labels. This means that for Gmail, each message may exist in several different IMAP folders. If a message has the labels work and github, and is in the users' inbox, then copies of that message will exist in the imap folders for work, github, and inbox.

Gmail also has specific naming conventions for it's IMAP folders. Any Gmail-specific folders, such as Starred or Sent-Mail, have the prefix [Gmail]/. Sent-Mail's folder is actually [Gmail]/Sent-Mail.

In the remote repository configuration, the nametrans directive specifies a python function that maps folder names on the IMAP server to local folder names. My nametrans section defines a lambda function that turns the input name into all-lowercase, and then runs a series of substitutions, in order:

nametrans is in the local repository configuration too, since offlineimap needs to know how to convert your local folder names back into IMAP (remote) ones.

Here's both of my nametrans settings next to each other so you can examine how the translation works:

local:

nametrans = lambda foldername:
    re.sub ('sent',      '[Gmail].Sent Mail',
    re.sub ('drafts',    '[Gmail].Drafts',
    re.sub ('flagged',   '[Gmail].Starred',
    re.sub ('important', '[Gmail].Important',
    re.sub ('archive',   '[Gmail].All Mail',
    re.sub ('spam',      '[Gmail].Spam',
    re.sub ('trash',     '[Gmail].Trash',
    re.sub ('inbox',     'INBOX',
    re.sub ('-',         ' ',
    foldername)))))))))

remote:

nametrans = lambda foldername:
    re.sub ('^\[gmail\].', '',
    re.sub ('sent-mail',   'sent',
    re.sub ('all-mail',    'archive',
    re.sub ('starred',     'flagged',
    re.sub (' ',           '-',
    foldername.lower())))))

All of my labels in gmail are lowercase, so they are not actually affected by either nametrans function. Labels that use uppercase letters will not work by default using the above configuration - additional substitution rules will need to be added.

Passwords

In my offlineimap configuration, I have a line:

pythonfile = ~/bin/offlineimap.py

This line enables you to use a python file in your offlineimap configuration. Anything functions or variables defined in that file may be used in your .offlineimaprc.

The file I specified contains the following python function:

def mailpasswd(acct):
  cmd = '/Users/dan/bin/getpwd '+acct
  try:
    return subprocess.check_output([ 'bash /Users/dan/bin/getpwd',
                                     acct ]).strip()
  except subprocess.CalledProcessError:
      return ""

It defines a function, mailpasswd, that returns the password of the mail account name specified by acct. It calls an external script, getpwd, with acct as a parameter.

getpwd looks like this:

gpg --batch -qd $HOME/.passwd/mail-pwds.gpg | grep $1 | sed "s/${1}: \(.*\)/\1/g"

I have a gpg-encrypted file, ~/.passwd/mail-pwds.gpg that stores my mail account passwords. It looks like this:

account: password1
account2: password2
account3: password3

getpwd uses gpg to decrypt the mail-pwds.gpg file, and then uses sed to parse out the password for a given account. If you have gpg-tools or gpg-agent, this script should never prompt you for a password.

I keep getpwd and the mailpasswd function separate because I have other utilities, such as msmtp (introduced later) that use the getpwd script.

After specifying the offlineimap.py script in the offlineimap configuration, you can set offlineimap to use the mailpasswd function to retreive your passwords by placing this line:

remotepasseval = mailpasswd("gmail")

in any remote repository configurations. Here, I use "gmail" as the account name, because one of my accounts defined in my ~/.passwd/mail-pwds.gpg file has that name. Be sure to change the account name you pass in accordingly.

Sending Mail

I use msmtp to send mail. You can find it on the aur, or install it on OSX with brew install msmtp. The configuration file is located at ~/.msmtprc

I read mail from only one address, but send mail from several, so I have three smtp accounts defined in my msmtp configuration file. I'll walk through the configuration section for my gmail account - any other smtp accounts you'd like to define here should be very similar:

account         gmail
host            smtp.gmail.com
port            587
protocol        smtp
auth            on
from            dan@dcalacci.net
user            dan@dcalacci.net
passwordeval    /Users/dan/bin/getpwd gmail
tls             on
tls_starttls    on
tls_trust_file ~/.mutt/Equifax_Secure_CA.cert

You can find the host and port definitions from your mail provider.

We're using smtp, and we want authentication on.

The mail address for gmail is the same as the username.

I use the same password script, getpwd, to retreive the account password.

I set tls on, and the other tls directives are necessary to ensure that a secure connection is reached. In Arch Linux, I was able to set

tls_trust_file /etc/ssl/certs/ca-certificates.crt

and have it work properly. On OSX, I had to download the certificate manually and store it on my machine. This is the certificate file I use.

Mutt

First, install mutt. You can do this from the AUR (I recommend mutt-great-dane), or, if you have OSX, you can use homebrew:

brew install mutt --with-sidebar-patch --with-pgp-verbose-mime-patch

The sidbar patch will give you a sidebar to explore your mailboxes with, and the pgp verbose mime patch will allow you to rename MIME PGP signatures.

Mutt is incredibly customizable. There are dozens of patches that increase it's functionality, and there are hundreds of options you can set in your configuration file.

It is very difficult to start using mutt from scratch without an already-working configuration. I suggest you use an existing one, and examine the documentation as you get used to using it.

Much of my configuration is taken from Steve Losh's excellent guide. As such, my configuration mimics his.

If you want to be able to PGP-sign and/or encrypt your mail, read on. If this is of no interest to you, I recommend you take Steve Loshs' configuration and roll with it.

If you want to be able to send mail from multiple accounts, you have to have these settings in your mutt config:

set use_from             = yes
set envelope_from        = yes
set sendmail             = "msmtp"
set sendmail_wait        = 0 # wait until it's actually sent

These settings use the From: field in the message to determine what account to send mail from. This is sometimes a pain, since every time you compose mail, you have to enter in the account you want to send mail from:

specifying the account to send from

You can, however, make some macros to make setting the account a little easier:

:set from='dan@dcalacci.net' realname='Dan Calacci'"
:set from='dcalacci@zimbra.ccs.neu.edu' realname='Dan Calacci'"

Just replace the accounts and real names I defined up there with your own, make sure msmtp is configured to send mail from the accounts you set, and you should be ready to go.

PGP with Mutt

I source the following file in my mutt configuration. It specifies lots of pgp-related settings that "just work" if you have gpg installed.

set pgp_decode_command="gpg %?p?--passphrase-fd 0? --no-verbose --batch --output - %f"
set pgp_verify_command="gpg --no-verbose --batch --output - --verify %s %f"
set pgp_decrypt_command="gpg --passphrase-fd 0 --no-verbose --batch --output - %f"
set pgp_sign_command="gpg --no-verbose --batch --output - --passphrase-fd 0 --armor --detach-sign --textmode %?a?-u %a? %f"
set pgp_clearsign_command="gpg --no-verbose --batch --output - --passphrase-fd 0 --armor --textmode --clearsign %?a?-u %a? %f"
set pgp_encrypt_only_command="pgpewrap gpg --batch --quiet --no-verbose --output - --encrypt --textmode --armor --always-trust -- -r %r -- %f"
set pgp_encrypt_sign_command="pgpewrap gpg --passphrase-fd 0 --batch --quiet --no-verbose --textmode --output - --encrypt --sign %?a?-u %a? --armor --always-trust -- -r %r -- %f"
set pgp_import_command="gpg --no-verbose --import -v %f"
set pgp_export_command="gpg --no-verbose --export --armor %r"
set pgp_verify_key_command="gpg --no-verbose --batch --fingerprint --check-sigs %r"
set pgp_list_pubring_command="gpg --no-verbose --batch --with-colons --list-keys %r"
set pgp_list_secring_command="gpg --no-verbose --batch --with-colons --list-secret-keys %r"

# specify the uid to use when encrypting/signing
set pgp_sign_as=0xYOUR_UID

# this set the number of seconds to keep in memory the passpharse used to encrypt/sign
# the more the less secure it will be
set pgp_timeout=60

# it's a regexp used against the GPG output: if it matches some line of the output
# then mutt considers the message a good signed one (ignoring the GPG exit code)
set pgp_good_sign="^gpg: Good signature from"

# mutt uses by default PGP/GPG to sign/encrypt messages
# if you want to use S-mime instead set the smime_is_default variable to yes

# automatically sign all outgoing messages
set crypt_autosign
# sign only replies to signed messages
set crypt_replysign

# automatically encrypt outgoing messages
set crypt_autoencrypt=yes
# encrypt only replies to signed messages
set crypt_replyencrypt=yes
# encrypt and sign replies to encrypted messages
set crypt_replysignencrypted=yes

# automatically verify the sign of a message when opened
set crypt_verify_sig=yes

To make this work properly for you, you only have to set the pgp_sign_as directive to your gpg UID. To do this list your secret keys by running gpg --list-secret-keys, and choose the secret key you would like to encrypt your mail with. GPG should list a sec, uid, and ssb for each secret key. Under sec, there should be a hex identifier for the key:

sec  2048R/XXXXXXXX 2011-08-04

The 0xXXXXXXXX is the identifier. set pgp_sign_as to 0xXXXXXXXX.

Comment and uncomment other lines (set crypt_autosign, set crypt_autoencrypt=yes) according to your preference.

Now, when you attempt to send a message, after composing, you can press p to open the pgp menu:

signing a message

It will prompt you for an action - you can sign/encrypt your mail from here, or cancel encryption/signing if you have it turned on by default.

Using mutt and descriptions of all its' features is beyond the scope of this document. By now, you should have be able to read mail, and send PGP-encrypted email using mutt.


About Me

I'm interested in building technological platforms that leverage what we know about social dynamics to help people live their lives better.

I'm currently working at the Human Dynamics Group at the MIT Media Lab, creating systems that attempt to measure and impact human social and health behaviors.

I've also worked with the Lazer Lab, inferring partisan dynamics from congressional public statements.

You can e-mail me at dan@dcalacci.net

Send me encrypted messages using my PGP key. (via keybase)

Resume here.

see what music I listen to