Transferring Email to Gmail

michael Minor Distraction

For the past decade or more I’ve been using Dreamhost to host some websites and some email accounts.  Both for myself and for a couple other people.  Dreamhost is an okay service provider and my point here isn’t to discredit them.  I have a $20 a month account there for a long time.  But there are several reasons why I need to move on.  For one, and to the point of this post, I’m ready to try Gmail out for my mail business account.  For my account that I’m moving I use POP mail and store my emails on my desktop machine.  But some of the other accounts that I’m moving the user was IMAP and hosted email on the server.  Some with 10’s of 1000’s of emails in various mailboxes.  Those users want to continue to have access to all their old emails.  So how do you easily move email accounts from Dreamhost (or another IMAP server) to Gmail?

Hidden in a discussion forum thread from 2008 was a method that used a linux utility called “mailutil“.  This isn’t to be confused with mailutils (with an ‘s’) which is a completely different utility.  The thread makes it sound like this utility will move the mail.  The problem is that the utility mailutil isn’t to be found.  It’s not in the dreamhost usr/bin folder and it’s not on my Mac.  But after hours of research looking for an easy (cheap) way to do this I decided that mailutil was the best sounding method.  And then it took several more hours to find and build the utility to run under Mac OS X 10.10 (Yosemite).  I’m no linux expert but I made it through and I have the utility working.  If you want to do it, here’s how I did it…

tl;dr

I can be a little long winded in my writing.  This post is a detail oriented account of what I did and how I did it.  So I’ll give the summary first for those with no patience for details.  The first section is about building the utility.  If you are already on a mac 10.10 save some time and download the utility here. After that the short version of how to do it is on that thread I linked to above.  The broad strokes is that you’ll use transfer to copy everything over in one command. If that stops for some reason you’ll have a half finished job.  The transfer mode seems to go box by box, oldest to newest.  So then you’ll finish the box that it died on using append.  And then the rest of the boxes with copy.

Building mailutil

First, the utility is hosted as part of some IMAP toolkit utilities at the University of Washington.  You can find a page about the toolkit here : http://www.washington.edu/imap/

The software is hosted on an ftp site here: ftp://ftp.cac.washington.edu/imap  Grab the zip file called: map-2007f.tar.Z

Next I created a folder inside my user folder called “bin” and I copied the file there.  Then double click it to unzip / untar it.

The rest will be Terminal heavy.  You’re going to eventually need to add this folder to your $PATH.  Read about that here: http://coolestguidesontheplanet.com/add-shell-path-osx/

You’re also going to need the Xcode command line utilities.  In terminal do

sudo xcode-select --install

I found about that on stack overflow here.

Then ls to inside the unzipped imap-2007f folder and make the utilities:

cd ~/bin/imap-2007f
make osx

I fought with this step for hours.  Mostly I was getting errors with missing include files.  I modified the Makefile, tried different system type specifiers, and googled extensively.  The trick that got me past these errors was to install the Xcode command line stuff.  Once I did that all the errors went away.  The other thing was to figure out which “system type” to use.  If you do ‘make’ instead of ‘make os x’ it will give you an error message about needing to specify the  system type.  Once you have the Xcode stuff you can use ‘os x’ as the system type and it should work.

After a successful build I copied the mailutil executable to the root level of my /~/bin folder so it can be executed easily.

Setting up

The mailutil utility is a command line executable.  You’ll need to stay in a terminal window to run it.  But from terminal on your own mac you can move the mail from Dreamhost (or probably whatever your email server is) to the Gmail server.  But first you will need to set up your email on gmail, and do a couple of things.  Here’s what the post on the thread said:

migrating email to Gmail 

1. Setup your gmail account — either regular <username>@gmail.com or setup Google apps for your domain, so you get <username>@<yourdomain>.com2. Using “Settings” in gmail interface, ENABLE imap3. Then, from a unix command line at dreamhost, do the following:

Code:
mailutil transfer -merge append '{a1.balanced.<your-dreamhost-mail-server>.mail.dreamhost.com/imap/ssl/novalidate-cert/user=<your-dreamhost-username>}' '{imap.gmail.com}/ssl/user=<your-gmail-username>}'

Its now going to prompt you twice for password. The first password is for your dreamhost account. Second is for your gmail account.

Note: 1. Look up “man mailutil” in google to see what it does.
2. Depending on the size of your mailbox, it can take a long time. On my approx 5000 email mailbox, it took about 2 hours
3. The IMAP folders from your dreamhost account will end up becoming “labels” in gmail interface
4. You may have to manually move messages between “Sent Items” and “Sent Mail”

Some additional notes on setting up:

Step one is to set up your gmail account.  One thing I will say is that in that process you might want to make sure you can still access both your old and new email accounts.  They might have the same email address.  But only one of the servers is directed to receive the mail.  On Dreamhost if you’re using their DNS you can basically set it up with a single click to use gmail instead of their hosted mail.  But if you do that you won’t be able to access your old email via webmail.yourDomain.com.  So what I had to do was set it to still use Dreamhost for mail.  Then I can access both accounts at the same time, one using mail.google.com and the other using webmail.yourDomain.com.

Step two is to turn on IMAP within the gmail settings.  Use the gear icon and hit up ‘Settings’ then go to the ‘Forwarding and POP/IMAP’ tab.  Half way down is a button to enable IMAP.  You’ll need to do this on every account that you’re migrating.

gmail Settings

IMAP Enabled

My step three is to make sure you’ve got the passwords set up on both accounts so that it’s going to be easy to enter them.  Because you’ll be entering them a lot.  It’s easiest if the passwords are the same on both accounts but that might not always be possible.

Then their step three is to use the utility to do the migration.  Let’s dive in to how to do that.

Gotcha’s

Before diving it let me give you a few notes and gothcha’s…

First, it’s not super fast.  It seems to take about 1 second per email.  That’s a very rough average.  But at that rate 10k emails would take 2-3 hours.  Pick out a good show on Netflix to binge watch while you’re doing this.  Cause you will need to babysit it.

Oh and babysitting.. it is probably not going to transfer smoothly in just one terminal command.  The mailutil tool has a transfer mode that is supposed to do all mailboxes recursively.  If it doesn’t run in to a problem it chugs along really nicely.  The problem is that sometimes it will just stop and lose the connection.  When that happens, rerunning the tool won’t just pick up where it left off.  Instead it will start over doing a new merge append.  This is ok in that it won’t duplicate messages that are already in there.  But not ok in that it is time consuming to re-move emails.  And likely it will stop again at the same spot the second time.

[[CLOSED] IMAP connection broken (server response)] [CLOSED] IMAP connection broken (server response)
[[CLOSED] IMAP connection lost] [CLOSED] IMAP connection lost
IMAP protocol error: Invalid Arguments: Unable to parse message
Invalid Arguments: Unable to parse message
[Failed] [LOGOUT Requested]

From what I can tell it’s stopping when it hits emails that have large attachments.  Sometimes it’s stopping after moving the message and sometimes it’s stopping before moving it.  I haven’t been able to nail down an exact attachment size limit.  It seems to be able to do messages with less than 20 megs worth of attachments.  But this can be a pain.  One mailbox had about 10,000 emails and it kept stopping somewhere around the 9000th email and it couldn’t get past it.  In that case you’re not going to want to blow 2-3 more hours just to find out that it stops again in the same place (like I did).  You’ll want to pull the remaining 1000 emails off to another mailbox and them move them with a different command.  Transfer seems to go oldest to newest in each box, so you should easily be able to find what messages in that box have transferred and which ones haven’t.

I should also add here that in moving 3 mailboxes (so far) I’ve only come across 2 or 3 messages that choked it up.  But it only takes one to halt a transfer and force you in to an incremental mode.

You can try transfer first and see if it works.  But the transfer mode can’t pick up where it left off.  So if it dies then you’ll have to switch to move messages mailbox by mailbox and use the copy and append modes instead.  And those modes are not recursive to sub-mailboxes.

One hint to try to make the transfer mode work without stopping is to first delete any messages that have more than 20 megs of attachments.  Those messages probably won’t go across with this utility anyway.  So you could go through each mailbox, sort by size and then move all the emails larger than 20 megs to a new mailbox.  Back that mailbox up to your Mac and then delete the messages on the IMAP server.  Then do transfer and cross your fingers that it will make it all the way through.

If you do end up having to do incremental moves or if you decide from the beginning to do this, you still might want to check and remove large emails first.  It can be real pain to move 1000’s of emails to another box.  Especially in a web mail app like squirrel mail.  In squirrel mail to split a box that has 10k messages in it you’d have to click the check boxes one by one on the messages you have to move.  And that goes against the whole point of using a utility which is to automate the process.

Next, to make incremental migration easier, in your source server you’ll want to create some new mailboxes to handle organization and as holding spots when you need to do that last 1000 emails after a crash.  I created ones called MOVED, PROBLEMS, and REMAINING.  It doesn’t matter what they’re called but you’ll want some extra empty boxes to segregate messages in after you’ve moved them or when you have 1000 left to move.  The PROBLEMS box is where I put messages that had attachments that were too large.

And last, open a text editor.  Don’t type these commands in fresh each time.  Stage them in a text editor like TextMate.  Then you can keep different version of the command around, make changes and copy paste the command to the terminal.  In mine I even write notes between the commands on what they do.  And then I save the document so I can get back to it later if I need to do this again.  By the way TextExit (the mac notepad app) actually doesn’t work too well for this.  If you’re not paying attention it can swap our your single quote marks with the angled variety.  That angled quote mark will make the command error out.  So it’s better to use something like TextMate.

Using mailutil

The man pages (manual) for mailutil can be found here.  The basic format as described on the original discussion was this:

mailutil transfer -merge append '{a1.balanced.<your-dreamhost-mail-server>.mail.dreamhost.com/imap/ssl/novalidate-cert/user=<your-dreamhost-username>}' '{imap.gmail.com/ssl/user=<your-gmail-username>}'

Source and Destination Formatting:

I’ll start with how to format the source and destination.  If you are migrating from dreamhost you’ll need your mail server name.  This page tells you how to find yours.  If you are coming from some other mail client I can’t help you.  I haven’t done it.  Possibly you’d just use your mail.domainName.com.  Since I am on dreamhost mine was homie.  I also removed the a1.balanced although I think it was working either way.

Again, the single quote marks have to be the straight kind.  Then angled ones won’t work.

The curly brackets contain the server name and username.  It’s going to be something like username@servername.com.

For the copy and append commands (not transfer) when you do incremental moves can specify a mailbox.  You do this outside the curly brackets but still inside the quote marks.  You’ll see examples below.  On dreamhost the source side format is usually something like “INBOX.old-messages”.  On the gmail side you can specify it like “INBOX/old-messages” or you can rename it to anything you want.  For example you could copy all the messages from INBOX.old-messages to something more readable like “Old Messages”.  But please note that dreamhost uses the dot and gmail uses the slash.  If you get that switched you’ll get error messages.

Also note that on Dreamhost, at least in all our accounts, all the boxes were children of the INBOX.  When I did mailbox by mailbox incremental copies I renamed these on the move to make them more attractive.

-ignore -v

You’re going to want to add –ignore and -v (for verbose) to the commands.  Otherwise you’ll be staring at a cursor for hours with no idea what’s going on.  The -v option will give you feedback as to what’s happening.  And it feels good to see success and ok as it does stuff.

The -ignore will ignore and continue when it hits some errors.  For example during a transfer if it can’t create a mailbox because it’s already there it will continue on and copy the messages to that box.  Otherwise, without the -ignore you can’t even use the transfer command twice.  It’ll stop right away cause the mailbox can’t be created.  I use these two options every time now.

Sample Commands:

Each time you do the command it’s going to prompt you for two passwords.  The first one is the source (or dreamhost) password.  The second time is the gmail password.

The transfer command will try to move the entire account over including all mailboxes.  You can’t specify any mailboxes when using transfer because it has to start at the root level.  The basic transfer command for me was this:

mailutil transfer -ignore -v -merge append '{homie.mail.dreamhost.com/imap/ssl/novalidate-cert/user=name@domain.com}' '{imap.gmail.com/ssl/user=name@domain.com}'

To copy a single mailbox to a new destination use copy. The destination mailbox can’t already exist or it will fail.  I keep the -ignore in there even though it will fail if it can’t create that mailbox.  And notice on this one the source mailbox is INBOX.Test but on the destination I just put it in Test:

mailutil copy -ignore -v '{homie.mail.dreamhost.com/imap/ssl/novalidate-cert/user=name@domain.com}INBOX.Test' '{imap.gmail.com/ssl/user=name@domain.com}Test'

If after a transfer failure or after a failure during a copy, or to merge a second source mailbox to destination that already exists use append.  This will merge to a single mailbox.  So if the messages already exist in that box they won’t be duplicated.

mailutil append -ignore -v '{homie.mail.dreamhost.com/imap/ssl/novalidate-cert/user=user@domain.com}INBOX.old-messages' '{imap.gmail.com/ssl/user=user@domain.com}INBOX/old-messages'

Notice in the above case that I did INBOX.old-messages for the source and INBOX/old-messages for the gmail destination.  (Dot vs Slash)  The transfer command had already made this destination mailbox and this was what it called it.  To get these all in the same destination box I used the same name it made.

The big difference between the copy and append commands is that copy creates the destination mailbox which is therefore empty to start, and append copies to an existing mailbox merging the contents.  So the first time you’re migrating a particular mailbox you will use copy.  If you add to it (from a different source mailbox) or it fails and you retry it, use append.

To migrate the messages in your inbox you can leave the mailbox name off.  And since the default inbox always already exists you would use append. You’d do that like this:

mailutil append -ignore -v '{homie.mail.dreamhost.com/imap/ssl/novalidate-cert/user=user@domain.com}' '{imap.gmail.com/ssl/user=user@domain.com}'

Other Methods

While mailutil transfers mail from one mail server to another, I also wanted to archive all of my old emails from my desktop to gmail.  I was using mailutil more for other people’s accounts, where they primarily do IMAP and keep their emails on the server.  I haven’t been doing it that way,  I generally have done POP3 and therefore house my archive mail on my local machine.  But I’m considering changing that.

Google Email Uploader

Google does have a utility that can be used to upload messages from your Mac Mail database to the Gmail servers.  It’s called the “Google Email Uploader” and it can be found here.  A support guy at google told me about it and also mentioned that it was depreciated.  So I don’t know how long that app will be around.

gmailUploader

I tried using it and it is very problematic.  For one it showed that I had 148,000 emails but I really only have about 35,000 in archive now.  Probably that means that some of those accounts have overlapping items.  For example I haven’t used speakeasy.net in probably 8 years.  It might also mean that I have a mail database on my mac that it’s currently loaded in my Mail app.  It was also showing that it would take about 2 days to upload all that mail, which I suppose is irrelevant since I probably wouldn’t copy all 148k messages up.

When I tried moving a small mailbox up things got weird.  If you drill down in those accounts inside you just see numbered boxes.  When uploading them to gmail I ended up having all these numbered labels and the new messages weren’t in a box that I expected them to go in to.  Also stopping it mid-upload will result in the messages continuing to show up in your gmail box for about an hour.  I eventually gave up on this tool.  I’ll probably just try dragging messages to gmail from inside Mail.

IMAP Upload

A solution that did work for me to move my locally stored email to Gmail is IMAP Upload, which can be found here (http://imap-upload.sourceforge.net).  It’s a python app.  The docs are a little thin but I figured it out.  The way I did it was to export batches of email out of Mac Mail which creates an MBOX file.  For me exporting 35k emails took about 2 min.  Then run the python app and specify gmail and a box/label on your gmail account.  You DO have to create the box on your gmail account first.  And I suggest testing it with a small mbox with like 5 email in it to see what’s going to happen.  Here’s the command I used:

python imap_upload.py --gmail --box='Pre-Gmail Archive' --error Fails.mbox /Users/michael/bin/imap-upload-1.2/Test.mbox/mbox

When you enter that it will prompt you for the email address and password.  Note that it takes about 1 second per email, so it could take a long time.  But at least it’s verbose so you can see that something is happening.  Also important to know is that I couldn’t tell if it was a true merge or not.  On my test of 5 emails if I repeated it one of them duplicated and the others did not.  And last be sure to use the –error option and specify an mbox file to put failed messages in.

When I did the full 35k emails it took about 13-15 hours.  But it was able to run smoothly with no babysitting.  It also automatically reconnects itself to gmail when it loses the connection.  And at the end you can see how many emails failed to transfer. And if you look through the log you and search for ‘NG (‘ you can even find which emails failed.

upload results

 

In my case only 12 messages failed to upload.  One because the attachment was too large for gmail:

27208/34920 36.5 MB photoshop 2 from rob NG (APPEND command error: BAD ['[ALERT] Message too large. http://support.google.com/mail/bin/answer.py?answer=8770'])

The rest were all in the same thread and had a socket error:

24855/34920 6.0 kB Your Apple Store Proposal (Reconnect) NG (command: APPEND => socket error: EOF)

Conclusion

The mailutil utility worked well but it did require some babysitting.  The IMAP Upload utility worked even better because it didn’t require any babysitting, but it is only for uploading from a local mbox mailbox.  Overall I spent about 2-3 days on this distraction.  But that also includes the time it took to set up the gmail accounts.