Re: Expected performance of importing mail?

["Freddie Cash" <fjwcash@gmail.com>]

On Thu, May 1, 2008 at 11:31 PM, Steve Hillman <hillman@sfu.ca> wrote:
> Hi folks,
>   We have written some scripts to import users' mail from our legacy Webmail system and
> from homedir-based IMAP folders into Zimbra using their SOAP interface. I'm not getting
> stellar performance out of them, and I'm wondering what other sites have experienced in
> terms of throughput when importing mail.
>
>  The SOAP import mechanism is, as far as I know, the same mechanism that the Outlook
> migration wizard uses to get mail in, so I would expect to see similar performance numbers
> to it. On our servers, we're getting about 10-12 messages per second throughput, depending
> somewhat on the size of the mail (on large mailboxes, it all averages out anyway). The
> servers don't appear to be maxed out during the import - CPU is 10-20% and iowait is
> 10-30%. The /opt/zimbra volume, which is iSCSI-mounted from a NetApp, is seeing about
> 1-2MB/s in disk writes.

I would have to say there's either something wrong with the SOAP
interface, or something wrong with your setup.  Using imapsync running
on the Zimbra server, we were able to migrate 250+ MB mailboxes in
just over an hour.  (As this was just a test box, we didn't keep stats
on the migration, other than it took a couple hours for really large
accounts.  The largest was just under 2 GB and took just over 4
hours.)

The webmail server is a dual-Opteron 1.8 GHz box with 4 GB RAM and
SATA drives in RAID5.  The Zimbra box was a dual-P3 1.0 GHz box with 4
GB RAM and IDE drives in RAID5.  They were connected via 100 Mbps
ethernet, with a packet filtering NAT firewall in between.  The
webmail server is running Debian Linux 4.0 (Etch) while the Zimbra
server was running Ubuntu Server 6.06.1 LTS.

During the sync, 1 CPU on the Zimbra server would be pegged at 100%,
while the other was mostly idle.  Accessing the Zimbra server during
the transfer was slow, but still usable.

>  Are these numbers consistent with what other sites have seen?

I would say they are very low.

> Have you tried to optimize anything on the server to get better throughput than this?

Nope.  We just ran imapsync as a normal user and let the OS handle the rest.

> I'm not seeing a particular bottleneck on the server, so I'm wondering why it can't go faster
> (if I modify my client-side script to write to /dev/null instead of the http socket, I get about 60
> messages per second, so most of the latency is definitely coming from the Zimbra side)

Would it be possible to use IMAP for the transfer instead of the SOAP
interface?  Could you try using imapsync to see if things go faster?
That might help narrow down where the bottleneck is.

-- 
Freddie Cash
fjwcash@gmail.com