Re: Thoughts about storage

[Steve Hillman <hillman@sfu.ca>]

----- "Tom Golson" <tgolson@tamu.edu> wrote:


> We do deliver about 300,000 messages/day to the Zimbra complex, with
> around 6,500 concurrent mail readers (web and IMAP sessions) during
> peak usage.  iSCSI against the Sun storage appliances is certainly an
> option,

Actually, take it from us - it's not :)

The new Sun Storage appliances are based on a recent release of OpenSolaris that's been stripped right down, but unfortunately they haven't yet incorporated their yet-to-be-released kernel implementation of their iSCSI daemon - it's still a user-land daemon. As such, it.. well, sucks.  We tried to do iSCSI against our Thumpers running everything up to S10u6 and eventually gave up - it was quite unstable.  When I was at the LISA conference in November, I had a chance to chat with some of the Sun engineers and the user-land iscsi daemon was almost universally condemned. 

A new from-the-ground-up kernel implementation, called Comstar (Common SCSI Target) is almost ready and once that makes its way into the stable codebase, that would be the one to use.

The huge improvement you saw in NFS performance is due to improvements in the ZFS ZIL (ZFS Intent Log). Since NFS forces a commit for every file metadata operation, the ZIL gets beat on pretty hard and this really took its toll on NFS write performance. You can get around this by making use of what the ZFS folks term a "slog"  - separate log. By default, the ZIL is just scattered throughout your ZFS pool, but by using a dedicated device, you virtually eliminate seek delays because it's a write-only load. Ideally, that device would be high-write-speed flash, but it could also be your system disk, since the system disk is used so little - this from the ZFS guru at LISA.


At our site, we've put the Thumper to use as an HSM device using NFS. We've got our HSM period set to 4 months, so anything older gets moved off. People almost never read messages more than 4 months old, so the slower performance of NFS is not a big issue. The only impact we've noticed is for Mutt users hitting Zimbra with IMAP. Because Mutt wants to load the entire message list, and because the headers it asks for aren't stored in MySQL, every message must be read when Mutt fetches the Index. The fix for this (which we're currently just testing) is to use Mutt's "header_cache" feature to keep a local copy of headers. But the same would apply to any IMAP client the first time it downloads the headers of a folder.

By using a 4 month HSM retention, our experience has shown that about 70% of our users' data can be moved off to HSM, so we're running with 5gb quotas for everyone (yes, even students). On our NetApps, we're using SATA for the message store, giving us better storage density there. We've reserved the FCAL spindles for the index, db, and redolog volumes.


If you have the Briefcase feature enabled, be aware that Briefcase items aren't, by default, picked up by HSM and moved, so they'll gradually fill up your primary spool. We got around this by hacking the zimbra HSM module to move briefcase blobs instead of calendar blobs (the default is message, calendar, and task blobs)


For anyone who cares, we just completed our student migration over Christmas, taking us from about 12,000 to 42,000 accounts  (out of 50,000) moved over. So far, the SOAP session count has increased by less than 50% from pre-Christmas peaks, but it's only the 3rd day of classes :).  Most who've been migrated so far have elected to have all of their archived mail moved with them (we've had lots of staff switch from desktop POP clients to the web client), and with all of that data, our numbers look like this:

Total HSM usage: ~1.8TB
Total Primary spool usage: ~1.1 TB

This storage is spread pretty evenly across 4 mailbox servers (all of which mount LUNs from the NetApp and NFS volumes from the Thumper)

The Thumper we're using is a 48TB model and once formatted down and with 2 hot spares, gives us 32 usable TB of storage

I wouldn't recommend using the Thumpers for your primary storage pool. The two thumpers that we work fairly hard crash on a pretty regular basis - about once a month. I will shortly be rebuilding one of them with OpenSolaris 2008.11, which I'm hoping will improve the situation.  But by keeping them as HSM and NFS, even a crash goes pretty much unnoticed by the user community


-- 
Steve Hillman                                IT Architect
hillman@sfu.ca                               IT Infrastructure
778-782-3960                                 Simon Fraser University