Re: Thoughts about storage

[Tom Golson <tgolson@tamu.edu>]

Congratulations on your migration, Steve!


Steve Hillman wrote:
> 
> The new Sun Storage appliances are based on a recent release of OpenSolaris that's been stripped right down, but unfortunately they haven't yet incorporated their yet-to-be-released kernel implementation of their iSCSI daemon - it's still a user-land daemon. As such, it.. well, sucks.  We tried to do iSCSI against our Thumpers running everything up to S10u6 and eventually gave up - it was quite unstable.  When I was at the LISA conference in November, I had a chance to chat with some of the Sun engineers and the user-land iscsi daemon was almost universally condemned. 
> 
> A new from-the-ground-up kernel implementation, called Comstar (Common SCSI Target) is almost ready and once that makes its way into the stable codebase, that would be the one to use.
> 
> The huge improvement you saw in NFS performance is due to improvements in the ZFS ZIL (ZFS Intent Log). Since NFS forces a commit for every file metadata operation, the ZIL gets beat on pretty hard and this really took its toll on NFS write performance. You can get around this by making use of what the ZFS folks term a "slog"  - separate log. By default, the ZIL is just scattered throughout your ZFS pool, but by using a dedicated device, you virtually eliminate seek delays because it's a write-only load. Ideally, that device would be high-write-speed flash, but it could also be your system disk, since the system disk is used so little - this from the ZFS guru at LISA.
>

Actually, one of the many tuning efforts I made was to disable the ZIL.
 Even then, Solaris 10 x86 was inexcusably slow, even on the latest
2008/10 release.  The version of OpenSolaris we've installed is snv_101b
-- the 2008/11 release plus some updates.  With the ZIL enabled, we see
NFS performance comparable to that of our filers.  So I suspect it's
more than just ZIL improvements.

On a second thumper we're doing some virtualization development, and
under Solaris 10, the iSCSI stack was effectively inoperable.  With the
same build 101b of OpenSolaris as we're using for our backup device,
we're seeing very respecable iSCSI performance, now.  Considering it's a
bunch of 7200 rpm SATA drives, I really didn't expect them to do any
kind of meaningful I/O.  I've been pleasantly surprised.

The unified storage appliances (on the higher end) use flash drives for
the ZIL, and then drain writes off to attached SATA drives.  On the
7410, you can configure up to 12, 18 GB flash drives for ZIL writes and
6, 100 GB flash drives for read-caching, per controller head.
Consequently, I would expect much better performance.

Completely unrelated, Comstar does look promising if/when available.

> 
> If you have the Briefcase feature enabled, be aware that Briefcase items aren't, by default, picked up by HSM and moved, so they'll gradually fill up your primary spool. We got around this by hacking the zimbra HSM module to move briefcase blobs instead of calendar blobs (the default is message, calendar, and task blobs)
>

Interesting, since that is one of the options I'm looking at.  I'll
cruise the wiki for information on the HSM/Briefcase hack.

> 
> For anyone who cares, we just completed our student migration over Christmas, taking us from about 12,000 to 42,000 accounts  (out of 50,000) moved over. So far, the SOAP session count has increased by less than 50% from pre-Christmas peaks, but it's only the 3rd day of classes :).  Most who've been migrated so far have elected to have all of their archived mail moved with them (we've had lots of staff switch from desktop POP clients to the web client), and with all of that data, our numbers look like this:
> 
> Total HSM usage: ~1.8TB
> Total Primary spool usage: ~1.1 TB
> 
> This storage is spread pretty evenly across 4 mailbox servers (all of which mount LUNs from the NetApp and NFS volumes from the Thumper)
> 
> The Thumper we're using is a 48TB model and once formatted down and with 2 hot spares, gives us 32 usable TB of storage
> 
> I wouldn't recommend using the Thumpers for your primary storage pool. The two thumpers that we work fairly hard crash on a pretty regular basis - about once a month. I will shortly be rebuilding one of them with OpenSolaris 2008.11, which I'm hoping will improve the situation.  But by keeping them as HSM and NFS, even a crash goes pretty much unnoticed by the user community
> 
>