Re: Do you do HSM?

[Steve Hillman <hillman@sfu.ca>]

We've been using HSM since day one. Comments in-line

----- "Xueshan Feng" <sfeng@stanford.edu> wrote:

> I am interested to know the real world experience of Zimbra HSM
> deployment. I found some archived articles on this mailing list from
> 2008 about using HSM, but things have changed so much since and I
> believe there are must be more customers actively using HSM now. 
> 
> There is a good wiki page http://wiki.zimbra.com/wiki/Ajcody-HSM-Notes
> that covers various implementation considerations and setup, but I'd
> like to know anyone who uses HSM in their production environment has
> lessons, concerns and 'howtos' to share. Here are my specific
> questions (with some answers for our university, when applicable). All
> in all, does it worth to trade potential system performance impact
> with cheaper storage? 
> 
> 1. How many user accounts per mailbox server

4 mailbox servers, about 14000 accounts one ach
> 2. How much mailbox quota per account 

5 GB quota, fac/staff can request an increase


> 3. Storage architecture: do you use different storage for zimbra root
> /opt/zibmra, and zimbra backup /opt/zimbra? 

Each MB server has the following iSCSI LUNs, all run back to a pair of clustered NetApp heads

/opt/zimbra (includes /opt/zimbra/db) (FC disks)
/opt/zimbra/index (FC disks)
/opt/zimbra/redolog (FC disks, but different shelf)
/opt/zimbra/store (SATA disks)

We do NOT do Zimbra backups (gave up on them as not scalable a long time ago). Instead, we do NetApp-side snapshots and replicate them to a remote NetApp

> If you use HSM, what kind of secondary storage do you use? Do you use
> NFS mounted file system for secondary volume?   

Each MB server has a /hsm volume that is mounted via NFS from a Sun Thumper with a ZFS-based 40TB array
 
> 4. Do you have dedicated network connection  to the HSM storage?
> 

No, but each MB server has two nets - one "up" for mail traffic and one "down" for NFS/iSCSI traffic


> 
> Will it be a performance problem if both backup and the secondary HSM
> volume share the same network connection?

Hard to say. We're talking about millions of small files here, so the raw bandwidth may not be the limiting factor.


> 
> 5. Impact with backups

Does not apply for us. Both the NetApp and ZFS support snapshots, so that's how everything is backed up. 

> 
> 6. Impact to server
> 
> Obviously moving data from primary disk to secondary disk will have
> impact to server performance. Do you let your HSM process run
> continuously? Is the performance impact end user email experience
> during the business hours? 

HSM is kicked off nightly by a cron job on each MB server, staggered to help reduce load on the HSM server. It typically takes 2-3 hours on each server to complete and moves a total of 50,000-75,000 blobs each

> 
> 7. What HSM Age do you use? If you adjusted it from the default 30
> days, why? 

120 days. The assumption going in was that our NFS storage was unstable (which has proven to be true!) so if it went away temporarily, we wanted the impact on users to be minimized. We also soft-mount so that we can force-dismount if the volume is going to be down for an extended period of time (has happened once I think). The user just gets a "no such blob" error if they try to access the message. Note that for this to work, you MUST ensure that calendar blobs aren't HSM'd, as they're often read by a calendar SOAP call (probably something to do with recurrence) which is quite disruptive if they're not there. We had an RFE to make this configurable and that was done for Zimbra 6.

> 
> 8. When you initially turn on HSM, how long did it take for a complete
> run? How long does it take after the intial run? 

Hard to remember now, but I think the initial run took a few days. But we let fac/staff migrate in all of their old mail from their desktops (which proved to be very popular!). Our migration scripts preserved dates properly so that Zimbra "knew" the mail was old, so as soon as HSM kicked in, all of that old mail got swept off to HSM.
 


-- 
Steve Hillman                                IT Architect
hillman@sfu.ca                               IT Infrastructure
778-782-3960                                 Simon Fraser University