Re: Mail.app DoS

[pam.buffington@oit.gatech.edu]

We have had a very similar issue, and have been working with Zimbra on it. (Doug might respond in a bit with more information) What is your maxmessagesize? and what is your zimbraMessageCacheSize ? Try increasing the zimbraMessageCacheSize to 100MB.


*Excerpt of a message*
When the Imap client issues N partial fetch commands for the same message, we expect the server parses the same message only once(The # of N is decided by the IMAP client 's buffer) 

Now the stack trace from GA confirmed that the same message was actually parsed again and again instead of  just being parsed once as we expected. This is root cause of the high CPU usage. 
a) between 16:00 ~ 17:00, ImapServer-690982 consumes 261493 CPU ticks 
b) between 16:00 ~ 17:00, ImapServer-690982 issues 1500 partial FETCH cmds for part 1 of the MIME message 29042.part 1 is about 47MB. 
c) between 16:00 ~ 17:00, The continuous thread dumps showed that the ImapServer-690982 was constantly doing parsing when we expect only one parsing for the very first fetch command of  mime message 29042. 
you can find log in qa24 
imap command trace: 
 /opt/support/gt/0503/ImapServer-690982.log
imap thread dumps stack trace:
 /opt/support/gt/0503/ImapServer-690982.stacktrace.log
d) GC was very active when ImapServer-690982 was doing partial fetch of that big mime message. it's possible that the MimeMessage was quickly purged from MessageCache right after it's parsed due to the small Message cache size and large amount of new parsed messages. When server receives the subsequent partial command for the same MimeMessage, it has to reparse because it fails to find that message in Message Cache. It could happen when the server is very busy and the size of the targeted mime message for partial fetch is very big. 
zimbraMessageCacheSize: 20971520
zimbraMailDiskStreamingThreshold: 1048576

zmailbox_mailbox_cache_size=1


Action Item: Thom asked Eng for next step. Increasing zimbraMessageCacheSize to 300MB is recommended  [wiki recommends 100MB, but server-team agrees to use 300MB: http://wiki.zimbra.com/wiki/Performance_Tuning_Guidelines_for_Large_Deployments]



**

----- "David N. Blank-Edelman" <dnb@ccs.neu.edu> wrote:

> From: "David N. Blank-Edelman" <dnb@ccs.neu.edu>
> To: "Helpful Zimbra Admins" <zimbra-hied-admins@sfu.ca>
> Sent: Wednesday, May 5, 2010 1:17:07 PM GMT -05:00 US/Canada Eastern
> Subject: Mail.app DoS
>
> Hi-
>   Given the preponderance of macs in edu, I'm curious if other sites
> have seen this particular phenomena. I'm about to burn a support
> ticket on this because it is getting a bit troublesome.
> 
> We're seeing an unpleasant situation where by if a Mail.app user
> deletes a bunch of messages and then empties their trash folder,
> sometimes Mail.app gets into this tight loop with out Zimbra 5.x
> server. Mail.app at that point starts to send commands at a rate of N
> per second when N is a surprisingly large number. Zimbra, in turn,
> logs tens of thousands of lines that look like this:
> 
> > 2010-05-04 11:41:35,306 INFO [ImapSSLServer-634329]
> [name=user@zimbra;mid=419;ip=ip;] imap - S: 291726.299 BAD parse
> error: invalid message sequence number: 1:*
> > 2010-05-04 11:41:35,362 INFO [ImapSSLServer-634329]
> [name=user@zimbra;mid=419;ip=ip;] imap - S: 291727.299 BAD parse
> error: invalid message sequence number: 1:*
> > 2010-05-04 11:41:35,432 INFO [ImapSSLServer-634329]
> [name=user@zimbra;mid=419;ip=ip;] imap - S: 291729.299 BAD parse
> error: invalid message sequence number: 2:*
> 
> while this is going on. As you'd expect, this drives the load up on
> the server significantly and only abates when the bad connection goes
> away. I haven't yet stuck a sniffer on the line to catch the precise
> command it is sending. If mail.app is restarted or the connection
> forcefully broken, the mail client becomes sane again.
> 
> I haven't found much in the forums (we added a message to this active
> thread
> http://www.zimbra.com/forums/administrators/32576-mailbox-log-full-invalid-message-sequence-number-errors.html#post182064).
> 
> So, anyone else having this fun, and were you able to do anything
> about it?
> 
>        -- dNb

-- 
Pam Buffington, Department Manager II - EAS
   Office of Information Technology/A&I :: Georgia Institute of Technology
   pam.buffington@oit.gatech.edu :: (404) 385-7518
   https://mail.gatech.edu/home/pc21?fmt=freebusy   (my calendar availability)