Re: High-volume server settings

[Xueshan Feng <sfeng@stanford.edu>]

Dmitry,

We have 8 mailbox servers, 3 ldap severs. Each server has around 5000 accounts.
Hardware is on Dell 1950 with 16GB memory. Our Zimbra store and backup partitions are on NetAPP SAN disks. 

Since you got http errors, you might want to look specifically  if you have lots of ldap timeout in /opt/zimbra/log/mailbox.log file.

In our load testing, we ran into ldap timeout issue which translated to http errors.  We ended up turning off 
tls between Zimbra mailservers and Zimbra ldap servers ( zmlocalconfig -e ldap_starttls_supported=0 ). Because Openldap connection pooling is not
supported for TLS connections, tuning TLS off enables connection pooling and gives better performance there.  

If you have dbconn errors, you'd want to tune mysql  - more cache for open_table and  more buffer size for innodb.  I attached our changes to my.cnf below. 

Also, "zmstat-chart" is very useful for trouble-shooting performance problem. We run it every 20 minutes and the chart is automatically posted on a website to help
analyze problems and to measure tunning result. 

Another major tuning we did was on storage (you will need to work with storage people and give them iostats on disk usage) which made a huge improvement. 

We still have  poor performance spikes but a lot better than a few months ago. 

Xueshan

Here is our configuration script:

#!/bin/sh

PATH=/usr/sbin:/sbin:/usr/bin:/bin:/opt/zimbra/bin ; export PATH

# This was changed from 25 
setuidgid zimbra zmlocalconfig -e mailboxd_java_heap_memory_percent=30

# Tune garbage collector to use all 8 cores. Recommendation from Zimbra
setuidgid zimbra zmlocalconfig -e mailboxd_java_options="-client -XX:+UseParallelGC -Xss256k -XX:NewRatio=2 -Djava.awt.headless=true -XX:M
axPermSize=256m -XX:SoftRefLRUPolicyMSPerMB=1"

# We bumped up from 50
setuidgid zimbra zmprov mcf zimbraHttpSSLNumThreads 250

# Monitor the thread usuage by???  We upped this from 200
setuidgid zimbra zmprov mcf zimbraImapNumThreads 2000

# We doubled the default from 20
setuidgid zimbra zmprov mcf zimbraLmtpNumThreads 50

# We changed from 100
setuidgid zimbra zmprov mcf zimbraPop3NumThreads 300

# Tune off TLS
setuidgid zimbra zmlocalconfig -e ldap_starttls_supported=0

# MySQL memory, increased from 30 percent
setuidgid zimbra zmlocalconfig -e mysql_memory_percent=40

# Maximun db connections
setuidgid zimbra zmlocalconfig -e zimbra_mysql_connector_maxActive=200

MySQL (/opt/zimbra/conf/my.cnf) also needs to be tuned to give mysqld more memory cache.

Here are the values that we changed:

thread_cache_size      = 150
max_connections   = 220

sort_buffer_size = 2M
read_buffer_size = 2M

# Increase the size of the table cache, since each mailbox has its
# own set of tables
table_cache = 1200

innodb_buffer_pool_size   = 6730209680
innodb_open_files         = 2660

----- "Dmitry S. Makovey" <dmitry@athabascau.ca> wrote:

| Hi everybody,
| 
| We are running Zimbra in a limited capacity right now. We've sorted
| out issues 
| with OS/hardware level (I think), but now it seems we're stuck with 
| application configuration. Apparently some on that list are running
| several 
| thousands accounts per mailbox just fine and ours is stuttering (lots
| of 50x 
| HTTP errors on proxy boxes) on a measly several hundred accounts. Any
| tips as 
| far as app config goes? For example: what are the thread numbers for 
| multi-thousand boxes? Ours right now are:
| 
| $ zmprov gs mailboxX.ca | grep -i threads
| zimbraHttpNumThreads: 1000
| zimbraHttpSSLNumThreads: 50
| zimbraImapNumThreads: 200
| zimbraLmtpNumThreads: 20
| zimbraPop3NumThreads: 100
| zimbraScheduledTaskNumThreads: 20
| 
| Is there anything else of particular interest for larger deployments?
| 
| -- 
| Dmitry Makovey
| Web Systems Administrator
| Athabasca University
| (780) 675-6245