Re: zimbra proxy crash

[Tom Golson <tgolson@tamu.edu>]

Hi Amos,

We recently upgraded our 5.0.22 servers to SLES10SP3, 64-bit and started
seeing this same error.  Configuring machines to generate a crash dump
had the interesting side-effect of changing what exactly was logged, but
the results were the same.

I've got open cases with Zimbra and with Novell.  Zimbra suggested
upgrading to the 6.0.8 proxy service.  Oops.  Novell is still analyzing
core dumps, but it looks like -- for us -- the kernel released at the
beginning of the month (2.6.16.60-0.68.1-smp) is resolving the problem.

If you're at kernel .68.1 and still seeing these errors, please let me
know off-list and I'll take that back to Novell.  Prior to the .68.1
update, though, we'd see the hang you describe at least every 48 hours.

Good luck and feel free to contact me directly if you'd like to chat
some more.

--Tom

--
Tom Golson
Computing and Information Services
Texas A&M University

On 9/13/10 1:52 PM, Amos wrote:
> I just opened a support ticket for this, but thought I'd share it here
> too.  on SUSE ES 10, 64-bit, and with Zimbra 6.0.8 (P1), got reports
> that our Zimbra proxy was no longer handling connections.  I checked
> /var/log/messages, and saw this little treat.  For now just telling
> our load balancer to talk directly to our backend message store (only
> one at the moment.)
> 
> Sep 13 12:09:09 zproxy1 kernel: Unable to handle kernel NULL pointer
> dereference at 0000000000000040 RIP:
> Sep 13 12:09:10 zproxy1 kernel: <ffffffff80287225>{sock_poll+17}
> Sep 13 12:09:10 zproxy1 kernel: PGD 24390067 PUD 29b86067 PMD 0
> Sep 13 12:09:10 zproxy1 kernel: Oops: 0000 [1] SMP
> Sep 13 12:09:10 zproxy1 kernel: last sysfs file:
> /devices/pci0000:00/0000:00:11.0/irq
> Sep 13 12:09:10 zproxy1 kernel: CPU 1
> Sep 13 12:09:10 zproxy1 kernel: Modules linked in: nfs xt_tcpudp
> xt_pkttype ipt_LOG xt_limit autofs4 nfsd exportfs lockd nfs_acl sunrpc
> i
> pt_REJECT xt_state iptable_mangle iptable_nat ip_nat iptable_filter
> ip_conntrack nfnetlink ip_tables ip6_tables x_tables apparmor loop dm
> _mod floppy shpchp i2c_piix4 e1000 pci_hotplug i2c_core mptctl ide_cd
> intel_agp cdrom parport_pc lp parport reiserfs ata_piix sg ahci lib
> ata edd fan thermal processor mptspi mptscsih mptbase
> scsi_transport_spi piix sd_mod scsi_mod ide_disk ide_core
> Sep 13 12:09:10 zproxy1 kernel: Pid: 27031, comm: nginx Not tainted
> 2.6.16.60-0.67.1-smp #1
> Sep 13 12:09:10 zproxy1 kernel: RIP: 0010:[<ffffffff80287225>]
> <ffffffff80287225>{sock_poll+17}
> Sep 13 12:09:10 zproxy1 kernel: RSP: 0018:ffff81003f677ed0  EFLAGS: 00010246
> Sep 13 12:09:10 zproxy1 kernel: RAX: 0000000000000000 RBX:
> ffff81007f46e340 RCX: 0000000000000000
> Sep 13 12:09:10 zproxy1 kernel: RDX: 0000000000000000 RSI:
> ffff810036470100 RDI: ffff81003d9a55c0
> Sep 13 12:09:10 zproxy1 kernel: RBP: ffff8100b3bdc110 R08:
> ffff8100456e39a8 R09: 2029646963756c28
> Sep 13 12:09:10 zproxy1 kernel: R10: 000000000000657b R11:
> 0000000000000246 R12: ffff81003f677f38
> Sep 13 12:09:10 zproxy1 kernel: R13: ffff8100b3bdc0c0 R14:
> 0000000000000000 R15: 00000000005d4060
> Sep 13 12:09:10 zproxy1 kernel: FS:  00002af5b1fd33b0(0000)
> GS:ffff8100bfecd540(0000) knlGS:0000000000000000
> Sep 13 12:09:10 zproxy1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0:
> 000000008005003b
> Sep 13 12:09:10 zproxy1 kernel: CR2: 0000000000000040 CR3:
> 000000002fbe6000 CR4: 00000000000006e0
> Sep 13 12:09:10 zproxy1 kernel: Process nginx (pid: 27031, threadinfo
> ffff81003f676000, task ffff810047bb8810)
> Sep 13 12:09:10 zproxy1 kernel: Stack: ffffffff801ae5f9
> 0000000000000200 ffff8100456e3980 000000000000195f
> Sep 13 12:09:10 zproxy1 kernel:        0000000000000002
> ffff81004022b3a8 0000000000000147 00000000006beef0
> Sep 13 12:09:10 zproxy1 kernel:        ffffffff80187967 ffff810037da2898
> Sep 13 12:09:10 zproxy1 kernel: Call Trace:
> <ffffffff801ae5f9>{sys_epoll_wait+633}
> <ffffffff80187967>{vfs_write+328}
> Sep 13 12:09:10 zproxy1 kernel:        <ffffffff8010ae36>{system_call+126}
> Sep 13 12:09:10 zproxy1 kernel:
> Sep 13 12:09:10 zproxy1 kernel: Code: 4c 8b 59 40 41 ff e3 48 89 f0 48
> 8b b7 c8 00 00 00 48 89 c2
> Sep 13 12:09:10 zproxy1 kernel: RIP <ffffffff80287225>{sock_poll+17}
> RSP <ffff81003f677ed0>
> Sep 13 12:09:10 zproxy1 kernel: CR2: 0000000000000040