[Zope] zope 2.6.4 lockups

Mon Mar 22 07:20:34 EST 2004

Hello all,

Zope 2.6.4 has got stuck several times on my server. I have Debian sarge
with the zope 2.6.4-1 package and python 2.2.3-7 (I know Zope 2.6 should
use python2.1, but the debian package uses 2.2). I don't use ZEO.

The problems started recently but I cannot pin down any specific event
that triggered it. It may be related to upgrading zope from 2.6.2, or a
Plone upgrade (I now have 2.0RC6, used to have RC3), or a kernel
upgrade. The machine is a 2-processor Xeon currently running a SMP
kernel (kernel-image-2.4.25-1-686-smp). Zope runs with the default 4
threads.

Zope is running behind Apache. The symptom is that zope stops responding
to requests, so the requests just hang until timeout. The processes are
running, but a kill -TERM won't kill them. kill -9 helps.

I haven't yet been able to repeat this in debug mode because the event
is fairly rare. Also I haven't debugged the locked up processes with
gdb, but will do so the next time it happens following instructions I've
seen on this list. However, if you have any advice on how to further
investigate this problem, I'd love to hear it.

Below are some excerpts from Z2-detailed.log that show the sequence of
events leading to lockup. It seems that two requests come in at the same
time, and a crash happens in the middle of processing.

I'm grateful for any help you can provide,
-Osma

Z2-detailed.log.1:A 190481452 2004-03-16T12:40:24 200 7607
Z2-detailed.log.1:E 190481452 2004-03-16T12:40:24
Z2-detailed.log.1:B 190559468 2004-03-16T12:40:24 GET /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//tintr/ploneIEFixes.css
Z2-detailed.log.1:I 190559468 2004-03-16T12:40:24 0
Z2-detailed.log.1:A 190559468 2004-03-16T12:40:24 200 1702
Z2-detailed.log.1:E 190559468 2004-03-16T12:40:24
Z2-detailed.log.1:B 181840020 2004-03-16T12:40:26 GET /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//lpl/uutiset
Z2-detailed.log.1:I 181840020 2004-03-16T12:40:26 0
Z2-detailed.log.1:B 190159428 2004-03-16T12:40:26 GET /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//tintr/asukkaan_opas
Z2-detailed.log.1:I 190159428 2004-03-16T12:40:26 0
Z2-detailed.log.1:U 000000000 2004-03-16T12:40:27 System startup

Z2-detailed.log.1:A 204515844 2004-03-16T15:00:20 200 1702
Z2-detailed.log.1:E 204515844 2004-03-16T15:00:20
Z2-detailed.log.1:B 173323884 2004-03-16T15:00:23 POST /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//
Z2-detailed.log.1:I 173323884 2004-03-16T15:00:23 312
Z2-detailed.log.1:A 173323884 2004-03-16T15:00:23 200 718
Z2-detailed.log.1:E 173323884 2004-03-16T15:00:23
Z2-detailed.log.1:B 207153412 2004-03-16T15:00:23 POST /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//aks/foorumi1/toijala/forum_add_topic_form
Z2-detailed.log.1:I 207153412 2004-03-16T15:00:23 1492
Z2-detailed.log.1:B 207618796 2004-03-16T15:00:24 GET /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//tintr
Z2-detailed.log.1:I 207618796 2004-03-16T15:00:24 0
Z2-detailed.log.1:U 000000000 2004-03-16T15:00:24 System startup

Z2-detailed.log.1:B 184582692 2004-03-17T11:31:50 GET /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//lpl/navtree-item-active.gif
Z2-detailed.log.1:I 184582692 2004-03-17T11:31:50 0
Z2-detailed.log.1:A 184582692 2004-03-17T11:31:50 500 1749
Z2-detailed.log.1:E 184582692 2004-03-17T11:31:50
Z2-detailed.log.1:B 186258404 2004-03-17T11:31:50 GET /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//lpl/bg_footer.gif
Z2-detailed.log.1:I 186258404 2004-03-17T11:31:50 0
Z2-detailed.log.1:A 186258404 2004-03-17T11:31:50 500 1749
Z2-detailed.log.1:E 186258404 2004-03-17T11:31:50
Z2-detailed.log.1:A 185418812 2004-03-17T11:31:50 500 709
Z2-detailed.log.1:E 185418812 2004-03-17T11:31:50
Z2-detailed.log.1:U 000000000 2004-03-17T11:31:50 System startup

Z2-detailed.log.1:A 174372980 2004-03-20T18:03:58 304 316
Z2-detailed.log.1:E 174372980 2004-03-20T18:03:58
Z2-detailed.log.1:B 174079364 2004-03-20T18:04:03 GET /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//tintr
Z2-detailed.log.1:I 174079364 2004-03-20T18:04:03 0
Z2-detailed.log.1:B 175770628 2004-03-20T18:04:03 GET /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//tintr/asukkaan_opas
Z2-detailed.log.1:I 175770628 2004-03-20T18:04:03 0
Z2-detailed.log.1:A 174079364 2004-03-20T18:04:03 200 24993
Z2-detailed.log.1:E 174079364 2004-03-20T18:04:03
Z2-detailed.log.1:B 176679788 2004-03-20T18:04:03 GET /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//tintr/asukkaan_opas
Z2-detailed.log.1:I 176679788 2004-03-20T18:04:03 0
Z2-detailed.log.1:U 000000000 2004-03-20T18:04:04 System startup

Z2-detailed.log.2.gz:A 243458436 2004-03-11T17:29:34 200 1700
Z2-detailed.log.2.gz:E 243458436 2004-03-11T17:29:34
Z2-detailed.log.2.gz:B 210607836 2004-03-11T17:29:34 GET /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//nkst/plone_javascripts.js
Z2-detailed.log.2.gz:I 210607836 2004-03-11T17:29:34 0
Z2-detailed.log.2.gz:A 210607836 2004-03-11T17:29:34 304 316
Z2-detailed.log.2.gz:E 210607836 2004-03-11T17:29:34
Z2-detailed.log.2.gz:B 243686164 2004-03-11T17:29:36 GET /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//nkst/tarjouspyynto
Z2-detailed.log.2.gz:I 243686164 2004-03-11T17:29:36 0
Z2-detailed.log.2.gz:B 243745428 2004-03-11T17:29:36 GET /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//nkst/toimi_nain/paatos/paatos_kaikki
Z2-detailed.log.2.gz:I 243745428 2004-03-11T17:29:36 0
Z2-detailed.log.2.gz:U 000000000 2004-03-11T17:29:36 System startup

Z2-detailed.log.2.gz:B 181253292 2004-03-12T10:21:24 GET /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//aks/document_icon.gif
Z2-detailed.log.2.gz:I 181253292 2004-03-12T10:21:24 0
Z2-detailed.log.2.gz:A 181253292 2004-03-12T10:21:24 304 299
Z2-detailed.log.2.gz:E 181253292 2004-03-12T10:21:24
Z2-detailed.log.2.gz:B 180156052 2004-03-12T10:21:24 GET /VirtualHostBase/http/prm.mbc.fi:80/prmmedia/VirtualHostRoot//aks/arkisto/bar_1_10.gif
Z2-detailed.log.2.gz:I 180156052 2004-03-12T10:21:24 0
Z2-detailed.log.2.gz:A 180156052 2004-03-12T10:21:24 304 298
Z2-detailed.log.2.gz:E 180156052 2004-03-12T10:21:24
Z2-detailed.log.2.gz:A 178613132 2004-03-12T10:21:24 500 709
Z2-detailed.log.2.gz:E 178613132 2004-03-12T10:21:24
Z2-detailed.log.2.gz:U 000000000 2004-03-12T10:21:25 System startup

-- 
*** Osma Suominen *** ozone at iki.fi *** http://www.iki.fi/ozone/ ***