[Zope] Re: Zope Lockup (ZPublisher stops responding?)
Mark Thomas
markt@kooee.com.au
Tue, 22 Apr 2003 07:58:22 +1000
Just wanted to add my experiences too, although no solutions as yet.
1) I don't use the ZEO either. Nor do I use the CMF.
2) I am still having a slow memory leak problem, but I'm currently
managing it by restarting the Zope server every 4 or 5 days. (I'm on
Zope 2.6.0, Python 2.1.3, Red Hat linux.) I may have slowed up the
memory leak by reorganising some SESSION variable usage, but no hard
evidence on this.
3) I too experience the 'lockup', or 'spinning' or 'runaway memory'
phenomenon that you've mentioned, however it hasn't happened for a
couple of weeks now.
4) I used to think the runaway memory problem was related to the slow
memory leak problem. It seemed, that when the slow memory leak had
passed a certain threshold (maybe 50Mb, where my normal memory usage is
around 35Mb), then the runaway memory problem was much more likely. I'm
no longer sure that this is the case, as I've quite often seen the
runaway memory thing happen just hours after restarting Zope. So, they
appear to be two separate problems.
5) I think the runaway memory thing may be related to 'spinning' processes.
6) When the memory runs away (climbs up to the limit of actual system
memory within a few minutes), Zope first locks up, then often will
restart itself a few minutes later. Maybe this is linux killing the
Zope process to try and free up memory, I don't know.
7) I have studied the Zope logs on a couple of occasions after the Zope
lockup, and have found no unusual requests or consistent behaviour.
8) I have witnessed on a couple of occasions, something that may be
described as a spinning thread, but one that resolves itself. I kick
off a batch-type process (usually takes 2-3 minutes to run and the
browser sits their waiting till it is finished). The Zope process kicks
off some MySQL selects, runs some scripts, does some SQL updates, and
presents a report at the end. By looking at the Zope debug information
page (in the control panel), I have seen the process run, then abort,
then restart itself on the same thread. This loop happened 4 or 5
times, then the process finished normally (and all used memory was freed
correctly). I don't know why this happens sometimes, but could it be
that this is the problem that causes the memory runaway if the process
stays in that loop for some reason? The browser session may have been
closed while this process was running - which may be contributing somehow?
There's some good thoughts on debugging a spinning Zope process by Mr
4am, but it would be a very rare occasion for me to catch the spinning
process. Usually, if I'm online and notice the problem, the system
rapidly slows to a point where I can't SSH in and do anything
worthwhile. My main priority at that point is to kill the Zope process
before it eats up all the system memory. Once that happens, I usually
have to wait for my hosters to reboot the server for me. I need to
investigate ways of configuring this in Linux, to limit the Zope
processes to certain memory limits so as not to kill the entire machine.
There's probably a lot of us out there dealing with these types of
problems, but maybe just putting up with them. I'm quite concerned
about ramping up my web business on this Zope site though, because I
know it will aggravate the memory problem. It surprises me at times,
that large corporations can be using Zope in very active web sites. Do
they NOT have these stability problems at all, or do they just restart
Zope regularly to avoid problems? I feel quite foolish that I can't get
to the bottom of this problem, but wonder if there are many other zope
developers out there dealing with the same thing?
MT
Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
> Dieter,
>
> Thanks for the update ...
>
> My situation might be somewhat different though:
>
> 1) I don't use ZEO
> 2) This happens without me trying to "kill" anything, so basically, if
> there's a "kill" signal being generated, it's not from me!
> 3) Which brings on the old problem of memory leakage ... I went through
> EXTENSIVE reworking of my Zope setup, and it looks like the leak was gone
> ... Could it be it's really not? When I still had a noticeable memory
> problem, sometimes the master thread/process would actually get a SIG11 or
> SIG25 (If I remember correctly) and restart Zope properly. Could it be I'm
> actually STILL having a memory problem, only Zope no longer reacts nicely to
> the signal?
>
> I'm pretty sure my memory leak is gone honestly, the numbers do stabilize
> after a day or so ... But the "lockup" problem occurs so much more often now
> that I can't really tell!
>
> Like I said, storage operations actually only happen internally to our
> organization, so I'll try to track it down to see if there's a correlation
> next time I see this happening.
>
> Thanks for the help!
> J.F.
>
> -----Original Message-----
> From: Dieter Maurer [mailto:dieter@handshake.de]
> Sent: Friday, April 18, 2003 4:58 PM
> To: Jean-Francois.Doyon@CCRS.NRCan.gc.ca
> Cc: zope@zope.org
> Subject: Re: [Zope] Zope Lockup (ZPublisher stops responding?)
>
>
> Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote at 2003-4-17 14:09 -0400:
> > ....
> > The symptoms:
> >
> > Zope stops responding to HTTP requests
> > Zope memory usage increases suddenly quite a bit
> > The number of apache processes increases quite a bit also
> > Can't kill Zope normally, need to do a -9 ...
>
> We see this quite regularly but not yet often enough that
> we could analyse it.
>
> We understand the necessity to use "kill -9" to kill Zope.
>
> It indicates that Zope was performing a storage operation
> when the "kill" signal was received. In this
> case, the main thread (executing the "kill" signal handler)
> blocks on acquiring the storage lock. When the storage is
> a ZEO storage, deadlock occurs as the (now blocked) main thread is needed
> for ZEO communication.
>
> There is a collector report with patch for it.
>
>
> Dieter
>
> _______________________________________________
> Zope maillist - Zope@zope.org
> http://mail.zope.org/mailman/listinfo/zope
> ** No cross posts or HTML encoding! **
> (Related lists -
> http://mail.zope.org/mailman/listinfo/zope-announce
> http://mail.zope.org/mailman/listinfo/zope-dev )
>