[Zope] ZServer stops responding !? Help !?

Sat Apr 24 16:30:31 EDT 2004

Jean-Francois,

What processor are you using?  How many CPUs?  

	-d

On Sat, 24 Apr 2004 Jean-Francois.Doyon at CCRS.NRCan.gc.ca wrote:

> Chris,
> 
> Thanks for tips.
> 
> Here's what I've tried:
> 
> When the sites looks like it's no longer responding, I connect with
> the monitor.
> 
> Running Zope.app() is very slow, sometimes so slow I give up and try again.
> 
> Once I get that setup, I run:
> 
> app.Control_Panel.DebugInfo.dbconnections()
> 
> >From what I gather, not all connections are taken up.  I have at least 2
> that are free:
> 
> [{'info': ' (1391)', 'version': '', 'opened': 'Sat Apr 24 15:38:59 2004
> (31.63s)'}, {'info': ' (7290)', 'version': '', 'opened': 'Sat Apr 24
> 15:30:36 2004 (534.16s)'}, {'info': "({'HTTP_ACCEPT': ...
> 
> I try again a couplle of minutes later and I see:
> 
> [{'info': ' (1391)', 'version': '', 'opened': 'Sat Apr 24 15:38:59 2004
> (181.71s)'}, {'info': ' (7290)', 'version': '', 'opened': 'Sat Apr 24
> 15:30:36 2004 (684.24s)'}, {'info': "({'HTTP_ACCEPT': ...
> 
> The 4 other requests (I have 6 threads) are the same, they haven't changed.
> So I think we can exclude running out of threads/db connections as a source
> of the problem.  The other 4 requests are for various content types. the
> content types themselves work fine, so I'm going ot take note of which ones
> they are and see if one keeps recurring or something.  Actually 2 of those
> do backend http calls, could there be some socket/timeout issue ? The call
> is to a CGI on the very same server though, so I'm confident it's running
> fine.
> 
> At this point the monitor stops responding, in the middle of outputting that
> second list. I hit enter and I get:
> 
> error: uncaptured python exception, closing channel <__main__.monitor_client
> connected at 0x400d5c0c> (socket.error:(9, 'Bad file descriptor')
> [/usr/local/lib/python2.3/asynchat.py|initiate_send|218]
> [/usr/local/lib/python2.3/asyncore.py|send|337])
> 
> I go look at my trace log, and stuff is still appearing in there ... Though
> nothing gets returned.
> 
> CPU usage right now is not particularely big ...
> 
> [zope at tincup log]$ ps auxw | grep Zope
> zope     31498  0.0  0.2  6484 4576 ?        S    15:26   0:00
> /usr/local/bin/python2.3 /usr/local/Zope-2.7-Core/lib/python/zdaemon/
> zope     31499  0.9  8.5 192848 176496 ?     S    15:26   0:13
> /usr/local/bin/python2.3 /usr/local/Zope-2.7-Core/lib/python/Zope/Sta
> zope     31500  0.0  8.5 192848 176496 ?     S    15:26   0:00
> /usr/local/bin/python2.3 /usr/local/Zope-2.7-Core/lib/python/Zope/Sta
> zope     31501  7.0  8.5 192848 176496 ?     S    15:26   1:37
> /usr/local/bin/python2.3 /usr/local/Zope-2.7-Core/lib/python/Zope/Sta
> zope     31502  5.8  8.5 192848 176496 ?     S    15:26   1:21
> /usr/local/bin/python2.3 /usr/local/Zope-2.7-Core/lib/python/Zope/Sta
> zope     31503  4.2  8.5 192848 176496 ?     S    15:26   0:58
> /usr/local/bin/python2.3 /usr/local/Zope-2.7-Core/lib/python/Zope/Sta
> zope     31504  0.5  8.5 192848 176496 ?     S    15:26   0:07
> /usr/local/bin/python2.3 /usr/local/Zope-2.7-Core/lib/python/Zope/Sta
> zope     31505  5.1  8.5 192848 176496 ?     S    15:26   1:11
> /usr/local/bin/python2.3 /usr/local/Zope-2.7-Core/lib/python/Zope/Sta
> zope     31506  1.6  8.5 192848 176496 ?     S    15:26   0:22
> /usr/local/bin/python2.3 /usr/local/Zope-2.7-Core/lib/python/Zope/Sta
> 
> I'm going ot go do a requestprofiler now see what comes out ...
> 
> Thanks again,
> J.F.
> 
> 
> -----Original Message-----
> From: Chris McDonough [mailto:chrism at plope.com]
> Sent: April 24, 2004 2:15 PM
> To: Jean-Francois.Doyon at CCRS.NRCan.gc.ca
> Cc: zope at zope.org
> Subject: Re: [Zope] ZServer stops responding !? Help !?
> 
> 
> On Sat, 2004-04-24 at 13:53, Jean-Francois.Doyon at CCRS.NRCan.gc.ca wrote:
> > G'day,
> > 
> > I've got a rather bizarre but catastrophic problem.
> > 
> > ZServer seems to stop responding.  Sometimes it does so after days of
> > running, sometimes after a few seconds or minutes of uptime.
> > 
> > I know it's ZServer because I can talk to the monitoring port without
> > problem.
> 
> That may be a bit of flawed logic, because ZServer also runs the monitor
> port.
> 
> > Also, the apache processes just pile up up to the limit allowed,
> > suggesting the proxying is not getting replies from the downstream
> > server.
> > 
> > The strange thing is the cause seems to be occacional, or vary.  For
> > hours on end I can sit there and restart it, and within minutes it stops
> > responding ... Then suddenly the problem "disappears", I restart it, and
> > I wait .... and nothing happens, it just keeps running.  Nothing else
> > abnormal is going on the server so far as I can tell, there is very
> > little memory swapped, and the CPU usage is not abnormally high.
> 
> It sounds as if Zope is doing something which blocks, consuming all
> database threads.
> 
> > I used to have this problem very very rarely in the past, but since I
> > upgraded to Zope 2.7, it seems to have gotten much worse :(
> > 
> > I tried accesing the DebugPanel from the monitor, but can't seem to get
> > it to do anything useful ... I don't know where else to look to find the
> > cause of this.
> > 
> > This causes serious uptime problem on our main, high traffic site, which
> > is Very Bad.
> > 
> > I'm on RedHat 7.3 (fully patched)
> > Python 2.3.3 (custom compiled)
> > Zope 2.7
> > CMF 1.4.x (I forget ... the latest!)
> > Psycopg (Latest also)
> > And a variety of other products.
> 
> I'd suggest using the "big M" or "trace" logging features along with the
> requestprofiler script to find out where the problem might be.
> 
> - C
> 
> _______________________________________________
> Zope maillist  -  Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists - 
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev )
> 