Leaking HTTP requests (was: RE: [Zope] Leaking Acquisition.Im plic itAcquirerWrapper)

Jean-Francois.Doyon at CCRS.NRCan.gc.ca Jean-Francois.Doyon at CCRS.NRCan.gc.ca
Mon May 17 13:14:14 EDT 2004


G'Day,

OK, so I've tried to flush the cache, and look at the ref counts, and
nothing jumped out at me other than the usual suspects.

I often see refcounts to a variety of my own content types/classes, but
they seem pretty stable.  Since they represent a large portion of our
content, I'm not worrying too much about them, though I probably should
and soon will.

Here's an interesting technique I'm using:

First, in ZPublisher I patch HTTPRequest.__init__ to maintain a global
list of weakref's to the instances created.  This gives me a way to access
existing HTTPRequest instances, notably the ones that are sticking around
that shouldn't be.

I'm also adding a unique id (In the form of a counter) to each request so
I can uniquely identify them.

Then, I write an external method and use the gc module to find out the
referrers/referants.

I'm hoping this will give me some clues as to what's holding the requests
around.

Right now I've managed to use gc.get_referants() with great success.

I can dump (using pprint) the request object and read it's contents.

Once I've got a few samples taken, I'll hopefully be able to track down
whihc ones are actual leaked requests (by their id), and see if they
have anything in common.

I tried briefly gc.get_referrers() but I get ugly errors.

Another idea I had was to through the frame stack upon __del__ of a req
and log that info to see if there's a commonality to those also.

One question:

is this bad when applied to a persistent object?:

setattr(self,'thumbnail', Image(id='thumbnail') )

(I've monkey patched CMFImage to have thumbnails)

Thanks and cheers,
J.F.


-----Original Message-----
From: Dieter Maurer [mailto:dieter at handshake.de]
Sent: May 16, 2004 5:48 PM
To: Jean-Francois.Doyon at CCRS.NRCan.gc.ca
Cc: stefan at epy.co.at; zope at zope.org
Subject: RE: Leaking HTTP requests (was: RE: [Zope] Leaking
Acquisition.Im plic itAcquirerWrapper)


Jean-Francois.Doyon at CCRS.NRCan.gc.ca wrote at 2004-5-14 17:23 -0400:
> ...
>I'll leave the iste running throught the week-end and we'll see what shows
>up.
>"high" refcounts is somewhat arbitrary I guess ... When you say high you
>mean
>high and stable or high and growing?

In fact, I mean unexpectedly high. The relevant references counts
are not necessarily the highest ones.

For example, when you make catalog searches, it is completely
normal that you see lots of "DateTime" objects. Nevertheless,
you can be sure that they are not the cause of memory leaks.
The same applies for acquisition wrappers.
You must look for usually rare objects that suddenly show up
in large numbers.

It may help very much to clear the ZODB cache
("Control_Panel" --> "Database management" --> <databases> --> "Flush Cache"
--> "Minimize"). This should delete all objects from memory that
are "normal" artefacts of Zope processing. Apart from a few objects
only the leaked ones should remain.

Be warned, there was a bug in "cPickleCache.c" that led "Minimize"
enter an infinite loop. I am not sure whether this bug is
fixed in the current Zope 2.7 release.

>Right now, the only suspect class as far as refcounts (Other than the HTTP
>requests) is the DateTime one.  I have some content ones that look high,
>but then they're highly used (CMFImage for example).

"DateTime" are normal, "HTTP requests" is strange....


>Apart from using aq_explicit in a few places, I do also use __of__ in one
>case ... To do things like:
>
>    def getAbstract(self):
>        try:
>            pt = ZopePageTemplate( '', self.abstract,
>'text/html').__of__(self)
>            cookedbody = pt()

Harmless.

The problem in my code has been that I stored an acquisition
wrapped object in my object.

> ...
>How about this?
>
>pt = ZopePageTemplate( '', self.aq_explicit.intropart1, 'text/html'
>).__of__(self)
>
>Could THAT be bad?

Unlikely (unless you store it as attribute of "self").


-- 
Dieter



More information about the Zope mailing list