[Zope-dev] Preliminary findings: Zope 2.7 leakage caused by e rrors

[Jean-Francois.Doyon at CCRS.NRCan.gc.ca]

Dieter,

Thanks yet again for all the valuable input.

I've started tracking down/running into the whole ExtensionClasses
instances/issues.

As you suggest, the dict is probably help by BaseRequest.RequestContainer,
which I've
seen grow in parallel to the HTTPRequest refcounts ... Which seems to make
sense.

And as you suggest, as I've been doing this, I've been thinking I'm really
looking forward to getting ri dof ExtensionClass, makes this kind of work
much easier.

The requests themselves look clean, but if they contain ExtensionClasses
based instances
gc wouldn't find them for the same reason.  I've gotten far enough to be
able to tell
there's a variety of them.  I suspect the instance the error occured ON is
what's leaked
(So if it's a CMFDocument, then that said Document ends up leaked).  I was
just getting
to this yesterday, hopefully I'll solve that today.

As Tres suggested, I'm going to go back to looking at Localizer ... It's the
likliest suspect
at this time.  It's just hard for to test since pretty much everything I
have depends on
i18n PT's, and in some cases I have gettext() calls in Python code.
Localizer "wraps"
the Publish.publish, and I'm thinking that when exceptions occur in there
Localizer
needs to be aware of it and react accordingly, which it doesn't right now.

The hunt goes on ...

Cheers,
J.F.

-----Original Message-----
From: Dieter Maurer [mailto:dieter at handshake.de]
Sent: May 20, 2004 3:52 AM
To: Jean-Francois.Doyon at CCRS.NRCan.gc.ca
Cc: zope-dev at zope.org; tseaver at zope.com; chrism at plope.com;
brian at zope.com
Subject: Re: [Zope-dev] Preliminary findings: Zope 2.7 leakage caused by
errors


Jean-Francois.Doyon at CCRS.NRCan.gc.ca wrote at 2004-5-19 14:44 -0400:
> ...
>Right now I've focused on finding out why the Requests are still around.
>
>Using the gc module, I've found that all leaked requests are being held by
a
>dictionary.  If I look at said dictionaries, this is what they might look
>like:
>
>{'REQUEST': <HTTPRequest,
>URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_v
h
>_site/francais/mapsatoz>},
> {'REQUEST': <HTTPRequest,
>URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_v
h
>_site/francais/mapsatoz>},

These dictionaries look like the "instance dict"s of
the "ZPublisher.BaseRequest.RequestContainer"s.

When you have any cycle involving an AcquisitionWrapper,
then its reference to the "RequestContainer" will keep this alive
together with its instance dict.

You may analyse the content of your requests.
Do they contain acquisition wrapped objects (or other objects
with (maybe indirect) references to this dict)?

If not, "REQUEST" does not seem to be the root of your cycles
but rather a side effect.


More important notes:

  *  When you see "AcquisitionWrapper"s in the reference information
     in "Control_Panel --> Debug Information", then
     there *must* also be other prominent ExtensionClass'es --
     the "aq_base" of these wrappers.

     These classes may provide a valuable clue towards the cause
     of the cycles -- of global variables (by the way).

  *  The reference information in "Control_Panel" covers only
     instance objects but not elementary Python data types.

     Thus, when cycles are caused by elementary Python objects,
     we may not directly see them in the list.

     GC's "DEBUG_LEAK" may provide some information.

  *  The leaks may not be caused by cycles at all (though it is
     the most likely cause) but could result from a global
     cache containing references to acquisition wrapped objects.

     Such a cache should never exist -- global variables and
     class attributes must never directly or indirectly reference
     persistent objects -- as this results in nasty persistency bugs.

     Nevertheless, sometimes software is buggy. I noticed
     such a cache in Archetypes 1.3a2 (fixed in newer versions).

....
>Said dictionary does not seem to be referred to by anything, at least not
>anything the gc module can find (gc.get_referrers()).

That's probably because "RequestContainer" is an ExtensionClass
and ExtensionClasses are not yet (only from Zope 2.8 on) GC aware.

> ...
>Also I
>understand zope.org suffers from this? :)

Leaks can have innumerable causes. It is by no means clear
that your leak and the one on Zope.org has the same cause.

>Based on this information also, I have no reason to believe this is caused
>by product code ... I have to guess it's a Zope bug?

I do not observe leaks in my Zope installation -- even when
I make an error stress test.

This does not mean it could not be a Zope bug --
but it implies that such a bug does not surface regularly, at most
under special conditions.

>In some ways this is motivating me to reduce the number of errors that
could
>occur, but because ALL errors seems to cause the problem, I ultimately have
>no control over it and under certain circumstances I suffer heavily from
>this.  404's alone probably cause this (I'm plannign to test that
>explicitely as well).

It does not here....

Something is special with your installation -- maybe "Localizer" as
Tres suggested, maybe another misbehaving product.

-- 
Dieter