[Zope] Memory Use problems

[Dieter Maurer]

Douwe Osinga wrote at 2004-3-2 17:47 +0100:
> ...
>The memory use of my Zope instance (2.6.2 with python 2.1 on FreeBSD4)
>increases over time until the whole thing hits the limit (at 512Mb) and
>restarts. I have installed LeakFinder, but there doesn't seem to be an
>obvious memory leak.
>
>>From what I've learned from LeakFinder, it seems that the Catalog is the
>problem, or rather, my use of the catalog. We have 50.000. objects in the
>Catalog, which are instances of ZClasses. These ZClasses have sometimes
>rather large string/text properties and some DateTime() properties. When
>memory use approaches the 500Mbyte, I typically see around 100.000.
>instances of BTrees._IOBTree.IOBucket,

Many of them are likely "ZCatalog" metadata blocks.
They can become huge monsters.

Unless you do statistics or present huge numbers of hits in
a single page, you do not need metadata. Access the object
instead (via "proxy.getObject()").

Under no circumstances put large fields in the metadata table.

Also remove the "bobobase_modification_time" from this table.


> ...
>How do I find where this memory goes? The pages do not take a long time to
>lot (2 seconds max) and the cpu usage is not that high that I'd expect some
>kind of deadlocks there. Also, Zope never seems to release memory, but just
>takes more and more, even if I flush the cache and/or check the perform
>garbage collection box in LeakFinder.

Older versions of Zope had a "cacheExtremDetail" page in
"Control_Panel --> Database".
This is lost with Zope 2.7.

But "ZODB.DB" still has the respective method "cacheExtremeDetail".
It tells you which objects (given by their "oid" and class)
are in the various caches. Given the "oid" you can find
the actual objects (although the analysis is very tedious).

-- 
Dieter