[ZWeb] ZCatalog Issues

[Jens Vagelpohl]

> Note that zope.org currently has its per-connection database cache 
> size set to
> 23,000 objects.  The catalog can not fit in that space, and even if it 
> did,
> we'd run out of memory.  The box has 2 GB, and between two app 
> servers, there
> are eight connections.  Each connection maintains its own copy of the
> database.  340 MB is probably a low estimate of the catalog's full 
> resident
> unpickled size, but I'll use it anyway: keeping this catalog in memory 
> would
> take at least 340 MB * 8 = 2.7 GB.  That's also ignoring the size of 
> other
> objects loaded from the database connections.

The first step I would take is to reduce the number of database 
connections to 3 for both appservers, just to see how much relief that 
brings. I would also fill the box with 4 GB RAM (it's cheap), and then 
start to manipulate the cache sizes.


> There is a bit of good news: zope.org is not consuming gobs of RAM due 
> to a
> memory leak.  I wrote a small Python C extension that uses mallinfo() 
> to
> reveal how much heap a Python process is actually using for objects, 
> which is
> often much smaller than the process size as the operating system sees 
> it.
> Whenever I flush the caches in Zope, its heap usage shrinks to less 
> than 10%
> of its process size.  That means most of the memory is consumed by
> reclaimable ZODB objects.  (I'll post the C extension on the web if 
> anyone is
> interested.)

I'd definitely be interested in this tool.

jens