[Zope-dev] Weird BTrees/ZODB interaction under Solaris

[Andreas Jung]

I made the following reproducable observation running Zope 2.5 under
Solaris:

I was indexing a CMF site using TextIndexNG with about 1600 objects.
During the commit phase (after all objects were indexed), the Zope
process grow from about 100MB to more than 1.2GB.

I could track this down to TextIndexNG. There is a function written
in C that inserts the list of splittet words into the BTrees
for the forward and reverse index (by generating the WordIDs
internally). Replacing this method with a pure Python implementation
made Zope behave fine.

Torturing the corresponding C implementation in an isolated environment
did not show any memory leaks (neither under Solaris nor under Linux
i386). To make it short: my C implementation has three args: the forward
index, the reverse index and a list of words to be inserted. The indexes
are an OIBTree and an IOBTree. Internally I call PyObject_SetItem()
to store the word-wid mapping. This seems to work fine isolated but
seems to blow up the ZODB for unknown reasons.

Note that his behaviour is only reproducable under Solaris. I made some
tests by indexing about 100.000 docs under Linux with TextIndexNG and
the Zope memory usage was very fine (up to 150MB).

Cheers,
Andreas


    ---------------------------------------------------------------------
   -    Andreas Jung                     http://www.andreas-jung.com   -
  -   EMail: andreas at andreas-jung.com                              -
   -            "Life is too short to (re)write parsers"               -
    ---------------------------------------------------------------------