[Zope] Re: CST 0.9 Problems under load?

Tim Hicks tim@sitefusion.co.uk
Sun, 17 Feb 2002 23:34:34 -0000


----- Original Message -----
From: "Chris McDonough" <chrism@zope.com>
To: "Tim Hicks" <tim@sitefusion.co.uk>; <zope@zope.org>
Cc: "Paul Browning" <paul.browning@bristol.ac.uk>
Sent: Sunday, February 17, 2002 10:27 PM
Subject: Re: CST 0.9 Problems under load?


> Damn.  I wish I had an easy answer but I don't.

Damn ;-)

> Terry Kerr came up with an important bugfix for CST 0.9 lately, and I'm
sure
> if I dug in to my inbox I could find a few others that folks have
submitted.
> These may or may not solve your particular issues.  CST 0.9 seems to have
> some fundamental problems with supporting load that are proving very
> difficult to pin down because they are particularly subtle and only show
up
> in situations that are irreplicable.

I did a quick search on the nip list archive, but couldn't find Terry Kerr's
patch.  Where can I find it?  I'd like to give the patches a try.  The only
problem is, the server probably won't see the same kind of load until this
time next year.  Do you have any good ways of programmatically loading the
server and invoking the session machinery?  Is this where 'ab' comes in?

> I'm afraid for now that I'm going to have to suggest that you either wait
> for CST 1.0 (which has not ETA as of now) or try the Zope 2.5 Sessioning
> machinery.

OK, fair enough.  I realise that CST has turned into a bit of a side-project
for you now.  If I manage to reproduce the same sort of results using the
2.5.0 session machinery, will I stand a better chance of seeing a fix?  I'd
love to be able to help, but I'm well over my head at the moment.

cheers

tim

> Sorry,
>
> - C
>
> ----- Original Message -----
> From: "Tim Hicks" <tim@sitefusion.co.uk>
> To: <zope@zope.org>
> Cc: "Chris McDonough" <chrism@zope.com>; "Paul Browning"
> <paul.browning@bristol.ac.uk>
> Sent: Sunday, February 17, 2002 1:28 PM
> Subject: CST 0.9 Problems under load?
>
>
> > Zopers and Chris,
> >
> > I have written a little zope app that allows people to search an oracle
> > database thru-the-web.  Relevant details:
> >
> > Zope 2.3.3
> > Win NT
> > DC Oracle 1
> > CST 0.9 - used in just about every request as it provided shopping
basket
> > functionality
> > Session timeout set to 20 mins
> > Using ZServer (i.e. no apache/squid in front)
> > P2 450MHz
> > Can't remember how much RAM, but we didn't use it all.
> > [From Analog]
> > Day:  reqs:  pages:
> > Wed: 133586:  2458:
> > Thu:  48310:   831:
> > Fri:  24890:   438:
> >
> > We launched the site last Wednesday (10am), and the little box it was
> > running on was a bit overwhelmed.  We started to see KeyErrors relating
to
> > the session data manager (more on that below), so backed off to an
earlier
> > search app that had no CST component.  CPU had been running at 100%
almost
> > solidly for about half an hour, but we still had half our RAM free.  At
> this
> > stage, we weren't logging the errors.
> > After the load died down a bit (the first hour was far and away the
> worst),
> > we switched back to the CST'ified app.  Server load was lower, and
> although
> > we still weren't logging, we got the impression that the errors had
gone.
> > Sadly, the KeyErrors were spotted again.  At this point, I setup
> > standard_error_message to email me with the REQUEST and traceback
whenever
> > it caught a KeyError.  That began at 5pm Wednesday.  From then until 1pm
> > Friday, I received 174 error messages of the following form:
> >
> > -----
> > <snipped REQUEST>
> > Traceback (innermost last):
> >   File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 171,
in
> > publish
> >   File C:\PROGRA~1\zope233\lib\python\ZPublisher\mapply.py, line 160, in
> > mapply
> >     (Object: results)
> >   File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 112,
in
> > call_object
> >     (Object: results)
> >   File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 189, in
> > __call__
> >     (Object: results)
> >   File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py,
line
> > 540, in __call__
> >     (Object: results)
> >   File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 182, in
> > __call__
> >     (Object: standard_html_header)
> >   File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py,
line
> > 540, in __call__
> >     (Object: standard_html_header)
> >   File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_Util.py, line
> 339,
> > in eval
> >     (Object: sdm.getSessionData().has_key('your_results'))
> >     (Info: sdm)
> >   File <string>, line 0, in ?
> >   File
> >
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
> > er.py, line 264, in getSessionData
> >     (Object: sdm)
> >   File
> >
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
> > er.py, line 375, in _createSessionDataObject
> >     (Object: sdm)
> >   File
> >
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataConta
> > iner.py, line 191, in set
> >     (Object: /accom2/search/sdm)
> >   File
> >
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataConta
> > iner.py, line 344, in __setitem__
> >     (Object: /accom2/search/sdm)
> > KeyError: 64396498A0K2Wz9uuoA
> > -----
> > As far as I can tell, the only difference between each of the 174 errors
> is
> > a different key value (corresponding to different sessions I presume).
> >
> > I also received 4 of this sort:
> >
> > -----
> > Traceback (innermost last):
> >   File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 171,
in
> > publish
> >   File C:\PROGRA~1\zope233\lib\python\ZPublisher\mapply.py, line 160, in
> > mapply
> >     (Object: results)
> >   File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 112,
in
> > call_object
> >     (Object: results)
> >   File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 189, in
> > __call__
> >     (Object: results)
> >   File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py,
line
> > 540, in __call__
> >     (Object: results)
> >   File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 182, in
> > __call__
> >     (Object: standard_html_header)
> >   File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py,
line
> > 540, in __call__
> >     (Object: standard_html_header)
> >   File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_Util.py, line
> 339,
> > in eval
> >     (Object: sdm.getSessionData().has_key('your_results'))
> >     (Info: sdm)
> >   File <string>, line 0, in ?
> >   File
> >
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
> > er.py, line 258, in getSessionData
> >     (Object: sdm)
> >   File
> >
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
> > er.py, line 390, in _getSessionDataObject
> >     (Object: sdm)
> >   File C:\PROGRA~1\zope233\lib\python\ZODB\Connection.py, line 508, in
> > setstate
> >   File
> >
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionStorage.p
> > y, line 156, in load
> >     (Object: SessionStorage)
> > KeyError:
> > -----
> > <No key value given!?>
> >
> > As I said, the relevant REQUEST objects for each of these errors if they
> can
> > help shed any light on the problems I'm seeing... just ask.
> >
> > In addition, I also managed to salvage this traceback from about 10.15am
> on
> > Wednesday, when the CST site was getting hammered.  I only saw it once,
> but
> > wasn't logging at that stage...
> >
> > -----
> > Error Type: TypeError
> > Error Value: function requires exactly 1 argument; 2 given
> >
> > Traceback (innermost last):
> >   File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 223,
in
> > publish_module
> >   File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 187,
in
> > publish
> >   File C:\PROGRA~1\zope233\lib\python\Zope\__init__.py, line 221, in
> > zpublisher_exception_hook
> >     (Object: Traversable)
> >   File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 175,
in
> > publish
> >   File C:\PROGRA~1\zope233\lib\python\Zope\__init__.py, line 235, in
> commit
> >   File C:\PROGRA~1\zope233\lib\python\ZODB\Transaction.py, line 300, in
> > commit
> >   File C:\PROGRA~1\zope233\lib\python\ZODB\Connection.py, line 377, in
> > commit
> >     (Info: (('BTrees.OOBTree', 'OOBTree'),
> > '\000\000\000\000\000\000\000\020', ''))
> >   File
> >
>
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionStorage.p
> > y, line 182, in store
> >     (Object: SessionStorage)
> >   File C:\PROGRA~1\zope233\lib\python\ZODB\ConflictResolution.py, line
> 177,
> > in tryToResolveConflict
> >     (Object: SessionStorage)
> > TypeError: (see above)
> > -----
> >
> > I don't really no how to try to reproduce these errors short of getting
a
> > few thousand people to access the site at the same time again.  The CST
> site
> > is still up and in use under much lighter load.  I haven't received any
of
> > these errors since Friday lunch time, so it really does seem to be a
load
> > issue.
> >
> > I don't have much Zen, so to me it looks like the first error is CST not
> > being able to find (losing?) some session information.  The second one
is
> a
> > bit of a mystery to me, and the third looks like it might just be a bug
in
> > the way the CST code calls ConflictResolution.<some
> > class>.tryToResolveConflict()
> >
> > I'd really appreciate any enlightenment anyone could give about these
> > errors, and how to fix them.
> >
> > cheers
> >
> > tim
> >
> >
>