[Zope] Need Caching advice for CMF site ...

[Lynn Walton]

Thanks to bak (kedai) for asking the question that led to this explanation as
I
definitely needed the additional info. Thanks Toby for explaining further.
Actually, we haven't moved to the 3 zope client set up yet so the numbers I
showed are for 1 zope client using the ZEO Server.  We didn't pass any -T
parameters so I guess we'd have the default of 4. (I searched for info on
publisher threads and came up empty .... so I wouldn't have known the default

was 4.)

I don't know if there is any relation between your guess that so many objects
in
cache is a sign of memory stress, and the fact that we are experiencing Zope
crashes 1-3 times per day (always associated with our higher load times). We
have been working for many weeks with ZC support to find the cause and so far

nothing we've tried has made any difference. (compiling without pymalloc,
restrictedPython patch, etc)

Our load on that site is probably around 70000 requests per day but we have
our
images being served up by Apache rather than in zope and so if you subtract
the
requests for images, I estimate no more than 20000 requests per day that zope
is
handling.  Most of those are CMF Documents that get rendered with a zpt
template
that does make an external method call but not one that is "highly
intensive".
In fact hardly any zope requests at this point are to anything "intensive".

If you have anymore insights based on the info I've now shared, please let me

know. I appreciate it.
Lynn


Toby Dickenson wrote:

> On Tuesday 22 January 2002 9:13 am, bak (kedai) wrote:
> >On Tuesday 22 January 2002 02:50 am, Toby Dickenson wrote:
> >> On Mon, 21 Jan 2002 12:20:41 -0600, Lynn Walton <waltonl@franklin.edu>
> >>
> >> >Total number of objects in the database             20518
> >> >Total number of objects in all caches                    2100
> >> >Target Size
> >> >400
> >> >Target max time between accesses                           60
> >> >
> >> >Can someone point me to understanding if this level of caching stops
the
> >> >client from making the trip to the ZODB on the ZEO Server.
> >>
> >> That stops Zope needing to talk to its storage (FileStorage or ZEO
> >> ClientStorage), then unpickle the result, for every object access.
> >>
> >> The http client is not involved in this level.
> >>
> >>
> >> ps interesting numbers. How many publisher threads do you have? (thats
> >> -T on the command line)
> >
> >what's interesting?  when i looked at thenumbers, it's like staring at the

> >wall :P
>
> The target size of 400 applies to each ZODB object cache. Each zope
instance
> has more than one cache; generally its one per publisher thread (-T), plus
a
> few more if you are using Sessions.
>
> The default -T value is 4. That is probably more than enough to support 3
> clients, so I guess Zope is only using 2 of those threads (exceptions would

> be if some of those clients are a high-concurrency system stress tool, or
if
> the site has some custom methods that take a long time to render)
>
> If Lynn is using that default and no sessions then each cache contains
> somewhere between 2100/4 = 525 objects (if all threads/caches are used) and

> 2100/2 = 1050 objects (following my guess that only 2 threads/caches are
> used).
>
> Either way, that is quite a bit more than the target of 400. The ZODB cache

> does behave that way, but only under reasonable memory pressure. I define
> memory pressure as objects touched per transaction (or http request). I am
> suprised that CMF involves sufficient memory pressure.
>
> I am currently looking at redesigning the ZODB cache to have a stiffer
> response to memory pressures.... hence my interest.