[ZWeb] Zope.org general performance

[Chris Withers]

Andrew Sawyers wrote:
>>> 1. Some search engine or other spider is ocassionally crawling some
>>> non-cached and very CPU intensive pages.
> 
> I definately think this is happening and was going to check the logs in 
> related to our load warnings.

There are load warning? Where do these go?
Where does Squid (if that's what does the master logging) keep its logs?

>> Hypthesis 3: it's the network.  Perhaps the load balancer or some 
>> other box is dropping packets.  I don't think I saw a delay when I 
>> accessed the site through an SSH tunnel (to reach a specific app server).
> 
> Not this, we didn't put anything behind a loadbalancer.  There are two 
> app servers behind a squid though; I have been doing a lot of squid fine 
> tuning for another project and I can take a look at this and see if 
> there are any areas I know will immediately improve the cahce performance.

That would be very cool, thanks!

>>> FWIW, I'm not sure how useful it really is to cache a site that should
>>> be as dynamic as Zope.org tothe extent we appear to have to to make it
>>> work fast enough. To me, it feels like there's something fundamentally
>>> wrong somewhere given that the load doesn't seem THAT high, and the site
>>> seems to be erratically crippled rather than generally bogged down :-/
> 
> I don't believe much caching happens - it's primarily images and files.

That's probably the way it should stay, unless zope.org really is 
getting 10's to 100's of hits a second...

> Casey and I were working on a problem recently that requires intensive 
> catalog queries to display pages, but often always only needed to 
> display the X recent items.  Today is the first day I can monitor the 
> performance results of this; but we expect it to be massive.  I would 
> expect Zope org could benefit from this type of index also -- it's been 
> released as OpenSource.

Cool, I have a few projects that would also love this. Where can I find it?

And, from a Zope.org point of view, what version of Zope does it 
require? :-S

Chris

-- 
Simplistix - Content Management, Zope & Python Consulting
            - http://www.simplistix.co.uk