[Zope] Options for handling concurrency?

[Martijn Pieters]

From: Tony McDonald [mailto:tony.mcdonald@ncl.ac.uk]
> At 9:43 am +0100 27/2/00, Martijn Pieters wrote:
> >
> >- Have a look at /Control_Panel/manage_debug. It'll show you 
> reference
> >counts and currenlty open connections, which can be very helpful in
> >targeting hotspots.
> 
> Aye carumba! - what a lot of information. I'm going to start delving 
> around there and see what happens. Is there any additional 
> documentation available on what the info. means? - I've looked in the 
> ApplicationManager.py code is a wee bit over my head.

It shows you reference counts, and how they change over time. Reference
counts are how many references there are to classes and methods. This
generally lets you see how many instances there are of those classes. It
allows you to see where the hotspots might be.

> 
> As an aside (to the list), looking around the ApplicationManager.py 
> reveals something called manage_profile. It looks like the main Zope 
> functions are being profiled here (ie not at the object level). It's 
> interesting, but it does seem to be a resource hog...

But so is something in your app. You might want to run it and see what
analysis you get. For more documentation of the data the profiler
returns, see:

  http://www.python.org/doc/lib/profile.html  

> 
> >- Also, what DBMS are you using? If you are using a level 1 
> or 2 DA (see
> >http://www.zope.org/Members/petrilli/DARoadmap) then you can 
> do only one DB
> >query at a time. It might be time to choose a different DB, 
> or if you are
> >using the old MySQL DA, to upgrade.
> 
> It's MySQL, but using the latest DA - I'm pretty sure I'm doing more 
> than one query at a time as I set off a ZSQL query that takes 9 
> seconds from the MySQL command line, and then started to use aspects 
> of the site that I *know* use MySQL lookups. BTW, the 9 second SQL 
> query isn't used in the site! :)
> 

This only shows that MySQL is multithreaded, not that the DA is. Run the
expensive, 9 second query in a ZSQL Method. Then try and access another
ZSQL Method on the same Zope server. If it is multithreaded, you should
get results about as fast as when the heavy one isn't running. If it is
the old MySQL DA, you'll have to wait until the 9 sec query is finished.

> >- If you are using ZSQL a lot, do you use caching? Even a 
> cache expiry of 1
> >second would help a  heavily hit site tremendously, noone 
> will notice the
> >data returned being at most 1 second out of date.But all 
> hits with equal
> >parameters on the same thread in Zope within that second 
> need only one DB
> >query.
> 
> I do use caching Martijn, but I must admit that it's been a hit and 
> miss affair. I'm trying to do as many of my SQL queries in external 
> methods (ironically for the performance), so I may well be losing a 
> lot of the caching performance breakthrough there .... hmmm. I wonder 
> if I could do something like...

This shouldn't make any difference. Only in doing excessive loops in
loops and the like, would an External Method be any use.

-- 
Martijn Pieters, Software Engineer 
| Digital Creations http://www.digicool.com 
| Creators of Zope      http://www.zope.org 
| mailto:mj@digicool.com       ICQ: 4532236
| PGP:
http://wwwkeys.nl.pgp.net:11371/pks/lookup?op=get&search=0xA8A32149 
-------------------------------------------