[Zope] Multithreading issue - python, zope, opensbd, et al

[Paul Horbal]

Graham,

I'm no expert, by any means, but I was involved in a thread last year re:
Solaris/threading w/ Zope.

Firstly...

> First assumption - if multithreading was enabled, we would see a number of
> python/zope tasks running (ps, top, etc) when numerous simultaneous
requests
> come in... right? (as we do with apache - lots of httpd processes).

Processes != threads

One of the reasons Apache 1 has so many processes running is because it is
actually fairly poor at multithreading...  Ideally, you'd like to have fewer
processes than threads because there is usually more overhead associated
with creating a process than a thread.  Improved multithreading (the
developers are using phrases like "vastly improved")  is one of the main
benefits of Apache 2.

So, by extension, the number of processes you see running is not indicative
of Zope/python threading ...

IIRC, the way we tested Zope threading in the earlier discussion was by
having Zope run a Python method which is guaranteed to hog cycles and check
to see if the site still responded...  As luck would have it, I deleted the
test script about one week ago...  Luckily, the Internet never forgets and
the entire exchange is still here:
http://zope.nipltd.com/public/lists/zope-archive.nsf/Main/95DE5A56E970374480
256B14006FAC96?OpenDocument&mode=subject&topic=[Zope]BuildingZopew/SunForte(
ratherthangcc)

Another - probably better - test is to use the Python-supplied threading
test... You should find this in
/path/to/python/lib/python2.1/test/test_thread.py

Run the test simply by doing "python
/path/to/python/lib/python2.1/test/test_thread.py".  You should see output
like this:

creating task 1
creating task 2
...
creating task 10
waiting for all tasks to complete
task 1 will run for 2.1 sec
task 2 will run for 3.7 sec
...
task 10 will run for 7.1 sec
task 7 done
task 1 done
...
task 5 done
all tasks done

If you see the "task # done"'s come up out of numerical order (they should
complete in chronological order), this is a pretty good indication that
python threading is working fine.  Also, it's been noted that Solaris
threads work differently from Linux threads...  Linux will start threads
almost immediately, while Solaris seems to queue them up and start them at
some convenient time...


> We are using python/zope on both Solaris and openBSD, and in looking at
the
> detail log of Python configure on either, I see compile and then loader
> failures (different details for each OS though). Meanwhile the python
build
> process goes merrily along and creates a python evironment, unbeknown to
us
> at the time. (yes I know one shouls scour the logs, but time is always an
> issue...).

I can't speak for OpenBSD, but I have had a good experience using the binary
build of Zope for Solaris (it is built for 2.6, but I use it with 8 just
fine).  This comes with a pre-built/included python install - so you don't
have to mess with installing python yourself.

> So with openBSD, there seems a requirement for both pthread.h (found) and
> thread.h (not found). I assume with these that if they wre both found we
> might expect multithreading to work.

For OpenBSD, are you building Python from scratch?  Have you tried a
port/package?  These will usually address major problems like threads, if
any.  I would try to install an OpenBSD package version of Python (or ports,
though IIRC, OpenBSD doesn't have quite the ports system of the other BSDs),
run the threads test, then tackle a source build of Zope.


Hope this helps...
Paul.