[Zope-dev] Zope/OpenBSD: Too many open files

[Ricardo Bermell]

Zope (started from a shell) in OpenBSD gets locked 
too easyly because of the low default limit for 
open file descriptors (64).   

When Zope gets locked shows repeatedly this message
"PROBLEM(100) ZServer warning: server accept() threw an exception",
locking effectively also the console.

Conditions in wich i have tested the problem:

- OpenBSD 2.7
- Zope 2.3.0 
- 'bash' shell

How to reproduce the problem:

1. Anotate your current limit for open files
   (after this test you will want to restore this value):
   type "ulimit -a | grep 'open files'"
   My systems gives "open files   64"

2. If you want to crash zope sooner, set a lower limit 
   (for example, 32) with "ulimit -n 32".

3. Start Zope from your shell (mine is 'bash') with 
   zope 'start' script.

   Before, in the call to z2.py (in 'start') include option 
   -D for 'debugging' (if not, all will be the same, exept 
   that you won't see the "Problem(100)" message

4. Type "fstat | grep run-zope | grep stream | wc -l",
  (where 'run-zope' is the user account you use to run zope)

  or "fstat | grep 11789 | grep stream | wc -l"
  (where '11789' is the pid of zope process)

  That gives you the number of open streams for zope,
  which initially is 1.

  Each open stream consumes one file descriptor.

5. From a browser of another computer, open any (one) page
   of your zope site (http://your.zope.server:8080, where 
   8080 is the port you've open for zope).

   That will open 2 tcp streams in your server. 
   Test, as before: "fstat | grep run-zope | grep stream | wc -l"
   This gives me 3 streams in total (1 initial + 2 for browser)

6. Close your browser and see how the open streams persists
   Type again: "fstat | grep run-zope | grep stream | wc -l"
   You will see 3 yet.

7. Repeat this operation many times (open browser, get http page,
   close browser), observing how the number of open streams increases 
   in steps of 2.

   After m repetitions your browser does not get response from Zope:
   your Zope service and your console stops attending requests.

8. Stop your Zope from another console. (This liberates also
   the locked console)

9. Test distinct values for 'ulimit -n'. You will observe that, 
   for a given value of 'ulimit -n' the number of repetitions (m) 
   needed to hang your zope will be always the same.

   (You must run 'ulimit -n' _before_ run zope for it to get effect,
   not _while_ running zope)


Questions which i'm not able to answer:

- How many time an open tcp stream lives after you open a page from
  your browser?

- How many time an open tcp stream lives after you close the browser
  (wich originated the stream)?

the answer to those two questions will help to know:

- How can i predict an adequate number for 'ulimit -n' (max. number of
  file/stream descriptors) in function of the utilization of my server?

Unknowing wich is the perfect election, i've selected a limit of 1024

- Is it too many 1024? Too many resources utilization? 

But, the main question, IMHO, is that:

- It is Zope robustness what fails. It should deny access gracefully when
  resources are exhausted. (Or at least, it should give comprehensive error
  information). Is it so? Or is it a wrong observation? 

I am now using zope only for developing static pages, not for a public 
dynamic service, and the 'ulimit' patch suffices, but i am afraid for 
when i'll want to put it in a public service with a heavy load.


-- Ricardo




P.S. I have tested also Zope 2.2.4 and 2.3.1b1 in OpenBSD. 
I didn't perform the 'ulimit' test in these versions but i do 
have experienced the "PROBLEM(100)" error (or similar).
I have tested Zope 2.3.1b1 and 2.3.0 thru Apache and standalone.
When thru Apache: with pcgi and with fastcgi.
When with fastgi: with port and with socket.