[Zope] Ever seen this problem?

[Sam Gendler]

"'Eric W. Sink'" wrote:

> Ian Blenke writes:
>
> > Something looks like it is attempting to walk your Zope site. The fact
> > that it checks for /robots.txt first shows that it is trying to play
> > nice. When it walks your site, it is clobbering your Zope server.
> >
> > Here is a sample robots.txt that you should use:
> >
> >       User-agent: *
> >       Disallow: /
> >
> > - Ian C. Blenke <icblenke@2c2.com>
>
> If your hypothesis is correct, then I concede that using such a
> robots.txt file would be a plausible workaround.  However:
>
> 1.  WHY would a Zope server go down simply because something is
> walking it?
>
> 2.  I have run wget -r on this server quite a bit, simulating a robot,
> walking everything on the site.  There has been no correlation between
> this activity and problems with the site.  I therefore assert that our
> Zope server does NOT go down simply because something is walking it.
>
> 3.  What if I want the crawlers to be able to walk my site?  Perhaps I
> want the search engines to be indexing our content.
>
> I appreciate the suggestion, but using a robots.txt file to prevent
> walking seems like fixing the symptom rather than the problem.
> Furthermore, I'm not even convinced that this would fix the symptom,
> because of my experiences mentioned in point 2, above.
>
> --

Actually, it would appear that something locked up some resources in Zope before
the last request for robots.txt, since robots.txt is the first file that fails
to enter the A and E stages of the transaction.  It would seem that the previous
requests failed to free up some thread or database lock or something.
Obviously, the python interpreter is still active or the B and I stages would be
not occur either.

--sam