[Zope] Can't stop Zope, machine hanging

[Chris McDonough]

On Sep 13, 2006, at 1:57 PM, Ken Ara wrote:

> I have a new problem but it's related, so I'll keep
> this thread alive.
>
> To recap: possibly due to some network problems at my
> hosting provider, my Zope would periodically stop
> responding and I was unable to stop, restart or kill
> it, or even restart the machine remotely. I am
> operating within a FreeBSD jail. Finally, I was given
> a password to the main os environment, allowing me to
> restart, which I have needed to do once.
>
> While waiting for my ISP to resolve his problems, I
> thought it would be a good idea to relieve any excess
> strain on Zope. Part of the strategy involved setting
> Squid to ignore no-cache headers. Call me an
> anti-social protocol violator, but it's still my
> website and I set the caching headers according to the
> page update schedule. But I digress.
>
> Anyway, with response times improved, friendly search
> engines went into a feeding frenzy and within 48 hours
> my ZODB had reached 9GB. Packing time...


Ouch.  What on your site generates these writes?

>
> Now, perhaps due to the machine restart I get the
> following error when attempting to pack:
>
> Traceback (innermost last):
>   Module ZPublisher.Publish, line 101, in publish
>   Module ZPublisher.mapply, line 88, in mapply
>   Module ZPublisher.Publish, line 39, in call_object
>   Module App.ApplicationManager, line 428, in
> manage_pack
>   Module ZODB.DB, line 555, in pack
>   Module ZODB.FileStorage, line 1570, in pack
>   Module ZODB.fspack, line 700, in pack
>   Module ZODB.fspack, line 455, in findReachable
>   Module ZODB.fspack, line 475, in buildPackIndex
>   Module ZODB.fspack, line 166, in checkTxn
>   Module ZODB.fspack, line 161, in fail
> CorruptedError:
> /usr/local/www/Zope/zope01/var/Data.fs:3116771801:time-stamp
> reduction: 0x03681c50ebb292dd <= 0x03681c530eb31b99
>
> The bulk of the database was added following the
> restart so truncating may not be realistic. I haven't
> tried - was it fsrecover? - what is the current best
> tool or strategy?

Fsrecover is a good thing to try.  Note that I believe this error  
indicates that your system's clock "went back in time" significantly  
between two transactions.  You're using a fairly old version of Zope,  
so I don't know what provisions its ZODB has to protect you against  
this sort of thing; I do know that some work was put into dealing  
with this sort of case at some point.

- C