[Zope] CorruptTransactionError (Bad news for production site!)

Jim Fulton jim@digicool.com
Wed, 05 Jul 2000 13:40:28 -0400


Richard Taylor wrote:
> 
> Today I had to role back two days of transactions from my production
> site because when I packed the database I was informed of a
> CorruptTransactionError.

Did anything else happen previous to this? Did you run out of space
or anything like that?

You should have been able to use Data.fs.old, which is a copy
of the database before the pack to restore the data. Or was the
error in there too?  I'd be interested in looking at the Data.fs
file before the pack to try to figure out what might have gone wrong.

(If you send my your Data.fs file, please remember to send it
to me privately and to zip or compress it. :)

> We are using Zope for an internal knowledge management application
> where anyone in the organization can add objects. So I have no way of
> know what was added after the fateful transaction and no way of
> getting any of it back.
> 
> Bummer!

Indeed.
 
> I think this raises a few questions about ZDB:
> 
> 1) We need some tools for selectively removing bad transactions
>    rather than just truncating Data.fs back to the last good one and
>    loosing everything that comes after it.

Zope 2.2 has just such a tool. In the ZODB directory, there is a 
Python script, fsrecover.py which simply calls the recover function
in the FileStorage module. This will work with any 2.x databases.
It scans from both the beginning and the end of the file until 
it finds a corrupted section and then removes the corrupted portion
from the file.  You utility modifies the file in-place, so you need
to shut-down the site, or work on a copy when you use it.
 
> 2) We could do with a tools that can verify the ZODB offline. This
>    could then be run at regular intervals (maybe once an hour from
>    cron) so that corruptions can be picked up earlier.

You could use a little Python script that did something like:

  import ZODB.FileStorage
  file_name='../../var/Data.fs'
  file=open(file_name, 'r+b')
  index={}
  vindex={}
  tindex=[]
  ZODB.FileStorage.read_index(
        file, file_name, index, vindex, tindex)

This basically reads the FileStorage index as would normally
be done during startup.
 
> 3) Some way to find out what was added after a corrupt transaction is
>    needed so that at least I could see who I need to ask to re-add
>    their stuff.

The fsrecover script should avoid the need for this.
 
Jim

--
Jim Fulton           mailto:jim@digicool.com   Python Powered!        
Technical Director   (888) 344-4332            http://www.python.org  
Digital Creations    http://www.digicool.com   http://www.zope.org    

Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission.  Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
repeats.