[Zope] Problem with database packing (major data loss)

[Martijn Pieters]

On Fri, Jul 07, 2000 at 02:20:00PM +0100, Richard Taylor wrote:
> We have discovered a rather nasty problem. 
> 
> On Zope 2.1.6 if you you pack the database with '0' in the days box
> and then pack it again with '1' in the days box bad things happen. In
> our case the first pack resulted in a Data.fs of 17MBytes (not bad
> from a started point of nearly 2GBytes) the second pack (which I
> assumed would do nothing) resulted in a Data.fs of 3MBytes. Great I
> thought, however it also broke most of my site. Some of the symptoms
> are:
> 
> Accessing existing folders gives KeyError
> Many of my ZClasses are turned into DTML Methods (that is the icon
> next to them in the folder view is a DTML Method icon)
> All the instances of the broken classes are broken.
> 
>  
> Thankfully I kept backups before performing the packs.
> 
> This is repeatable on both the servers I run.
> 
> I am now nervous of packing but have no choice. 

This is a known problem, Jim Fulton checked a fix in several days ago. This
fix will be in 2.2.0b4, which should be out soon. The problem is in the second
pack, which packs to an earlier date than the previous pack.

When packing, Zope makes a backup of your Data.fs as well, renaming it
Data.fs.old. You can always put back this copy.

If you want to protect yourself from doing this again in 2.1.6, here is a
patch against lib/python/FileStorage.py:

--- FileStorage.py.orig	Fri Jul  7 16:29:34 2000
+++ FileStorage.py	Fri Jul  7 16:29:16 2000
@@ -862,6 +862,12 @@
 
         return r
 
+    def _redundant_pack(self, file, pos):
+        file.seek(pos-8)
+        p=u64(file.read(8))
+        file.seek(p+16)
+        return file.read(1) not in ' u'
+
     def pack(self, t, referencesf):
         """Copy data from the current database file to a packed file
     
@@ -900,6 +906,10 @@
             
             packpos, maxoid, ltid = read_index(
                 file, name, index, vindex, tindex, stop)
+
+            if self._redundant_pack(file, packpos):
+                raise FileStorageError, (
+                    'The database has already been packed to a later time')
     
             rootl=[z64]
             pop=rootl.pop

-- 
Martijn Pieters
| Software Engineer            mailto:mj@digicool.com
| Digital Creations          http://www.digicool.com/
| Creators of Zope               http://www.zope.org/
| ZopeStudio: http://www.zope.org/Products/ZopeStudio
-----------------------------------------------------