[Zope3-checkins] CVS: Zope3/src/zope/fssync - README.txt:1.9.2.1 fsmerger.py:1.2.2.1 fssync.py:1.24.2.1 main.py:1.15.2.1 merger.py:1.8.2.1 metadata.py:1.2.2.1 snarf.py:1.1.2.1 compare.py:NONE

Grégoire Weber zope@i-con.ch
Sun, 22 Jun 2003 10:24:13 -0400


Update of /cvs-repository/Zope3/src/zope/fssync
In directory cvs.zope.org:/tmp/cvs-serv24874/src/zope/fssync

Modified Files:
      Tag: cw-mail-branch
	README.txt fsmerger.py fssync.py main.py merger.py metadata.py 
	snarf.py 
Removed Files:
      Tag: cw-mail-branch
	compare.py 
Log Message:
Synced up with HEAD

=== Zope3/src/zope/fssync/README.txt 1.9 => 1.9.2.1 ===
--- Zope3/src/zope/fssync/README.txt:1.9	Fri May 16 08:52:28 2003
+++ Zope3/src/zope/fssync/README.txt	Sun Jun 22 10:23:42 2003
@@ -8,6 +8,13 @@
 This version is based loosely on a prototype written by Jim Fulton and
 Deb Hazarika.  It is now maintained by Guido van Rossum.
 
+Possibly also relevant background:
+
+  http://dev.zope.org/Zope3/ThroughTheWebSiteDevelopment
+
+The "bundles" mentioned there are likely candidates for filesystem
+synchronization.  (See section "Working with bundles" below.)
+
 
 User stories
 ------------
@@ -50,49 +57,68 @@
   status, and the simplest form of diff) must be performed entirely
   offline.
 
+* An interesting possibility: you could couple your filesystem copy to
+  a revision control system like CVS or Subversion, to have an
+  auditable revision history of a site.  Typically, you'd do a cvs
+  commit after each sync update and after each sync commit, after
+  verifying that the state committed to Zope actually works.  It would
+  be handy if files added to or removed from Zope are automatically
+  added or removed from CVS.  The "binary" flag for CVS might be set
+  automatically based on the Zope object type.
+
+* Another possibility: export and import (a la Zope 2 export/import)
+  should be easily implemented on top of this.  Export would be done
+  with checkout; import could be a new "checkin" command.  (This is
+  now implemented.)
+
+* And last but not least, this will form the basis of bundles; see the
+  ThroughTheWebSiteDevelopment reference above.
+
 
 BUGS
 ----
 
-* Sometimes when committing additions or removals, the Entries.xml
-  file doesn't get updated properly.
+* When committing an added file, you must commit the directory
+  containing it; you can't commit the file itself, since the command
+  tries to send the request to a view of the corresponding object,
+  which doesn't exist yet.
+
+* When doing an update, somehow the absolute pathnames of all files
+  are reported rather than the nice relative names.
 
 
 TO DO
 -----
 
-* Rewrite fromFS to integrate uptodate checking; because the db is
-  transactional it's ok to have made some changes and later raise an
-  exception.  Then it could also update the disk copy in-place to
-  reflect changes, ready to be zipped and sent back.
-
-* Don't rely on external zip/unzip tools.  Maybe switch to tar as the
-  archival format, because it is easier to stream and compress at the
-  same time.  The file can be probably streamed right to the socket,
-  without going to a temp file first, assuming the receiver can handle
-  not having a Content-length header.
-
-* more unit tests for fsmerge, to check that entries are handled
-  correctly in all cases (including addition/deletion/change of
-  type).
-
-* unit tests for the fssync core functionality
-
-* more refactoring and cleanup of the fssync core functionality
-
-* more diff options:
-  -2 diffs between local and remote
-  -3 diffs between original and remote
-  -N shows diffs for added/removed files as diffs with /dev/null
-  more GNU diff options?  e.g. --ignore-space-change etc.
+- Implement bundle commands.
+
+- On the server side:
+
+  * Nothing ATM.
+
+- In the sync application:
+
+  * Implement diff using difflib.
 
-* something akin to cvs -n update, which shows what update would do
-  without actually doing it
+  * More diff options:
+    -2 diffs between local and remote
+    -3 diffs between original and remote
+    -N shows diffs for added/removed files as diffs with /dev/null
 
-* commit shouldn't commit new versions of unchanged objects to ZODB
+  * More GNU diff options?  e.g. --ignore-space-change etc.
 
-* refine the adapter protocol or implementation to leverage the
-  file-system representation protocol
+  * Something akin to cvs -n update, which shows what update would do
+    without actually doing it.
+
+- Code maintenance:
+
+  * Rewrite toFS() to use the Metadata class, and add unit tests.
+
+  * Unit tests for the fssync utility.
+
+  * More refactoring and cleanup of the fssync utility.
+
+  * Use camelCase for public method names.
 
 
 TO DO LATER
@@ -100,12 +126,23 @@
 
 * Work out security details.
 
+* A commit unpickles user-provided data.  Unpickling is not a safe
+  operation.  Possible solution: have an unpickler that finds globals
+  in a secure way.  Use an import on a security proxy for sys.modules.
+
+* The adapters returned by the fs registry should optionally have
+  a permission associated with them.  If you have an adapter that
+  calls removeAllProxies, the adapter should require a permission.
+
+* Refine the fssync adapter protocol or implementation to leverage the
+  file-system representation (== FTP, WebDAV) protocol.
+
 * In common case where extra data are simple values, store extra data
   in the entries file to simplify representation and updates.  Maybe
   do something similar w annotations.
 
 * Maybe do some more xmlpickle refinement with an eye toward
-  impproving the usability of simple dictionary pickles.
+  improving the usability of simple dictionary pickles.
 
 * Maybe leverage adaptable storage ideas to assure losslessness.
 
@@ -116,3 +153,147 @@
 * Commit to multiple Zope instances?
 
 * Diff/merge multiple working sets (a la bitkeeper)?
+
+
+Working with bundles
+--------------------
+
+- Bundles aren't quite as easy to use as they are supposed to be as
+  described in the ThroughTheWebSiteDevelopment Wiki page referenced
+  above, but you can do some basic bundle-ish things.  All these need
+  is a little better packaging.
+
+- The fssync command.  Below, examples use a command named fssync.
+  This doesn't yet exist.  Best is to have a shell alias that points
+  to the file <Zope3>/src/fssync/main.py, where <Zope3> is the root of
+  the Zope3 tree.
+
+- Permissions.  Everything described here requires the
+  zope.ManageServices permission, which usually requires being logged
+  in with the manager role.
+
+- Bundle status.  There is not yet an explicit notion of "bundle-ness"
+  for site management folders.  Any site management folder can be
+  treated as a bundle.  Exception: the Bundle view works for the
+  default folder but the form included in the view refuses to change
+  it.  This is a safety measure: the bundle form can do a lot of
+  damage, e.g. it can disable all services at once.  By convention, I
+  propose that bundles have a folder name of the form
+  <name>-<version>, where <version> is two or more decimal numbers
+  separated by dots and <name> is unconstrained.
+
+- Creating a bundle.  There is no specific command to create a
+  bundle.  Instead, you create a new site management folder by going
+  to the Contents view of the site (e.g. /++etc++site/@@contents.html)
+  and clicking on "Add" in the actions menu.  A box will appear in
+  which you should type the name + version of your bundle.  Then in
+  that bundle you should create the things that you want to go into
+  the bundle, e.g. modules, templates, services, utilities, etc.
+
+- Creating a bundle from an existing folder.  If you have some
+  existing work done in the default folder or another non-bundle
+  folder, you can save your work to the filesystem using the fssync
+  checkout command, and then check it in under a different name using
+  the fssync checkin command.  Example; replace u:p with your manager
+  username and password:
+
+  $ fssync checkout http://u:p@localhost:8080/++etc++site/default
+  <lots of output>
+  All done.
+  $ fssync checkin http://u:p@localhost:8080/++etc++site/bundle-1.0 default
+  $
+
+  Now go back to your web browser and check out the contents of
+  /++etc++site/; a new folder bundle-1.0 should exist, containing a
+  copy of the default folder.  You should delete unnecessary things;
+  especially the standard service definitions are not needed.
+
+- Exporting a bundle.  First deactivate the bundle bu using the
+  "Deactivate bundle" button on the Bundle tab (see below).  Then save
+  the bundle to the filesystem using fssync checkout.  Finally tar or
+  zip it up.  Make sure to include the @@Zope directory at the same
+  level as the bundle directory in the archive.  Example:
+
+  $ fssync checkout http://u:p@localhost:8080/++etc++site/bundle-1.0
+  <lots of output>
+  All done.
+  $ tar tf - bundle-1.0 @@Zope | gzip >bundle-1.0.tgz
+  $
+
+  Now distribute the gzipped tar file file via the web.
+
+- Importing a bundle.  First extract the zip or tar file to the
+  filesystem.  Then use fssync checkin command to add it to your Zope
+  server.  Warning: the checkin command will happily overwrite an
+  existing site management folder!
+
+- Activating a bundle.  An imported bundle is completely inactive.
+  Its configuration records (the objects in the bundle's
+  RegistrationManager subfolder, and in RegistrationManager subfolders
+  of subfolders of the bundle) are not registered with their
+  respective services.  To activate the bundle, navigate your web
+  browser to its contents and select the Bundle tab; it is probably
+  the second tab from the left.  The Bundle tab displays two sections
+  and a few buttons.
+
+  - Section one of the Bundle tab shows the services needed by the
+    bundle.  This list is created by inspection of the bundle's
+    configuration records: for example, if there is a configuration
+    record for a utility, the service needs the utility service.
+    For each needed service, there are three possibilities:
+
+    1) The service is already active in the site.  This is probably
+       because it exists in the default folder or in a previous
+       bundle.
+
+    2) The service is not yet active in the site but the bundle
+       provides a configuration for the service.
+
+    3) No usable definition of the service can be found.  Note that a
+       service active in a parent site cannot be used.  This is called
+       an unfulfilled dependency.  This means that the bundle cannot
+       be activated.  A helpful link to the "Add service" view of the
+       default folder is provided, where you can create (and
+       activate!) the service and then navigate back to the bundle;
+       but you may also import the service as part of another bundle.
+
+  - Section two of the Bundle tab shows, for each of the service types
+    shown in section one, all configuration records in the bundle for
+    that service type.  Initially, all configurations are in the
+    "Unregistered" state.  At the bottom of the list you will find a
+    button which will register all configurations, and activate the
+    ones that aren't in conflict with pre-existing registrations.
+    Conflicts are indicated in red and provide a link to the
+    conflicting active configuration record, probably in another
+    bundle.  The automatic resolution of conflicts in favor of a newer
+    version of the same bundle, mentioned in the Wiki, is not yet
+    implemented; by default, whenever there is a conflict the
+    conflicting configuration record in the bundle is not activated.
+    You can resolve conflicts yourself in favor of the new bundle by
+    clicking the radio button labeled "Register and activate".  You
+    can also leave a configuration record inactive by clicking the
+    radio button "Register only".  When you are satisfied with the
+    selections, click the "Activate bundle" button below the list to
+    register and activate the bundle's configuration records; this
+    performs the actions selected by the radio buttons.  If you later
+    change your mind, you can always go back to the Bundle tab and
+    change your selections.
+
+  - At the very bottom of the page is a button labeled "Deactivate
+    bundle".  This is used for uninstalling a bundle; it makes all
+    configuration records contained in the bundle inactive and
+    unregistered.  It is also used for exporting a bundle; before you
+    export a bundle, you should deactivate it (see above).  In
+    contrast to the description in the Wiki, deactivating a bundle
+    does not reactivate any configuration records that were active
+    before the bundle was activated, because the configuration
+    registries don't record this information; it can't distinguish
+    between previously active and previously registered.  A redesign
+    of the registries would be necessary to accommodate this feature.
+
+- To delete a deactivated bundle, go to the site manager's contents
+  display (/++etc++site/@@contents.html), select the checkbox in front
+  of the bundle name, and click the Delete button below the list.
+  Deleting an active bundle usually doesn't work because of the
+  dependencies between the configuration records and the configured
+  objects in the bundle.


=== Zope3/src/zope/fssync/fsmerger.py 1.2 => 1.2.2.1 ===
--- Zope3/src/zope/fssync/fsmerger.py:1.2	Thu May 15 11:32:23 2003
+++ Zope3/src/zope/fssync/fsmerger.py	Sun Jun 22 10:23:42 2003
@@ -17,10 +17,13 @@
 """
 
 import os
+import shutil
 
 from os.path import exists, isfile, isdir, split, join
 from os.path import realpath, normcase, normpath
 
+from zope.xmlpickle import dumps
+
 from zope.fssync.merger import Merger
 from zope.fssync import fsutil
 
@@ -53,19 +56,49 @@
             self.reporter("XXX %s" % local)
         self.merge_extra(local, remote)
         self.merge_annotations(local, remote)
+        if not exists(local) and not self.metadata.getentry(local):
+            self.remove_special(local, "Extra")
+            self.remove_special(local, "Annotations")
+            self.remove_special(local, "Original")
 
     def merge_extra(self, local, remote):
+        """Helper to merge the Extra trees."""
         lextra = fsutil.getextra(local)
         rextra = fsutil.getextra(remote)
         self.merge_dirs(lextra, rextra)
 
     def merge_annotations(self, local, remote):
+        """Helper to merge the Anotations trees."""
         lannotations = fsutil.getannotations(local)
         rannotations = fsutil.getannotations(remote)
         self.merge_dirs(lannotations, rannotations)
 
+    def remove_special(self, local, what):
+        """Helper to remove an Original, Extra or Annotations file/tree."""
+        head, tail = fsutil.split(local)
+        dir = join(head, "@@Zope", what)
+        target = join(dir, tail)
+        if exists(target):
+            if isdir(target):
+                shutil.rmtree(target)
+            else:
+                os.remove(target)
+        if isdir(dir):
+            try:
+                os.rmdir(dir)
+            except os.error:
+                pass
+
     def merge_files(self, local, remote):
         """Merge remote file into local file."""
+
+        # Reset sticky conflict if file was edited or removed
+        entry = self.metadata.getentry(local)
+        conflict = entry.get("conflict")
+        if conflict and (not os.path.exists(local) or
+                         conflict != os.path.getmtime(local)):
+            del entry["conflict"]
+
         original = fsutil.getoriginal(local)
         action, state = self.merger.classify_files(local, original, remote)
         state = self.merger.merge_files(local, original, remote,
@@ -74,12 +107,12 @@
 
     def merge_dirs(self, localdir, remotedir):
         """Merge remote directory into local directory."""
-        lnames = self.metadata.getnames(localdir)
-        rnames = self.metadata.getnames(remotedir)
+        lentrynames = self.metadata.getnames(localdir)
+        rentrynames = self.metadata.getnames(remotedir)
         lentry = self.metadata.getentry(localdir)
         rentry = self.metadata.getentry(remotedir)
 
-        if not lnames and not rnames:
+        if not lentrynames and not rentrynames:
 
             if not lentry:
                 if not rentry:
@@ -87,9 +120,11 @@
                         self.reportdir("?", localdir)
                 else:
                     if not exists(localdir):
-                        fsutil.ensuredir(localdir)
+                        self.make_dir(localdir)
+                        lentry.update(rentry)
                         self.reportdir("N", localdir)
                     else:
+                        self.make_dir(localdir)
                         self.reportdir("*", localdir)
                 return
 
@@ -110,31 +145,51 @@
                 return
 
             if not rentry:
-                try:
-                    os.rmdir(localdir)
-                except os.error:
-                    pass
-                self.reportdir("D", localdir)
-                lentry.clear()
+                self.clear_dir(localdir)
                 return
 
         if exists(localdir):
-            self.reportdir("/", localdir)
+            if lentry.get("flag") == "added":
+                if exists(remotedir):
+                    self.reportdir("U", localdir)
+                    del lentry["flag"]
+                else:
+                    self.reportdir("A", localdir)
+            else:
+                if rentry or exists(remotedir):
+                    self.reportdir("/", localdir)
+                else:
+                    # Tree removed remotely, must recurse down locally
+                    for name in lentrynames:
+                        self.merge(join(localdir, name), join(remotedir, name))
+                    self.clear_dir(localdir)
+                    return
+
             lnames = dict([(normcase(name), name)
                            for name in os.listdir(localdir)])
         else:
-            if lentry.get("flag") != "removed" and (rentry or rnames):
-                fsutil.ensuredir(localdir)
+            flag = lentry.get("flag")
+            if flag == "removed":
+                self.reportdir("R", localdir)
+                return # There's no point in recursing down!
+            if rentry or rentrynames:
+                self.make_dir(localdir)
                 lentry.update(rentry)
                 self.reportdir("N", localdir)
             lnames = {}
 
+        for name in lentrynames:
+            lnames[normcase(name)] = name
+
         if exists(remotedir):
             rnames = dict([(normcase(name), name)
                            for name in os.listdir(remotedir)])
         else:
             rnames = {}
 
+        for name in rentrynames:
+            rnames[normcase(name)] = name
+
         names = {}
         names.update(lnames)
         names.update(rnames)
@@ -146,6 +201,45 @@
         for ncname in ncnames:
             name = names[ncname]
             self.merge(join(localdir, name), join(remotedir, name))
+
+    def make_dir(self, localdir):
+        """Helper to create a local directory.
+
+        This also creates the @@Zope subdirectory and places an empty
+        Entries.xml file in it.
+        """
+        fsutil.ensuredir(localdir)
+        localzopedir = join(localdir, "@@Zope")
+        fsutil.ensuredir(localzopedir)
+        efile = join(localzopedir, "Entries.xml")
+        if not os.path.exists(efile):
+            data = dumps({})
+            f = open(efile, "w")
+            try:
+                f.write(data)
+            finally:
+                f.close()
+
+    def clear_dir(self, localdir):
+        """Helper to get rid of a local directory.
+
+        This zaps the directory's @@Zope subdirectory, but not other
+        files/directories that might still exist.
+
+        It doesn't deal with extras and annotations for the directory
+        itself, though.
+        """
+        lentry = self.metadata.getentry(localdir)
+        lentry.clear()
+        localzopedir = join(localdir, "@@Zope")
+        if os.path.isdir(localzopedir):
+            shutil.rmtree(localzopedir)
+        try:
+            os.rmdir(localdir)
+        except os.error:
+            self.reportdir("?", localdir)
+        else:
+            self.reportdir("D", localdir)
 
     def reportdir(self, letter, localdir):
         """Helper to report something for a directory.


=== Zope3/src/zope/fssync/fssync.py 1.24 => 1.24.2.1 ===
--- Zope3/src/zope/fssync/fssync.py:1.24	Tue May 20 15:09:15 2003
+++ Zope3/src/zope/fssync/fssync.py	Sun Jun 22 10:23:42 2003
@@ -27,7 +27,6 @@
 import filecmp
 import htmllib
 import httplib
-import commands
 import tempfile
 import urlparse
 import formatter
@@ -38,6 +37,8 @@
 from os.path import dirname, basename, split, join
 from os.path import realpath, normcase, normpath
 
+from zope.xmlpickle import dumps
+
 from zope.fssync.metadata import Metadata
 from zope.fssync.fsmerger import FSMerger
 from zope.fssync.fsutil import Error
@@ -158,20 +159,22 @@
         finally:
             f.close()
 
-    def httpreq(self, path, view, datafp=None,
-                content_type="application/x-snarf"):
+    def httpreq(self, path, view, datasource=None,
+                content_type="application/x-snarf",
+                expected_type="application/x-snarf"):
         """Issue an HTTP or HTTPS request.
 
         The request parameters are taken from the root url, except
         that the requested path is constructed by concatenating the
         path and view arguments.
 
-        If the optional 'datafp' argument is not None, it should be a
-        seekable stream from which the input document for the request
-        is taken.  In this case, a POST request is issued, and the
-        content-type header is set to the 'content_type' argument,
-        defaulting to 'application/x-snarf'.  Otherwise (if datafp is
-        None), a GET request is issued and no input document is sent.
+        If the optional 'datasource' argument is not None, it should
+        be a callable with a stream argument which, when called,
+        writes data to the stream.  In this case, a POST request is
+        issued, and the content-type header is set to the
+        'content_type' argument, defaulting to 'application/x-snarf'.
+        Otherwise (if datasource is None), a GET request is issued and
+        no input document is sent.
 
         If the request succeeds and returns a document whose
         content-type is 'application/x-snarf', the return value is a tuple
@@ -196,42 +199,33 @@
             path += "/"
         path += view
         if self.roottype == "https":
-            h = httplib.HTTPS(self.host_port)
+            conn = httplib.HTTPSConnection(self.host_port)
         else:
-            h = httplib.HTTP(self.host_port)
-        if datafp is None:
-            h.putrequest("GET", path)
-            filesize = 0   # for PyChecker
+            conn = httplib.HTTPConnection(self.host_port)
+        if datasource is None:
+            conn.putrequest("GET", path)
         else:
-            datafp.seek(0, 2)
-            filesize = datafp.tell()
-            datafp.seek(0)
-            h.putrequest("POST", path)
-            h.putheader("Content-type", content_type)
-            h.putheader("Content-length", str(filesize))
+            conn.putrequest("POST", path)
+            conn.putheader("Content-type", content_type)
+            conn.putheader("Transfer-encoding", "chunked")
         if self.user_passwd:
             auth = base64.encodestring(self.user_passwd).strip()
-            h.putheader('Authorization', 'Basic %s' % auth)
-        h.putheader("Host", self.host_port)
-        h.endheaders()
-        if datafp is not None:
-            nbytes = 0
-            while True:
-                buf = datafp.read(8192)
-                if not buf:
-                    break
-                nbytes += len(buf)
-                h.send(buf)
-            assert nbytes == filesize
-        errcode, errmsg, headers = h.getreply()
-        fp = h.getfile()
-        if errcode != 200:
+            conn.putheader('Authorization', 'Basic %s' % auth)
+        conn.putheader("Host", self.host_port)
+        conn.putheader("Connection", "close")
+        conn.endheaders()
+        if datasource is not None:
+            datasource(PretendStream(conn))
+            conn.send("0\r\n\r\n")
+        response = conn.getresponse()
+        if response.status != 200:
             raise Error("HTTP error %s (%s); error document:\n%s",
-                        errcode, errmsg,
-                        self.slurptext(fp, headers))
-        if headers["Content-type"] != "application/x-snarf":
-            raise Error(self.slurptext(fp, headers))
-        return fp, headers
+                        response.status, response.reason,
+                        self.slurptext(response.fp, response.msg))
+        elif expected_type and response.msg["Content-type"] != expected_type:
+            raise Error(self.slurptext(response.fp, response.msg))
+        else:
+            return response.fp, response.msg
 
     def slurptext(self, fp, headers):
         """Helper to read the result document.
@@ -253,6 +247,30 @@
             return data.strip()
         return "Content-type: %s" % ctype
 
+class PretendStream(object):
+
+    """Helper class to turn writes into chunked sends."""
+
+    def __init__(self, conn):
+        self.conn = conn
+
+    def write(self, s):
+        self.conn.send("%x\r\n" % len(s))
+        self.conn.send(s)
+
+class DataSource(object):
+
+    """Helper class to provide a data source for httpreq."""
+
+    def __init__(self, head, tail):
+        self.head = head
+        self.tail = tail
+
+    def __call__(self, f):
+        snf = Snarfer(f)
+        snf.add(join(self.head, self.tail), self.tail)
+        snf.addtree(join(self.head, "@@Zope"), "@@Zope/")
+
 class FSSync(object):
 
     def __init__(self, metadata=None, network=None, rooturl=None):
@@ -298,36 +316,49 @@
                     for name in names:
                         method(join(target, name), *more)
 
-    def commit(self, target, note="fssync"):
+    def commit(self, target, note="fssync_commit", raise_on_conflicts=False):
         entry = self.metadata.getentry(target)
         if not entry:
             raise Error("nothing known about", target)
         self.network.loadrooturl(target)
         path = entry["path"]
-        snarffile = tempfile.mktemp(".snf")
+        view = "@@fromFS.snarf?note=%s" % urllib.quote(note)
+        if raise_on_conflicts:
+            view += "&raise=1"
         head, tail = split(realpath(target))
+        data = DataSource(head, tail)
+        fp, headers = self.network.httpreq(path, view, data)
         try:
-            f = open(snarffile, "wb")
-            try:
-                snf = Snarfer(f)
-                snf.add(join(head, tail), tail)
-                snf.addtree(join(head, "@@Zope"), "@@Zope/")
-            finally:
-                f.close()
-            infp = open(snarffile, "rb")
-            view = "@@fromFS.snarf?note=%s" % urllib.quote(note)
-            try:
-                outfp, headers = self.network.httpreq(path, view, infp)
-            finally:
-                infp.close()
-        finally:
-            pass
-            if isfile(snarffile):
-                os.remove(snarffile)
-        try:
-            self.merge_snarffile(outfp, head, tail)
+            self.merge_snarffile(fp, head, tail)
         finally:
-            outfp.close()
+            fp.close()
+
+    def checkin(self, target, note="fssync_checkin"):
+        rootpath = self.network.rootpath
+        if not rootpath:
+            raise Error("root url not set")
+        if rootpath == "/":
+            raise Error("root url should name an inferior object")
+        i = rootpath.rfind("/")
+        path, name = rootpath[:i], rootpath[i+1:]
+        if not path:
+            path = "/"
+        if not name:
+            raise Error("root url should not end in '/'")
+        entry = self.metadata.getentry(target)
+        if not entry:
+            raise Error("nothing known about", target)
+        qnote = urllib.quote(note)
+        qname = urllib.quote(name)
+        head, tail = split(realpath(target))
+        qsrc = urllib.quote(tail)
+        view = "@@checkin.snarf?note=%s&name=%s&src=%s" % (qnote, qname, qsrc)
+        data = DataSource(head, tail)
+        fp, headers = self.network.httpreq(path, view, data,
+                                           expected_type=None)
+        message = self.network.slurptext(fp, headers)
+        if message:
+            print message
 
     def update(self, target):
         entry = self.metadata.getentry(target)
@@ -378,8 +409,8 @@
             return
         print "Index:", target
         sys.stdout.flush()
-        os.system("diff %s %s %s" %
-                  (diffopts, commands.mkarg(orig), commands.mkarg(target)))
+        cmd = ("diff %s %s %s" % (diffopts, quote(orig), quote(target)))
+        os.system(cmd)
 
     def dirdiff(self, target, mode=1, diffopts=""):
         assert isdir(target)
@@ -390,7 +421,7 @@
             if e and "flag" not in e:
                 self.diff(t, mode, diffopts)
 
-    def add(self, path):
+    def add(self, path, type=None, factory=None):
         if not exists(path):
             raise Error("nothing known about '%s'", path)
         entry = self.metadata.getentry(path)
@@ -408,9 +439,22 @@
         zpath += tail
         entry["path"] = zpath
         entry["flag"] = "added"
-        if isdir(path):
-            entry["type"] = entry["factory"] = "zope.app.content.folder.Folder"
+        if type:
+            entry["type"] = type
+        if factory:
+            entry["factory"] = factory
         self.metadata.flush()
+        if isdir(path):
+            # Force Entries.xml to exist, even if it wouldn't normally
+            zopedir = join(path, "@@Zope")
+            efile = join(zopedir, "Entries.xml")
+            if not exists(efile):
+                if not exists(zopedir):
+                    os.makedirs(zopedir)
+                    self.network.writefile(dumps({}), efile)
+            print "A", join(path, "")
+        else:
+            print "A", path
 
     def remove(self, path):
         if exists(path):
@@ -426,6 +470,7 @@
         else:
             entry["flag"] = "removed"
         self.metadata.flush()
+        print "R", path
 
     def status(self, target, descend_only=False):
         entry = self.metadata.getentry(target)
@@ -495,3 +540,23 @@
         extra = fsutil.getextra(target)
         if isdir(extra):
             self.status(extra, True)
+
+def quote(s):
+    """Helper to put quotes around arguments passed to shell if necessary."""
+    if os.name == "posix":
+        meta = "\\\"'*?[&|()<>`#$; \t\n"
+    else:
+        meta = " "
+    needquotes = False
+    for c in meta:
+        if c in s:
+            needquotes = True
+            break
+    if needquotes:
+        if os.name == "posix":
+            # use ' to quote, replace ' by '"'"'
+            s = "'" + s.replace("'", "'\"'\"'") + "'"
+        else:
+            # (Windows) use " to quote, replace " by ""
+            s = '"' + s.replace('"', '""') + '"'
+    return s


=== Zope3/src/zope/fssync/main.py 1.15 => 1.15.2.1 ===
--- Zope3/src/zope/fssync/main.py:1.15	Thu May 15 18:22:58 2003
+++ Zope3/src/zope/fssync/main.py	Sun Jun 22 10:23:42 2003
@@ -23,6 +23,7 @@
 fssync [global_options] status [local_options] [TARGET ...]
 fssync [global_options] add [local_options] TARGET ...
 fssync [global_options] remove [local_options] TARGET ...
+fssync [global_options] checkin [local_options] URL [TARGETDIR]
 
 ``fssync -h'' prints the global help (this message)
 ``fssync command -h'' prints the local help for the command
@@ -38,7 +39,7 @@
 from os.path import dirname, join, realpath
 
 # Find the zope root directory.
-# XXX This assumes this script is <root>/src/zope/fssync/sync.py
+# XXX This assumes this script is <root>/src/zope/fssync/main.py
 scriptfile = sys.argv[0]
 scriptdir = realpath(dirname(scriptfile))
 rootdir = dirname(dirname(dirname(scriptdir)))
@@ -158,7 +159,7 @@
     fs.checkout(target)
 
 def commit(opts, args):
-    """fssync commit [-m message] [TARGET ...]
+    """fssync commit [-m message] [-r] [TARGET ...]
 
     Commit the TARGET files or directories to the Zope 3 server
     identified by the checkout command.  TARGET defaults to the
@@ -171,12 +172,15 @@
     The -m option specifies a message to label the transaction.
     The default message is 'fssync'.
     """
-    message = "fssync"
+    message = "fssync_commit"
+    raise_on_conflicts = False
     for o, a in opts:
         if o in ("-m", "--message"):
             message = a
+        if o in ("-r", "--raise-on-conflicts"):
+            raise_on_conflicts = True
     fs = FSSync()
-    fs.multiple(args, fs.commit, message)
+    fs.multiple(args, fs.commit, message, raise_on_conflicts)
 
 def update(opts, args):
     """fssync update [TARGET ...]
@@ -193,15 +197,35 @@
     fs.multiple(args, fs.update)
 
 def add(opts, args):
-    """fssync add TARGET ...
+    """fssync add [-t TYPE] [-f FACTORY] TARGET ...
 
     Add the TARGET files or directories to the set of registered
     objects.  Each TARGET must exist.  The next commit will add them
     to the Zope 3 server.
+
+    The options -t and -f can be used to set the type and factory of
+    the newly created object; these should be dotted names of Python
+    objects.  Usually only the factory needs to be specified.
+
+    If no factory is specified, the type will be guessed when the
+    object is inserted into the Zope 3 server based on the filename
+    extension and the contents of the data.  For example, some common
+    image types are recognized by their contents, and the extensions
+    .pt and .dtml are used to create page templates and DTML
+    templates, respectively.
     """
+    type = None
+    factory = None
+    for o, a in opts:
+        if o in ("-t", "--type"):
+            type = a
+        elif o in ("-f", "--factory"):
+            factory = a
+    if not args:
+        raise Usage("add requires at least one TARGET argument")
     fs = FSSync()
     for a in args:
-        fs.add(a)
+        fs.add(a, type, factory)
 
 def remove(opts, args):
     """fssync remove TARGET ...
@@ -210,6 +234,8 @@
     objects.  No TARGET must exist.  The next commit will remove them
     from the Zope 3 server.
     """
+    if not args:
+        raise Usage("remove requires at least one TARGET argument")
     fs = FSSync()
     for a in args:
         fs.remove(a)
@@ -254,17 +280,50 @@
     fs = FSSync()
     fs.multiple(args, fs.status)
 
+def checkin(opts, args):
+    """checkin [-m message] URL [TARGETDIR]
+
+    URL should be of the form ``http://user:password@host:port/path''.
+    Only http and https are supported (and https only where Python has
+    been built to support SSL).  This should identify a Zope 3 server;
+    user:password should have management privileges; /path should be
+    the traversal path to a non-existing object, not including views
+    or skins.
+
+    TARGETDIR should be a directory; it defaults to the current
+    directory.  The object tree rooted at TARGETDIR is copied to
+    /path.  subdirectory of TARGETDIR whose name is the last component
+    of /path.
+    """
+    message = "fssync_checkin"
+    for o, a in opts:
+        if o in ("-m", "--message"):
+            message = a
+    if not args:
+        raise Usage("checkin requires a URL argument")
+    rooturl = args[0]
+    if len(args) > 1:
+        target = args[1]
+        if len(args) > 2:
+            raise Usage("checkin requires at most one TARGETDIR argument")
+    else:
+        target = os.curdir
+    fs = FSSync(rooturl=rooturl)
+    fs.checkin(target, message)
+
 command_table = {
     "checkout": ("", [], checkout),
     "co":       ("", [], checkout),
     "update":   ("", [], update),
-    "commit":   ("m:", ["message="], commit),
-    "add":      ("", [], add),
+    "commit":   ("m:r", ["message=", "raise-on-conflicts"], commit),
+    "add":      ("f:t:", ["factory=", "type="], add),
     "remove":   ("", [], remove),
     "rm":       ("", [], remove),
     "r":        ("", [], remove),
     "diff":     ("bBcC:iuU:", ["brief", "context=", "unified="], diff),
     "status":   ("", [], status),
+    "checkin":  ("m:", ["message="], checkin),
+    "ci":       ("m:", ["message="], checkin),
     }
 
 if __name__ == "__main__":


=== Zope3/src/zope/fssync/merger.py 1.8 => 1.8.2.1 ===
--- Zope3/src/zope/fssync/merger.py:1.8	Wed May 14 18:24:42 2003
+++ Zope3/src/zope/fssync/merger.py	Sun Jun 22 10:23:42 2003
@@ -204,6 +204,10 @@
         lmeta = self.getentry(local)
         rmeta = self.getentry(remote)
 
+        # Special-case sticky conflict
+        if "conflict" in lmeta:
+            return ("Nothing", "Conflict")
+
         # Sort out cases involving additions or removals
 
         if not lmeta and not rmeta:
@@ -279,7 +283,8 @@
     def cmpfile(self, file1, file2):
         """Helper to compare two files.
 
-        Return True iff the files are equal.
+        Return True iff the files both exist and are equal.
         """
-        # XXX What should this do when either file doesn't exist?
+        if not (isfile(file1) and isfile(file2)):
+            return False
         return filecmp.cmp(file1, file2, shallow=False)


=== Zope3/src/zope/fssync/metadata.py 1.2 => 1.2.2.1 ===
--- Zope3/src/zope/fssync/metadata.py:1.2	Tue May 13 12:15:21 2003
+++ Zope3/src/zope/fssync/metadata.py	Sun Jun 22 10:23:42 2003
@@ -106,22 +106,16 @@
 
     def flushkey(self, key):
         entries = self.cache[key]
-        todelete = [name for name, entry in entries.iteritems() if not entry]
-        for name in todelete:
-            del entries[name]
-        if entries != self.originals[key]:
+        # Make a copy containing only the "live" (non-empty) entries
+        live = {}
+        for name, entry in entries.iteritems():
+            if entry:
+                live[name] = entry
+        if live != self.originals[key]:
             zdir = join(key, "@@Zope")
             efile = join(zdir, "Entries.xml")
-            if not entries:
-                if isfile(efile):
-                    os.remove(efile)
-                    if exists(zdir):
-                        try:
-                            os.rmdir(zdir)
-                        except os.error:
-                            pass
-            else:
-                data = dumps(entries)
+            if exists(efile) or live:
+                data = dumps(live)
                 if not exists(zdir):
                     os.makedirs(zdir)
                 f = open(efile, "w")
@@ -129,4 +123,4 @@
                     f.write(data)
                 finally:
                     f.close()
-            self.originals[key] = copy.deepcopy(entries)
+            self.originals[key] = copy.deepcopy(live)


=== Zope3/src/zope/fssync/snarf.py 1.1 => 1.1.2.1 ===
--- Zope3/src/zope/fssync/snarf.py:1.1	Tue May 20 15:09:15 2003
+++ Zope3/src/zope/fssync/snarf.py	Sun Jun 22 10:23:42 2003
@@ -16,13 +16,12 @@
 This is for transferring collections of files over HTTP where the key
 need is for simple software.
 
-The format is as follows:
+The format is dead simple: each file is represented by the string
 
-- for directory entries:
-  '/ <pathname>\n'
+    '<size> <pathname>\n'
 
-- for file entries:
-  '<size> <pathname>\n' followed by exactly <size> bytes
+followed by exactly <size> bytes.  Directories are not represented
+explicitly.
 
 Pathnames are always relative and always use '/' for delimiters, and
 should not use '.' or '..' or '' as components.  All files are read
@@ -58,6 +57,7 @@
             def filter(fspath):
                 return True
         names = os.listdir(root)
+        names.sort()
         for name in names:
             fspath = os.path.join(root, name)
             if not filter(fspath):
@@ -112,16 +112,13 @@
             if not infoline.endswith("\n"):
                 raise IOError("incomplete info line %r" % infoline)
             infoline = infoline[:-1]
-            what, path = infoline.split(" ", 1)
-            if what == "/":
-                self.makedir(path)
-            else:
-                size = int(what)
-                f = self.createfile(path)
-                try:
-                    copybytes(size, self.istr, f)
-                finally:
-                    f.close()
+            sizestr, path = infoline.split(" ", 1)
+            size = int(sizestr)
+            f = self.createfile(path)
+            try:
+                copybytes(size, self.istr, f)
+            finally:
+                f.close()
 
     def makedir(self, path):
         fspath = self.translatepath(path)

=== Removed File Zope3/src/zope/fssync/compare.py ===