[Zope3-checkins] CVS: Zope3/utilities/fssync - README.txt:1.1.2.1 add.py:1.1.2.1 checkout.py:1.1.2.1 commit.py:1.1.2.1 common.py:1.1.2.1 diff.py:1.1.2.1 sync.py:1.1.2.1 update.py:1.1.2.1 usage.py:1.1.2.1
Guido van Rossum
guido@python.org
Thu, 1 May 2003 17:39:23 -0400
Update of /cvs-repository/Zope3/utilities/fssync
In directory cvs.zope.org:/tmp/cvs-serv17867
Added Files:
Tag: fssync-branch
README.txt add.py checkout.py commit.py common.py diff.py
sync.py update.py usage.py
Log Message:
Checking in the fssync utilities. These haven't been converted yet.
=== Added File Zope3/utilities/fssync/README.txt ===
Usage: sync.py [options] operation
The available options are:
-f / --fspath=
Set the file system path where the object is to be dumped.
Is used with all the operation.
-o / --objpath=
Looks up for the Object in the specified path in the ZODB.
Is used with checkout operation.
-d / --dbpath=
The path of ZODB.
Is used with all the operation.
-s / --siteconfpath=
The path for site.zcml.
Is used with all the operation.
-t / --type=
The type of object to be added to the file system.
Is used with add operation.
OPERATION
----------------------------------------------------------------------------
checkout --- checks out ZODB object in File-system
example:
$python sync.py -f /home/user/sandbox
-d /zope3/data.fs
-s /zope3/site.zcml
-o /foo/bar
checkout
'or'
$python sync.py --fspath=/home/user/sandbox
--dbpath=/zope3/data.fs
--siteconfpath=/zope3/site.zcml
--objpath=/foo/bar
checkout
This will download the "bar" folder along with it's contents
into the /home/user/sandbox folder. In case if the "bar" module
already exist in the sandbox it will be overwritten.
Checkout generates the following output:
UPDATING foldername
U foldername/filename
update --- updates the sandbox
example:
$python sync.py -f /home/user/sandbox/bar
-d /zope3/data.fs
-s /zope3/site.zcml
update
'or'
$python sync.py --fspath=/home/user/sandbox/bar
--dbpath=/zope3/data.fs
--siteconfpath=/zope3/site.zcml
update
This will update the contents of "bar" directory in the sandbox.
An update can undergo various cases like :
Case1 : When the contents of all the three Sandbox, Original
and ZODB for an object is same an update command won't
do any changes.
Case2 : When the contents of the Sandbox and Original are same
but ZODB is different, an update command will overwrite
Sandbox and Original from ZODB and print the following:
U /path/filename.
Case3 : When the contents of the Original and ZODB are same and
the Sandbox is different an update command won't do any
changes and print the following:
M /path/filename.
If the Sandbox contains conflict data the following is
printed:
C /path/filename.
Case4 : This is a very unlikely case. When the contents of the
Sandbox and ZODB are same but the Original is different
an update command will overwrite the Original with the
ZODB and print the following:
U /path/filename.
Case5 : When the contents of all the three Sandbox, Original
and ZODB are different, an update command will mearge
all the changes into the Sandbox and the Original is
overwitten by the ZODB and the following is printed:
Merging changes in /path/filename
In case of a conflict the conflict data is copied into
the Sandbox and the Original is overwitten by the ZODB
and the following is printed:
C Merging changes in /path/filename
Conventionaly an update has to be done on the sandbox before
running commit.
commit --- A commit command copies data from the File-system samdbox into the ZODB
example:
$python sync.py -f /home/user/sandbox/bar
-d /zope3/data.fs
-s /zope3/site.zcml
commit
'or'
$python sync.py --fspath=/home/user/sandbox/bar
--dbpath=/zope3/data.fs
--siteconfpath=/zope3/site.zcml
commit
Commit copies the Sandbox into Original and ZODB, if the ZODB
and Original are same or there is no conflict data in the Sandbox
and prints the following:
/path/filename <-- filename
If the ZODB and Original are not same or the Sandbox contains conflict
data a commit command won't do anything and will print the following:
/path/filename Conflict, Uptodate checkin failed.
fcommit --- This is a force commit. A fcommit command will forcefully copy Sandbox
into Original and ZODB regardless of any constraints.
example:
$python sync.py -f /home/user/sandbox/bar
-d /zope3/data.fs
-s /zope3/site.zcml
fcommit
'or'
$python sync.py --fspath=/home/user/sandbox/bar
--dbpath=/zope3/data.fs
--siteconfpath=/zope3/site.zcml
fcommit
diff --- Difference between Three versions of an object
-1 -- Sandbox and Original
-2 -- Sandbox and ZODB
-3 -- ZODB and Original
A diff command will produce the difference of two objects
in context output format.
example:
$python sync.py -2 /home/user/sandbox/bar
-d /zope3/data.fs
-s /zope3/site.zcml
diff
'or'
$python sync.py -2 /home/user/sandbox/bar
--dbpath=/zope3/data.fs
--siteconfpath=/zope3/site.zcml
diff
Default option is -1
addtypes --- Displays all the list of types that can be added to the ZODB
from the filesystem.
example:
$python sync.py addtypes
add --- Adds objects to file system and saves it to the ZODB on
commit.
This command will give a list of all the available types
that can be added.
example:
$python sync.py -f /home/user/sandbox/bar
-d /zope3/data.fs
-s /zope3/site.zcml
-t file
add file1.html file2.txt file3.xyz
'or'
$python sync.py --fspath=/home/user/sandbox/bar
--dbpath=/zope3/data.fs
--siteconfpath=/zope3/site.zcml
--type=file
add file1.html file2.txt file3.xyz
This command will add file1.html, file2.txt, file3.xyz in
the bar folder of the sandbox and on commit this will be
checked in as file type objects in ZODB.
If type is not specified with -t option then the types will
be checked based on file extensions and only the valid type
will be added to the sandbox.
File/Folder names with spaces has to be enclosed withing quotes.
ENVIRONMENT VARIABLES
----------------------------------------------------------------------------
SYNCROOT --- Path for the File-system folder where the ZODB object
has to be checked out.If not set it takes the default
directory. Or can be specified with the --fspath or -f
option
ZODBPATH --- Path where the Data.fs file exist in the File-system.
if not set it takes the default is ../../Data.fs.
Or can be specified with the --dbpath or -d option.
SITECONFPATH --- Path for the site.zcml file.if not set it takes the
default is ../../site.zcml. Or can be specified with
the --siteconfpath or -s option.
=== Added File Zope3/utilities/fssync/add.py ===
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
from common import getZODBPath, getApplicationRoot, getObject, \
getObjectAdapter, setPrint
from Zope.App.FSSync.IObjectDirectory import IObjectDirectory
from Zope.App.FSSync.FSAddRequest import FSAddRequest
from Zope.ComponentArchitecture import getView, queryView
from Zope.App.OFS.Content.Folder.Folder import Folder
from Zope.App.FSSync.Syncer import toFS, fromFS
from Zope.XMLPickle.XMLPickle import loads
import string, os
from Zope.ComponentArchitecture.GlobalViewService import viewService
env = os.environ
def add(fspath
, dbpath
, siteconfpath
, newobjecttype
, newobjectname):
"""Adds objects to File system
Creates an object in the file system based on the object type
passed with -t option.
"""
fspath = os.path.abspath(fspath)
if not os.path.isdir(fspath):
return 'sync [add aborted] : %s is not a directory' %(fspath)
if not os.path.exists(os.path.join(fspath, '@@Zope')):
return 'sync [add aborted] : @@Zope administrative folder not found in %s' \
%(fspath)
objpath = getZODBPath(fspath)
root = getApplicationRoot(dbpath, siteconfpath)
container = getObject(objpath, root)
if container is None:
container, name, path = getObject(objpath, root, 'Y')
fsroot = os.path.abspath(fspath[:string.find(fspath, objpath)])
location = os.path.abspath(os.path.join(fsroot, path[1:]))
fromFS(container, name, location, 'T')
container = getObject(objpath, root)
adapter = getObjectAdapter(container)
if newobjecttype:
newobjecttype = '.'+newobjecttype
for file in newobjectname:
entries_path = os.path.join(os.path.join(fspath, os.path.dirname(file)),'@@Zope','Entries.xml')
entries = {}
if os.path.exists(entries_path):
entries = loads(open(entries_path).read())
if getObject(objpath+'/'+file, root) is not None:
setPrint('Object already exist in the ZODB : %s' \
%(os.path.join(fspath,file)))
elif entries.has_key(os.path.basename(file)):
setPrint('Object already exist in sandbox : %s' \
%(os.path.join(fspath,file)))
else:
err = addObject(adapter, fspath, newobjecttype, file, objpath)
if err is not None:
setPrint(err)
break
else:
for file in newobjectname:
entries_path = os.path.join(os.path.join(fspath, os.path.dirname(file)),'@@Zope','Entries.xml')
entries = {}
if os.path.exists(entries_path):
entries = loads(open(entries_path).read())
if getObject(objpath+'/'+file, root) is not None:
setPrint('Object already exist in the ZODB : %s' \
%(os.path.join(fspath,file)))
elif entries.has_key(os.path.basename(file)):
setPrint('Object already exist in sandbox : %s' \
%(os.path.join(fspath,file)))
else:
newobjecttype = '.'+string.split(file,'.')[-1]
err = addObject(adapter
, fspath
, newobjecttype
, file
, objpath)
if err is not None:
setPrint(err)
return None
def addObject(adapter
, fspath
, newobjecttype
, newobjectname
, objpath):
mode = 'N'
if IObjectDirectory.isImplementedBy(adapter):
request = FSAddRequest()
view = queryView(adapter, newobjecttype, request)
if view is not None:
newobjectpath = os.path.abspath(os.path.join(fspath, newobjectname))
if not os.path.exists(os.path.dirname(newobjectpath)):
return 'Nothing known about : %s'%(newobjectpath)
newobject = view.create()
objpath = objpath+'/'+newobjectname
objectname = os.path.basename(newobjectpath)
newobjectpath = os.path.dirname(newobjectpath)
dir_list = createAdminFolder(newobjectpath, objpath)
keys = dir_list.keys()
keys.sort()
for sl_no in keys:
ob = Folder()
toFS(ob, dir_list[sl_no][2], dir_list[sl_no][0], mode, dir_list[sl_no][1])
toFS(newobject, objectname, newobjectpath, mode, objpath)
else:
newobjectpath = os.path.abspath(os.path.join(fspath, newobjectname))
if not os.path.exists(newobjectpath):
return 'Nothing known about : %s'%(newobjectpath)
view = getView(adapter, '.', request)
newobject = view.create(newobjectpath)
objpath = os.path.abspath(objpath+'/'+newobjectname)
objectname = os.path.basename(newobjectpath)
newobjectpath = os.path.dirname(newobjectpath)
path_list = string.split(newobjectname, os.sep)
if len(path_list) > 1:
if path_list[-1] != objectname:
return 'Nothing known about : %s'%(path_list[-1])
dir_list = createAdminFolder(newobjectpath, objpath)
keys = dir_list.keys()
keys.sort()
for sl_no in keys:
ob = Folder()
toFS(ob, dir_list[sl_no][2], dir_list[sl_no][0], mode, dir_list[sl_no][1])
toFS(newobject, objectname, newobjectpath, mode, objpath)
else:
return 'Not a container type : %s' %(os.path.basename(fspath))
return None
def createAdminFolder(fspath, objpath, name=None, data={}):
if name is not None:
data[len(string.split(objpath, '/'))] = (fspath, objpath, name)
if not os.path.exists(os.path.join(fspath, '@@Zope')):
return createAdminFolder(os.path.dirname(fspath), os.path.dirname(objpath), os.path.basename(fspath))
return data
def addTypes(dbpath
, siteconfpath):
root = getApplicationRoot(dbpath, siteconfpath)
f = Folder()
adapter = getObjectAdapter(f)
request = FSAddRequest()
allviews = viewService.all()['default']
setPrint("\nALL AVAILABLE TYPES")
setPrint("====================================================\n")
for view in allviews:
try:
if len(view)>1 and view[0] == '.':
doc = getView(adapter, view, request).__doc__
setPrint(' %s \n %s \n\n' % (view, doc))
except:
pass
=== Added File Zope3/utilities/fssync/checkout.py ===
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
from Zope.App.FSSync.Syncer import toFS
from common import getObject, getApplicationRoot, \
makeNewEntry, getObjectAdapter
import string, os
def checkout(fspath
, dbpath
, siteconfpath
, objpath):
"""Checks out objects from ZODB to the file system
Downloads ZODB objects in file system to the specified path
specified but -f option.
"""
sandbox_path = fspath
if not os.path.isabs(objpath):
objpath = '/'+objpath
objpath = os.path.abspath(objpath)
root = getApplicationRoot(dbpath, siteconfpath)
main_ob = getObject(objpath, root)
ob_name = os.path.basename(objpath)
if main_ob is not None:
path = ''
for dir in string.split(objpath[:string.rfind(objpath,ob_name)], '/')[1:-1]:
path = os.path.join(path,dir)
sandbox_path = os.path.join(fspath, path)
admin_dir = os.path.join(os.path.dirname(sandbox_path), '@@Zope')
entries_path = os.path.join(os.path.dirname(sandbox_path),
'@@Zope'
, 'Entries.xml')
if not os.path.exists(sandbox_path):
os.mkdir(sandbox_path)
if not os.path.exists(admin_dir):
os.mkdir(admin_dir)
if not os.path.exists(entries_path):
makeNewEntry(admin_dir)
o_path = '/'+path
ob = getObject(o_path, root)
adapter = getObjectAdapter(ob)
type = adapter.typeIdentifier()
factory = adapter.factory()
name = os.path.basename(o_path)
makeNewEntry(admin_dir
, name
, type
, factory
, o_path)
fspath = sandbox_path
toFS(main_ob, ob_name, fspath)
return None
else:
return "Invalid object path"
=== Added File Zope3/utilities/fssync/commit.py ===
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
from Zope.App.FSSync.Syncer import fromFS
from common import getZODBPath, createTempfile, getObject, \
getApplicationRoot, traverseFS, checkConflictData, \
isNewObject, setPrint
from Transaction import get_transaction
import os, string, commands
env = os.environ
def commit(fspath
, dbpath
, siteconfpath
, mode=None):
"""Checks in from file system to ZODB
Saves the unconflict files in ZODB as Zope objects from the
file system.
"""
fspath = os.path.abspath(fspath)
vpath = '<>'
if os.path.isdir(fspath):
admin_dir = os.path.join(fspath,'@@Zope')
else:
admin_dir = os.path.join(os.path.dirname(fspath),'@@Zope')
if not os.path.exists(admin_dir):
return 'sync [commit aborted] : @@Zope administrative folder not found'
root = getApplicationRoot(dbpath, siteconfpath)
if mode == 'F':
err = doCommit(fspath, root, mode)
if err is not None:
return err
else:
mapping_paths = traverseFS(fspath, {})
for sandbox_path in mapping_paths.keys():
original_path = string.strip(mapping_paths[sandbox_path][0])
zopedb_path = string.strip(mapping_paths[sandbox_path][1])
if isNewObject(sandbox_path):
fsroot = os.path.abspath(sandbox_path[:string.find(sandbox_path
, zopedb_path)])
path = ''
for object in string.split(zopedb_path,'/')[1:]:
path = os.path.join(path,object)
newobjpath = os.path.join(fsroot,path)
if isNewObject(newobjpath):
err = doCommit(newobjpath, root, 'F')
if err is not None:
return err
if os.path.isdir(newobjpath):
vpath = newobjpath
break
else:
if string.find(sandbox_path,vpath)==-1:
fmt_sandbox_path = ''
fmt_original_path = ''
for dir in string.split(sandbox_path,os.sep):
if string.find(dir, ' ')<>-1:
fmt_sandbox_path = os.path.join(fmt_sandbox_path
, '\''+dir+'\'')
else:
fmt_sandbox_path = os.path.join(fmt_sandbox_path
, dir)
for dir in string.split(original_path,os.sep):
if string.find(dir, ' ')<>-1:
fmt_original_path = os.path.join(fmt_original_path
, '\''+dir+'\'')
else:
fmt_original_path = os.path.join(fmt_original_path
, dir)
fmt_sandbox_path = os.sep + fmt_sandbox_path
fmt_original_path = os.sep + fmt_original_path
ob = getObject(zopedb_path, root)
zopedb_temp_file = createTempfile(ob, zopedb_path)
diff_cmd1 = """diff -q -b %s %s;echo $?""" \
%(fmt_original_path, zopedb_temp_file)
diff_res1 = commands.getoutput(diff_cmd1)
isConflict = int(diff_res1[-1])
diff_cmd2 = """diff -q -b %s %s;echo $?""" \
%(fmt_sandbox_path, fmt_original_path)
diff_res2 = commands.getoutput(diff_cmd2)
isCommitRequired = int(diff_res2[-1])
if checkConflictData(sandbox_path, zopedb_path):
isConflict = 1
if isConflict:
msg = "%s Conflict, Uptodate checkin failed" \
%(zopedb_path)
setPrint(msg)
else:
if not (isCommitRequired == isConflict == 0):
msg = "%s <-- %s" \
%(zopedb_path, string.split(zopedb_path,'/')[-1])
err = doCommit(sandbox_path, root)
if err is not None:
return err
setPrint(msg)
if os.path.exists(zopedb_temp_file):
os.remove(zopedb_temp_file)
return None
def doCommit(fspath, root, mode=None):
container = ''
objpath = getZODBPath(fspath)
objname = string.split(objpath, '/')[-1]
path = string.split(objpath, '/')[:-1]
for item in path:
if item:
container = container +'[\''+item+'\']'
try:
if container:
container=eval('root'+container)
else:
container=root
except:
return 'sync [commit aborted] : invalid object path --- %s' \
%(objpath)
#copying to ZODB
fromFS(container, objname , os.path.dirname(fspath), mode)
get_transaction().commit()
if mode is None:
#copying to original
f = open(fspath, 'r')
data = f.read()
f.close()
original_path = os.path.join(os.path.dirname(fspath)
,'@@Zope'
,'Original'
,os.path.basename(fspath))
f = open(original_path, 'w')
f.write(string.strip(data))
f.close()
return None
=== Added File Zope3/utilities/fssync/common.py ===
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
from Zope.XMLPickle.XMLPickle import dumps, loads
from Zope.App.Traversing import traverse, getPhysicalPathString
from Zope.App import Application
from Zope.App.FSSync.IObjectEntry import IObjectEntry
from Zope.App.FSSync.IObjectFile import IObjectFile
from Zope.App.DublinCore.IZopeDublinCore import IZopeDublinCore
from Zope.App.FSSync.FSRegistry import getSynchronizer
from Zope.ComponentArchitecture import queryAdapter, getService
from Zope.App.FSSync.Syncer import toFS
from Zope.Exceptions import NotFoundError
import os, tempfile, string
seperator = os.sep
def getObjectAdapter(ob):
"""Returns the object adapter.
"""
syncService = getService(ob, 'FSRegistryService')
adapter = syncService.getSynchronizer(ob)
return adapter
def getApplicationRoot(dbpath
, siteconfpath):
"""Returns the application root.
"""
app = Application(dbpath, siteconfpath)
root = app()
return root
def getObject(objpath
, root
, prev=None
, name=None):
"""Gets the object from ZODB.
"""
if prev is None:
try:
ob = traverse(root,objpath)
return ob
except:
return None
else:
try:
ob = traverse(root,objpath)
return ob , name, objpath
except:
return (getObject(os.path.dirname(objpath)
, root
, 'Y'
, os.path.basename(objpath)), name, objpath)[0]
def getNewObject(ob
, folders
, files):
"""Returns the entire tree of a ZODB object.
"""
adapter = getObjectAdapter(ob)
path = ''
try:
path = str(getPhysicalPathString(ob))
except TypeError:
pass
if IObjectFile.isImplementedBy(adapter):
files.append(path)
else:
folders.append(path)
for cname, cob in adapter.contents():
getNewObject(cob, folders, files)
return folders, files
def getZODBPath(targetfile
, key=None):
"""Returns the physical path of an object in the ZODB
of a filesysten representation.
"""
targetfile = os.path.abspath(targetfile)
entries_path = os.path.join(os.path.dirname(targetfile)
,'@@Zope'
,'Entries.xml')
entries = loads(open(entries_path).read())
if key is None:
objectpath = entries[os.path.basename(targetfile)]['path']
else:
try:
objectpath = entries[os.path.basename(targetfile)][key]
except:
return None
return objectpath
def isNewObject(targetfile):
"""Determines if the object in the File-system is new.
"""
targetfile = os.path.abspath(targetfile)
entries_path = os.path.join(os.path.dirname(targetfile)
,'@@Zope'
,'Entries.xml')
entries = loads(open(entries_path).read())
if entries[os.path.basename(targetfile)].has_key('isNew'):
return 1
else:
return 0
def getObjectData(ob):
"""Returns the data of a ZODB object.
"""
adapter = adapter = getObjectAdapter(ob)
objectdata = adapter.getBody()
return objectdata
def getObjectDataTempfile(targetfile
, objpath
, dbpath
, siteconfpath):
"""Returns the data of a particular object in ZODB.
"""
root = getApplicationRoot(dbpath, siteconfpath)
objectpath = getZODBPath(targetfile)
ob = getObject(objectpath, root)
adapter = adapter = getObjectAdapter(ob)
dublincore_adapter = queryAdapter(ob, IZopeDublinCore)
modification_date = dublincore_adapter.ModificationDate()
object_data = adapter.getBody()
temp_file = tempfile.mktemp('tmp')
fo = open(temp_file,'w')
fo.write(string.strip(object_data))
fo.close()
return temp_file, objectpath, modification_date
def createTempfile(ob
, objectpath):
"""Creates a temporaty file in the File-system.
"""
objectdata = getObjectData(ob)
temp_file = tempfile.mktemp('tmp')
fo = open(temp_file,'w')
fo.write(string.strip(objectdata))
fo.close()
return temp_file
def checkFSPath(pathlist):
"""Checks the file system path.
"""
for path in pathlist:
if not os.path.exists(path):
err = "Invalid file system path --- "+str(path)
elif not os.access(path,2):
err = "Permission denied on --- "+str(path)
else:
err = None
if err is not None:
return err
def mapFS(mappings
, root
, objectroot
, fsroot):
"""Returns the new object list in the zopedb which
are not available in the File-system.
"""
ob = getObject(objectroot, root)
folders, files = getNewObject(ob, [], [])
for folder in folders:
path = fsroot
for part in string.split(folder, '/'):
path = os.path.join(path, part)
if not os.path.exists(path):
location = os.path.dirname(path)
ob = getObject(folder, root)
ob_name = string.split(folder, '/')[-1]
toFS(ob, ob_name, location)
for file in files:
path = fsroot
for part in string.split(file, '/'):
path = os.path.join(path, part)
if not os.path.exists(path):
location = os.path.dirname(path)
ob = getObject(file, root)
ob_name = string.split(file, '/')[-1]
toFS(ob, ob_name, location)
def traverseFS(fspath
, mapping_paths):
"""Traverse through the File-system.
"""
if os.path.isdir(fspath):
root=fspath
dirlist=os.listdir(root)
for item in dirlist:
if item!='@@Zope':
fspath=os.path.join(root,item)
if os.path.isdir(fspath):
traverseFS(fspath, mapping_paths)
else:
sandbox_path = fspath
original_path = os.path.join(os.path.dirname(fspath)
, '@@Zope'
, 'Original'
, os.path.basename(fspath))
if os.path.exists(original_path):
zopedb_path = getZODBPath(fspath)
mapping_paths[sandbox_path] = [original_path
, zopedb_path]
else:
sandbox_path = fspath
original_path = os.path.join(os.path.dirname(fspath)
, '@@Zope'
, 'Original'
, os.path.basename(fspath))
if os.path.exists(original_path):
zopedb_path = getZODBPath(fspath)
mapping_paths[sandbox_path] = [original_path
, zopedb_path]
return mapping_paths
def checkConflictData(sandbox_path
, zopedb_path):
"""Checks for conflict data in the File-system.
"""
isConflict = 0
f = open(sandbox_path,'r')
data = f.read()
f.close()
filter1 = '<<<<<<< %s' % (sandbox_path)
filter2 = '>>>>>>> %s' % (zopedb_path)
if string.find(data, filter1)<>-1 and string.find(data, filter2)<>-1:
isConflict = 1
else:
isConflict = 0
return isConflict
def makeNewEntry(admin_dir
, name=None
, type=None
, factory=None
, objpath=None
, isNew=None):
"""Edits the Entries.xml file.
"""
entries = {}
entries_path = os.path.join(admin_dir, 'Entries.xml')
if os.path.exists(entries_path):
entries = loads(open(entries_path).read())
if not entries.has_key(name) and name is not None:
if isNew:
entries[name] = {'type': type
, 'factory': factory
, 'isNew': 'Y'
, 'path': objpath}
else:
entries[name] = {'type': type
, 'factory': factory
, 'path': objpath}
open(entries_path, 'w').write(dumps(entries))
def getFSRoot(fspath
, next=None):
"""Returns the sandbox root.
"""
fspath = os.path.abspath(fspath)
if not os.path.exists(os.path.join(fspath,'@@Zope')):
return os.path.join(fspath,next)
else:
return getFSRoot(os.path.dirname(fspath)
, os.path.basename(fspath))
def setPrint(output_string):
print output_string
=== Added File Zope3/utilities/fssync/diff.py ===
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
from common import getObjectDataTempfile, setPrint
import os, commands, calendar, string
def getdiff(targetfile
, objpath
, dbpath
, siteconfpath
, diffoption=None):
"""Returns the difference between two files
Gets the diff between :
-- Current in Sandbox and Original of last sync
-- Current in Sandbox and Current in ZODB
-- Current in ZODB and Original of last sync.
"""
temp_file = ''
objectpath = ''
modification_date = None
zodbtargetfile = targetfile
if not os.path.isabs(targetfile):
targetfile=os.path.join(os.path.abspath(os.curdir)
,targetfile)
if not os.path.exists(targetfile):
return "sync [diff aborted] : Target file does not exist --- %s" \
% (str(targetfile))
if os.path.isdir(targetfile):
return "sync [diff aborted] : Target file found to be a directory --- %s" \
% (str(targetfile))
if not os.path.exists(os.path.join(os.path.abspath(os.path.dirname(targetfile))
,'@@Zope')):
return 'sync [diff aborted] : @@Zope administrative folder not found'
if string.find(os.path.basename(targetfile), ' ')<>-1:
targetfile = os.path.join(os.path.dirname(targetfile)
, '\''+os.path.basename(targetfile)+'\'')
if diffoption == '-1':
from_file = os.path.join(os.path.dirname(targetfile)
,'@@Zope'
,'Original')
to_file = targetfile
elif diffoption == '-2':
temp_file, objectpath, modification_date = getObjectDataTempfile(zodbtargetfile
, objpath
, dbpath
, siteconfpath)
from_file = targetfile
to_file = temp_file
elif diffoption == '-3':
temp_file, objectpath, modification_date = getObjectDataTempfile(zodbtargetfile
, objpath
, dbpath
, siteconfpath)
from_file = os.path.join(os.path.dirname(targetfile)
,'@@Zope'
,'Original'
,os.path.basename(targetfile))
to_file = temp_file
diff_cmd ="""diff -a -b -B -c %s %s""" \
% (from_file,to_file)
diff_res = commands.getoutput(diff_cmd)
diff_res = setOutput(diff_res
, temp_file
, objectpath
, modification_date)
setPrint(diff_res)
if temp_file:
os.remove(temp_file)
return None
def getDiffDate(strdt):
"""Returns date in Day Mon DD hh:mm:ss YYYY format.
"""
fmt_date=''
if strdt:
strdt = str(strdt)
weekdays={0:'Mon'
,1:'Tue'
,2:'Wed'
,3:'Thu'
,4:'Fri'
,5:'Sat'
,6:'Sun'}
months={1:'Jan'
, 2:'Feb'
, 3:'Mar'
, 4:'Apr'
, 5:'May'
, 6:'Jun'
, 7:'Jul'
, 8:'Aug'
, 9:'Sep'
, 10:'Oct'
, 11:'Nov'
, 12:'Dec'}
date=string.split(string.split(strdt,'T')[0],'-')
time=string.split(string.split(strdt,'T')[1],'.')[0]
day=weekdays[calendar.weekday(int(date[0])
,int(date[1])
,int(date[2]))]
fmt_date=day+' '+months[int(date[1])]+' '+date[2]+' '+time+' '+date[0]
return fmt_date
def setOutput(diff_res
, temp_file
, objectpath
, modification_date):
"""Sets the diff output replacing it with original object
path and modification datetime.
"""
changed_line = ''
modification_date = getDiffDate(modification_date)
if temp_file:
for line in string.split(diff_res,'\n'):
if string.find(line,temp_file)<>-1:
changed_line = line
break
if changed_line:
newline = changed_line[:string.find(changed_line,temp_file)]
newline = newline + objectpath+'\t\t'+modification_date
diff_res = string.replace(diff_res, changed_line, newline)
return "\n\n"+diff_res+"\n\n"
=== Added File Zope3/utilities/fssync/sync.py ===
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
import sys, string, getopt, os, commands
from diff import getdiff
from common import checkFSPath, setPrint
from checkout import checkout
from commit import commit
from update import update
from usage import USAGE
from add import add, addTypes
def main(argv):
short_options = ':f:o:d:s:1:2:3:t:'
long_options = ['fspath='
, 'objpath='
, 'dbpath='
, 'siteconfpath=']
operations = ['commit'
, 'checkout'
, 'update'
, 'diff'
, 'fcommit'
, 'add'
, 'addtypes']
err = ""
usage = USAGE % argv[0]
try:
optlist, args = getopt.getopt(sys.argv[1:]
, short_options
, long_options)
env=os.environ
objpath = ''
operation = ''
targetfile = ''
diffoption = '-1'
newobjectname = ''
newobjecttype = ''
if env.has_key('SYNCROOT'): fspath = env['SYNCROOT']
else: fspath = '.'
if env.has_key('ZODBPATH'): dbpath = env['ZODBPATH']
else: dbpath = '../../../Data.fs'
if env.has_key('SITECONFPATH'): siteconfpath = env['SITECONFPATH']
else: siteconfpath = '../../../site.zcml'
if len(optlist) > 0 or len(args) > 0:
for opt in optlist:
if (opt[0] == '-f') or (opt[0] == '--fspath'):
fspath = opt[1]
elif (opt[0] == '-o') or (opt[0] == '--objpath'):
objpath = opt[1]
elif (opt[0] == '-d') or (opt[0] == '--dbpath'):
dbpath = opt[1]
elif (opt[0] == '-s') or (opt[0] == '--siteconfpath'):
siteconfpath = opt[1]
elif (opt[0] == '-1' or opt[0] == '-2' or opt[0] == '-3'):
diffoption = opt[0]
targetfile = opt[1]
elif opt[0] == '-t':
newobjecttype = opt[1]
newobjectname = args[1:]
if len(args) <1:
err = "No operation has been specified"
raise err
elif args[0] != 'add' and len(args) >1:
err = "CommandlineError"
raise err
elif args[0] not in operations:
err = "Invalid operation---"+str(args[0])
raise err
else:
operation = args[0]
fspath, dbpath, siteconfpath = (os.path.abspath(fspath)
, os.path.abspath(dbpath)
, os.path.abspath(siteconfpath))
path=[fspath, dbpath, siteconfpath]
err=checkFSPath(path)
if err is not None:
raise err
else:
if len(args)>1:
if args[1]=='diff':
targetfile = args[0]
operation = args[1]
else:
setPrint(usage)
sys.exit(1)
else:
setPrint(usage)
sys.exit(1)
#Calls the specified operation
err = operate(operation
, fspath
, dbpath
, siteconfpath
, objpath
, targetfile
, diffoption
, newobjecttype
, newobjectname)
if err is not None:
raise err
except err:
setPrint(err)
sys.exit(1)
def operate(operation
, fspath
, dbpath
, siteconfpath
, objpath
, targetfile
, diffoption
, newobjecttype
, newobjectname):
"""Operates based on the operations passed
"""
err=""
if operation == 'checkout':
err = checkout(fspath
, dbpath
, siteconfpath
, objpath)
elif operation == 'commit':
err = commit(fspath
, dbpath
, siteconfpath)
elif operation == 'update':
err = update(fspath
, dbpath
, siteconfpath)
elif operation == 'diff':
err = getdiff(targetfile
, objpath
, dbpath
, siteconfpath
, diffoption)
elif operation == 'fcommit':
err = commit(fspath
, dbpath
, siteconfpath
, 'F')
elif operation == 'add':
err = add(fspath
, dbpath
, siteconfpath
, newobjecttype
, newobjectname)
elif operation == 'addtypes':
err = addTypes(dbpath
, siteconfpath)
return err
# If called from the command line
if __name__=='__main__': main(sys.argv)
=== Added File Zope3/utilities/fssync/update.py ===
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
import string, os, commands
from common import getZODBPath, createTempfile, getObject, \
mapFS, traverseFS, checkConflictData, getApplicationRoot, \
isNewObject, getFSRoot, setPrint
from Zope.App.FSSync.Syncer import toFS
env = os.environ
def update(fspath
, dbpath
, siteconfpath):
"""Updates the checked out objects in the file system from a ZODB
updates the files in the sandbox based on the file system path given
by -f option.
"""
root = getApplicationRoot(dbpath, siteconfpath)
if env.has_key('SYNCROOT'):
if os.path.abspath(fspath) == os.path.abspath(env['SYNCROOT']):
if os.path.isdir(os.path.abspath(fspath)):
module_list = os.listdir(os.path.abspath(fspath))
if '@@Zope' in module_list:
module_list.remove('@@Zope')
for module in module_list:
fspath = os.path.join(fspath, module)
updateSettings(fspath, root)
else:
return 'sync [update aborted] : @@Zope administrative folder not found in %s' \
%(fspath)
else:
updateSettings(fspath, root)
else:
updateSettings(fspath, root)
def updateSettings(fspath, root):
"""does the settings for update
"""
fspath = os.path.abspath(fspath)
if os.path.isdir(fspath):
admin_dir = os.path.join(fspath,'@@Zope')
else:
admin_dir = os.path.join(os.path.dirname(fspath),'@@Zope')
if not os.path.exists(admin_dir):
return 'sync [update aborted] : @@Zope administrative folder not found in %s'\
%(fspath)
mappings = {}
mapping_paths = {}
mapping_paths.update(traverseFS(fspath, mapping_paths))
for sandbox_path in mapping_paths.keys():
original_path = string.strip(mapping_paths[sandbox_path][0])
zopedb_path = string.strip(mapping_paths[sandbox_path][1])
if checkConflictData(sandbox_path, zopedb_path):
setPrint('C %s'%(zopedb_path[1:]))
else:
if not isNewObject(sandbox_path):
mappings[zopedb_path] = sandbox_path
fmt_sandbox_path = ''
fmt_original_path = ''
for dir in string.split(sandbox_path,os.sep):
if string.find(dir, ' ')<>-1:
fmt_sandbox_path = os.path.join(fmt_sandbox_path, '\''+dir+'\'')
else:
fmt_sandbox_path = os.path.join(fmt_sandbox_path, dir)
for dir in string.split(original_path,os.sep):
if string.find(dir, ' ')<>-1:
fmt_original_path = os.path.join(fmt_original_path, '\''+dir+'\'')
else:
fmt_original_path = os.path.join(fmt_original_path, dir)
fmt_sandbox_path = os.sep + fmt_sandbox_path
fmt_original_path = os.sep + fmt_original_path
ob = getObject(zopedb_path, root)
zopedb_temp_file = createTempfile(ob, zopedb_path)
diff3_cmd = """diff3 -a %s %s %s""" %(fmt_sandbox_path
, fmt_original_path
, zopedb_temp_file)
diff3_res = commands.getoutput(diff3_cmd)
if len(string.strip(diff3_res))>1:
diffverify=string.strip(diff3_res[0:5])
if diffverify=='====1':
setPrint('M %s'%(zopedb_path[1:]))
elif diffverify=='====2':
doUpdate('C', ob, zopedb_path, sandbox_path)
setPrint('M %s'%(zopedb_path[1:]))
elif diffverify=='====3':
doUpdate('U', ob, zopedb_path, sandbox_path)
elif diffverify=='====':
diff3_cmd = """diff3 -a -m %s %s %s;echo $?""" %(fmt_sandbox_path
, fmt_original_path
, zopedb_temp_file)
diff3_res = commands.getoutput(diff3_cmd)
diff3_res = string.replace(diff3_res
, zopedb_temp_file
, zopedb_path)
if diff3_res[-1]=='0':
setPrint('Merging changes in %s' \
%(zopedb_path[1:]))
else:
setPrint('C Merging changes in %s' \
%(zopedb_path[1:]))
f = open(sandbox_path, 'w')
f.write(string.strip(diff3_res[:-1]))
f.close()
doUpdate('C'
, ob
, zopedb_path
, sandbox_path)
if os.path.exists(zopedb_temp_file):
os.remove(zopedb_temp_file)
if os.path.isdir(fspath):
objectroot = getZODBPath(fspath)
fsroot = getFSRoot(fspath)
mapFS(mappings
, root
, objectroot
, fsroot)
def doUpdate(mode
, ob
, zopedb_path
, sandbox_path):
"""Updates data
"""
sandbox_path = os.path.dirname(sandbox_path)
ob_name = string.split(zopedb_path,'/')[-1]
if mode=='U':
if ob is not None:
toFS(ob, ob_name, sandbox_path)
elif mode=='C':
if ob is not None:
toFS(ob, ob_name, sandbox_path, mode)
=== Added File Zope3/utilities/fssync/usage.py ===
##############################################################################
#
# Copyright (c) 2001, 2002 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
USAGE="""
Usage: %s [options] operation
The available options are:
-f / --fspath=
Set the file system path where the object is to be dumped.
Is used with all the operation.
-o / --objpath=
Looks up for the Object in the specified path in the ZODB.
Is used with checkout operation.
-d / --dbpath=
The path of ZODB.
Is used with all the operation.
-s / --siteconfpath=
The path for site.zcml.
Is used with all the operation.
-t / --type=
The type of object to be added to the file system.
Is used with add operation.
OPERATION
----------------------------------------------------------------------------
checkout --- checks out ZODB object in File-system
example:
$python sync.py -f /home/user/sandbox
-d /zope3/data.fs
-s /zope3/site.zcml
-o /foo/bar
checkout
'or'
$python sync.py --fspath=/home/user/sandbox
--dbpath=/zope3/data.fs
--siteconfpath=/zope3/site.zcml
--objpath=/foo/bar
checkout
This will download the "bar" folder along with it's contents
into the /home/user/sandbox folder. In case if the "bar" module
already exist in the sandbox it will be overwritten.
Checkout generates the following output:
UPDATING foldername
U foldername/filename
update --- updates the sandbox
example:
$python sync.py -f /home/user/sandbox/bar
-d /zope3/data.fs
-s /zope3/site.zcml
update
'or'
$python sync.py --fspath=/home/user/sandbox/bar
--dbpath=/zope3/data.fs
--siteconfpath=/zope3/site.zcml
update
This will update the contents of "bar" directory in the sandbox.
An update can undergo various cases like :
Case1 : When the contents of all the three Sandbox, Original
and ZODB for an object is same an update command won't
do any changes.
Case2 : When the contents of the Sandbox and Original are same
but ZODB is different, an update command will overwrite
Sandbox and Original from ZODB and print the following:
U /path/filename.
Case3 : When the contents of the Original and ZODB are same and
the Sandbox is different an update command won't do any
changes and print the following:
M /path/filename.
If the Sandbox contains conflict data the following is
printed:
C /path/filename.
Case4 : This is a very unlikely case. When the contents of the
Sandbox and ZODB are same but the Original is different
an update command will overwrite the Original with the
ZODB and print the following:
U /path/filename.
Case5 : When the contents of all the three Sandbox, Original
and ZODB are different, an update command will mearge
all the changes into the Sandbox and the Original is
overwitten by the ZODB and the following is printed:
Merging changes in /path/filename
In case of a conflict the conflict data is copied into
the Sandbox and the Original is overwitten by the ZODB
and the following is printed:
C Merging changes in /path/filename
Conventionaly an update has to be done on the sandbox before
running commit.
commit --- A commit command copies data from the File-system samdbox into the ZODB
example:
$python sync.py -f /home/user/sandbox/bar
-d /zope3/data.fs
-s /zope3/site.zcml
commit
'or'
$python sync.py --fspath=/home/user/sandbox/bar
--dbpath=/zope3/data.fs
--siteconfpath=/zope3/site.zcml
commit
Commit copies the Sandbox into Original and ZODB, if the ZODB
and Original are same or there is no conflict data in the Sandbox
and prints the following:
/path/filename <-- filename
If the ZODB and Original are not same or the Sandbox contains conflict
data a commit command won't do anything and will print the following:
/path/filename Conflict, Uptodate checkin failed.
fcommit --- This is a force commit. A fcommit command will forcefully copy Sandbox
into Original and ZODB regardless of any constraints.
example:
$python sync.py -f /home/user/sandbox/bar
-d /zope3/data.fs
-s /zope3/site.zcml
fcommit
'or'
$python sync.py --fspath=/home/user/sandbox/bar
--dbpath=/zope3/data.fs
--siteconfpath=/zope3/site.zcml
fcommit
diff --- Difference between Three versions of an object
-1 -- Sandbox and Original
-2 -- Sandbox and ZODB
-3 -- ZODB and Original
A diff command will produce the difference of two objects
in context output format.
example:
$python sync.py -2 /home/user/sandbox/bar
-d /zope3/data.fs
-s /zope3/site.zcml
diff
'or'
$python sync.py -2 /home/user/sandbox/bar
--dbpath=/zope3/data.fs
--siteconfpath=/zope3/site.zcml
diff
Default option is -1
addtypes --- Displays all the list of types that can be added to the ZODB
from the filesystem.
example:
$python sync.py addtypes
add --- Adds objects to file system and saves it to the ZODB on
commit.
This command will give a list of all the available types
that can be added.
example:
$python sync.py -f /home/user/sandbox/bar
-d /zope3/data.fs
-s /zope3/site.zcml
-t file
add file1.html file2.txt file3.xyz
'or'
$python sync.py --fspath=/home/user/sandbox/bar
--dbpath=/zope3/data.fs
--siteconfpath=/zope3/site.zcml
--type=file
add file1.html file2.txt file3.xyz
This command will add file1.html, file2.txt, file3.xyz in
the bar folder of the sandbox and on commit this will be
checked in as file type objects in ZODB.
If type is not specified with -t option then the types will
be checked based on file extensions and only the valid type
will be added to the sandbox.
File/Folder names with spaces has to be enclosed withing quotes.
ENVIRONMENT VARIABLES
----------------------------------------------------------------------------
SYNCROOT --- Path for the File-system folder where the ZODB object
has to be checked out.If not set it takes the default
directory. Or can be specified with the --fspath or -f
option
ZODBPATH --- Path where the Data.fs file exist in the File-system.
if not set it takes the default is ../../Data.fs.
Or can be specified with the --dbpath or -d option.
SITECONFPATH --- Path for the site.zcml file.if not set it takes the
default is ../../site.zcml. Or can be specified with
the --siteconfpath or -s option.
"""