Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely revis ion based storage for Zope ?)

[Kapil Thangavelu]

On Mon, 2004-04-12 at 18:03, Jean-Francois.Doyon at CCRS.NRCan.gc.ca wrote:
> G'Day,
> 
> Well, step one is done ... I now have Zope + Ape using Subversion as it's
> "filesystem" !!
> 

cool! 

> This is step one because, as Shawn suggested (Thanks for the pointer, that's
> what I needed!), this simply means that Zope uses SVN purely as a
> filesystem.
> 

 
> Because of subversion's nature, I want to look at 2 things beyond this that
> traditional filesystems don't support:
> 
> - Use zope's username for SVN logging.

using AccessControl.getSecurityManager().getUser().getId() and setting
it up as a revision prop ( or directly when creating the repo
transaction) should do it.

> - History/Undo support: View past revisions of an object, and revert to such
> a past revision.

perhaps the code for this would be helpful, (remove image for code link)
http://zope.org/Members/k_vertigo/Products/CMFSubversionBrowser/FileRevisions.png

> - Zope Version support: SVN is fully transactional and atomic, this should
> allow for support of Zope versions (I think ?)
> 

zope version support isn't really all that worthwhile, esp in a cmf
context. in general zope's version support (or zodb more particularly)
is a database level feature masquerading as an application one. since
objects modified in a version are in essence locked from participating
in other transactions, actions like modifying content in a version in a
cmf site amounts to locking the catalog from changes outside of the
version, which amounts to shutting down write activities to a cmf site.
otoh, integration with zope's transaction manager would be a good thing,
although its problematic to integrate between svn and zope txn models,
more on that in a moment.

> In the longer term, there's great opportunity for:
> 
> - "Built-in" conflict management and resolution: No more need for a
> "SafetyBelt" type approach.  Right now I haven't looked at this at all.  I
> plan to implement smart merging where possible (It might work already
> actually, I just need to test it).  True conflicts (Where a merge can't be
> accomplished withouth user interaction) would raise some sort of conflict
> error.
> 

i don't know that conflict management is really useful in this context.
svn like zope relies on optimistic concurrency control, and currently
doesn't support dav locking (which zope does). ie, it will just throw an
error if the content has been changed since the transaction began. the
'normal' concurrency control of svn is better but dependent on using the
working copy (wc) layer, which is additional programming and storage
overhead. so at the layer of the svn client this is already done and
working and good, but integrating this functionality into zope is a bit
harder without wc overhard.

this also makes the transaction integration becomes harder because both
zope and svn are using what amounts to optimistic concurrency control
which makes it impossible afaics, to get real txn integration, ie in
zope's two phase commit strategy, the last chance for a participant to
safely abort is tpc_vote, but there is no real way of knowing if the svn
txn will suceed or not until its tried. if it is tried at this stage and
succeeds then there is the possibility of a latter txn participant
failing the tpc_vote and the txn being aborted, and if waits till
tpc_finish (last part of two phase commit) and the svn txn fails it can
hose the composite txn integrity. 

once svn supports dav locks, doing txn integration via resource locking
as part of tpc_vote (or earlier) would be possible, till then.. i dunno,
i can't see a way around this for real txn integration.

i'm also curious how you dealt with svn transactions as part of the ape
integration work to date.

> - Editing Zope content objects through interaction with the svn repository.
> I can checkout the repository, edit some objects, and chek them back in,
> never interacting with Zope directly ... I've already tried this !  Works
> great for text based content types such as PageTemplates or DTML Documents
> and so on ... I even did it with a JPG, though because the properties hold
> width and height, you get some weird looking pictures :) The concept is
> valid though.  There may someday be a way to leverage this functionality
> better with a purpose built client of some sort.


to me this is one of the fundamental benefits of using svn, giving users
the ability to use interfaces like tortoisesvn (win shell extensions) or
mac finder extensions to interact directly with content.

> 
> - Leveraging SVN's property management.  Content in SVN has properties, much
> like Zope does.  I haven't looked at it yet, but I've noticed ".properties"
> file appearing ... I'm guessing those are the Zope properties, which would
> be better handled by subversion's property mechanism.  And properties are
> versioned too !

definitely!



> 
> In the realm of the wishful thinking, there's even more:
> 
> Right now, HEAD (Latest/youngest revision) is always used and worked with.
> The really powerful feature I want to eventually get to is publsihing
> something of a given revision, while editing another.  One potential
> paradigm for distiguishing between the two modes of operation could be to
> use anonymous vs. authenticated.  This is not useful to everyone, but can be
> in certain circumstances, most notably where authenticated =
> authors/developpers and anonymous = normal users.  This however requires ZMI
> interfaces, and in my case CMF ones as well ... This would be global though
> ... Eventually it'd be nice to have per object control of this stuff.  Andy
> McKay says it can't be done, anybody care to contradict him ? :P I image I'd
> have to monkey pathc something DEEP in the Zope code base, but I find the
> mix-in class that's the commonn denominator ... why not ?
> 

to me this seems a separate point then svn usage at all, its really
about content view paradigms for revision workflow. the two predominant
ones are using staging to deploy different sites representing different
parts of the publication cycle/wf, and the other which afaik, is only
widely used by silva is view multiplexing where the view of a content is
responsible for retrieving the proper version to render a display for,
although afaicr silva support for such is initimately tied to its
workflow and versioning paradigm. 

> Anyways, I'm just rambling by now ... Comments, thoughts and constructive
> criticism welcome !

i've done some thinking and work with integration of svn as content in
zope as well, though using a different software stack than ape. some
notes/ideas culled from a dev diary i kept while working on it that
might be useful.

- using svn hook scripts for event notification back into zope (content
reindex), although it should be careful about modifying the content
otherwise some client which just committed might be put out of date.

- transform layers (using portal transforms) with a private content type
registry... although integrating at the ape layer this might not be as
useful.

cheers,

-kapil


> 
> Thanks,
> J.F.
> 
> -----Original Message-----
> From: zope-dev-bounces at zope.org [mailto:zope-dev-bounces at zope.org]On
> Behalf Of Shane Hathaway
> Sent: April 8, 2004 11:20 AM
> To: Jean-Francois.Doyon at ccrs.nrcan.gc.ca
> Cc: zope-dev at zope.org
> Subject: Re: [Zope-dev] Using a truely revision based storage for Zope ?
> 
> 
> Jean-Francois.Doyon at CCRS.NRCan.gc.ca wrote:
> > I've started looking at the ZODB and APE packages to try and get some
> > understanding of how the whole storage interaction works, but it'll take
> me
> > some time to figure it all out ... So I thought I'd get feedback on the
> idea
> > first ...
> 
> Sounds great!  If I were you, I would start by replacing 
> Ape/lib/apelib/fs/fileops.py with something that uses the Subversion 
> module.  It's a very simple module and you should be able to get pretty 
> far that way.
> 
> Shane