[Zope-dev] Change in repository approach to software sharing

[Jim Fulton]

Tim Peters wrote:
> [Jim Fulton]
> 
>>The Zope project includes a number of interrelated subprojects,
>>such as:
>>
>>   - Zope 2
>>
>>   - Zope 3
>>
>>   - ZODB
>>
>>   - ZConfig
>>
>>Software from the ZODB and ZConfig projects are shared by Zope 2
>>and Zope 3.
> 
> 
> Note that ZODB also depends on ZConfig.

Yup and zdaemon and zLOG

> 
>>We want this sharing to be very convenient for people working on Zope 2
>>and Zope 3.  We don't want users of the Zope 2 and Zope 3 repository to
>>have to do separate checkouts for ZODB and ZConfig.  CVS supports such
>>software sharing through it's "module" system. The module system has
>>some flaws, so we use symbolic links instead.
>>
>>In a response to:
>>
>>   http://dev.zope.org/Zope3/MovingSCMToSubversion
>>
>>John Belmonte has suggested that Zope 2 and Zope 3 should depend
>>on specific versions of shared packages, rather than on the head.
>>I'm inclined to agree.
> 
> 
> Didn't we endure a lot of pain over the last year to get away from that
> model?  At one point there were 7 lines of ZODB I was wrestling with every
> day.  For example, there was the ZODB 3.1 branch, which "in theory"
> corresponded to the Zope 2.6 branch, but in practice never matched it.  Life
> got much better (for both Zope 2.6 maintenance and ZODB 3.1 maintenance)
> when we abandoned the ZODB3-3_1-branch branch of ZODB3 and did all ZODB 3.1
> maintenance directly on the Zope-2_6-branch branch of Zope.  Likewise for
> ZODB3-3_2-branch versus Zope-2_7-branch:  the former was abandoned, and life
> got better for everyone because of that.  Currently the HEAD of the ZODB
> module is the same as the HEAD view of ZODB when viewed from the Zope and
> Zope3 modules, which isn't a pure win, but isn't a pure loss either.

I think that subversion will make it easier to manage the changes.
Periodically, changes will be merged into Zope 2 and Zope 3. This should be
straightforward.

> 
>>People working on ZODB and ZConfig have to test their changes against
>>both Zope 2 and Zope 3 to avoid breakage.
> 
> 
> I don't see that this part can change, no matter how names are shuffled.

ZODB needs to be tested with Zope 2 and Zope 3 whenever updates are merged into those
projects.  Of course, you'll probably want to test more frequently, but you don't have to
test every change you make.

> 
>>This is very burdensome
> 
> 
> That part either -- all the tests in all contexts still need to be run.

But not for every change.

> 
>>and causes much pain when they get it wrong.
> 
> 
> Ditto, although you're (re)introducing mechanism that alleviates this at the
> cost of increasing the number of ZODB snapshots in active use.
> 
> 
>>Fortunately, subversion provides a mechanism for sharing specific
>>revisions.  We'll be able to have the convenience of getting
>>ZODB and ZConfig (and other shared software) when we do a checkout
>>*and* we'll be able to control what parts we get.
>>
>>To see how this will work, we'll look at ZConfig as an
>>example (because it has a single package) of reusable software
>>that we will include in Zope 3.
>>
>>In the repository, we'll have a top-level Zope3 project directory,
>>with the standard svn subdirectories "trunk", "branches", and "tags".
>>
>>We'll also have a top-level ZConfig project directory.  The "trunk"
>>of the ZConfig Python package will be in ZConfig/trunk/src/ZConfig.
>>If we create a tag T1 of ZConfig, then the Python package for that tag
>>will be in ZConfig/tags/T1/src/ZConfig.
>>
>>Now, when we set up the Zope 3 repository, we will create the ZConfig
>>package in Zope 3 by copying a *tag* from the ZConfig project:
>>
>>   svn copy svn+ssh://svn.zope.org/repos/ZConfig/tags/T1/src/ZConfig \
>>            svn+ssh://svn.zope.org/repos/Zope3/trunk/src/ZConfig
>>            -m 'Bring ZConfig T1 into main branch'
>>
>>Then, whenever someone checks out the Z3 trunk, they'll get
>>the ZConfig from T1.
> 
> 
> I don't yet know enough about svn to guess how the next step works:  since
> ZODB all by itself depends on ZConfig, presumably similar stitching of
> ZConfig will take place in the top-level ZODB project. 

Yup

 > Now when we go on to
> stitch ZODB into Z3, how will ZODB's attempt to stitch in its own version of
> ZConfig play with Z3's attempt to stitch in T1 above?  I'm not picturing how
> this works.  Maybe it's simple!

You won't stich in ZODB's copy of ZConfig.  You'll need to work with the one
used in Zope.

> 
>>If we need to, we can even make Zope3-local changes to ZConfig.
> 
> 
> That would bloat ZConfig maintenance headaches exponentially.  I'm confident
> of that, because of ZODB experience with (at least) 7 concurrently active
> ZODB snapshots -- it's simply intractable to keep straight what should and
> shouldn't get cross-ported across this set; mistakes and oversights are as
> much the norm as the exception then; the non-ZODB people doing stitching of
> ZODB into their projects won't be confident about exactly which tag they
> should stitch in; expediency will cause them to stitch in "whatever seems to
> work" at the time; and then the latter becomes an active snapshot that needs
> to be maintained too.
> 
> Don't make Zope3-local changes to ZConfig or ZODB, though, and at least that
> nest of rats can be sidestepped.

We can decide to do that. We can even recopy (rather than merge) when we sync up.

The important thing to realize is that we won't *maintain* ZODB in Zope 3
as such. Rather, we'll periodically sync up ZODB changes since the last sync.


> 
>>Later, we may decide to upgrade the Zope 3 head to use ZConfig tag 3.
> 
> 
> Ya, and the Zope head may decide to use ZConfig tag 44, while the Zope 2.8
> maintenance branch decides to use ZConfig tag 37 despite that ZConfig 3.3.1
> (which is supposed to go out with the next Zope 2.8 release) is actually
> released at tag 39.  The dark side of "flexibility" is that it creates that
> many more ways to get out of synch.
> 
> 
>>At that point, we can recopy from the tag, or we can merge changes
>>made between the two tags.
> 
> 
> Right now, the burden for making ZConfig changes "work" is entirely on Fred,
> because the changes he makes show up everywhere at once.  Similarly for ZODB
> versus me.  The "we" in the above is muddy:  whose responsibility will it be
> to recopy/merge? 

The person using the shared code. For example, whoever merges ZODB updates into
Zope 3 will need to make sure they work.


 > A clear answer to that may make it work better.  As a
> counterexample, it was never clear whose responsibility(ies) it was to
> coordinate ZODB versions with Zope(2) releases, and before we eliminated the
> distinct brances for ZODB and Zope, the version of ZODB shipped with a given
> Zope release never corresponded with a ZODB release.  It's only in the last
> year that we managed to create ZODB releases that truly matched Zope
> releases.  Then again, there was a lot of hosage in those processes <0.5
> wink>.

The goal is for Zope to depend on specific releases of ZODB.  This should
work a bit like vendor imports, but subversion should make it a good bit more
convenient.

If the new approach turns out not to work, we'll try something else. :)

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org