-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On 16.03.2009 4:52 Uhr, Tres Seaver wrote:

Andreas Jung wrote:

...

On 15.03.2009 18:42 Uhr, Tres Seaver wrote:

...

...

-------- Original Message -------- Subject: [Bug 343079] [NEW] Broken distribution (2009-03-15) Date: Sun, 15 Mar 2009 07:42:00 -0000 From: dmaurer dieter@handshake.de Reply-To: Bug 343079 343079@bugs.launchpad.net To: tseaver@palladion.com References: 20090315074200.12457.19313.malonedeb@potassium.ubuntu.com Public bug reported: The current (2009-03-12) PyPI distribution for Zope2 2.12.0a1 is broken. 'easy_install'ing it leads to version conflicts for 'zope.component' (3.5.1 versus 3.6.0) in the call of 'mkzopeinstance'. ** Affects: zope2 Importance: Undecided Status: New

...

...

The breakage is due to the release of the new zope.prinipalregistry egg. We should probably manage a Zope2 indes and tell people to use it when running easy_install, because PyPI is not suitable for the task given setuptools' "incremental requirements discovery" design.

Easy_installing the a1 sdist should behave like using buildout since the versions within the sdist are pinned as well. It actually worked at the time of the a1 release. I don't understand right now why we get this failure.

I don't see any pinning at all here:

http://svn.zope.org/Zope/tags/2.12.0a1/setup.py?rev=97288&view=auto

Please look at the getPackages() method taking the version*cfg files into account. So all versions should be pinned. However there is obviously a difference between using buildout with pinned versions and setuptools or a small undetected hole in the process.

This kind of issue was the source of my contentiont that "released" versions should ship with exact pins of the egg versions (the full transitive closure): it should at least be possible to generate a 'Zope2-exact' distribution which provides a "known good" installation, even it a 'Zope2-upgradable' distribution might be better for some people.

The other option, as I said earlier, is to maintain an index for each "release branch" of Zope2, and populate it only with eggs which have been tested not to break the upgrade. We could specify that index in the install docs, and maybe even in the 'setup.cfg' of the package.

I hope we can discuss and resolve remaining issues during PyCon.

Andreas

Tres Seaver wrote:

Andreas Jung wrote:

...

Please look at the getPackages() method taking the version*cfg files into account. So all versions should be pinned. However there is obviously a difference between using buildout with pinned versions and setuptools or a small undetected hole in the process.

The issue must be that one of the "pinned" dependencies (zope.publisher?) has an unpinned dependency (maybe transitively?) which requires a newer version of zope.component.

What I think is more likely to have happened here is:

An additional package (like one under development) was installed first. This depends on some zope.foo package (maybe zope.publisher) which wants zope.component 3.6. setuptools goes and fetches the latest version of all of these. Now later on the Zope2 egg is put into the environment and requires zope.component 3.5.1 - result VersionConflict.

Setuptools loads packages and puts them into the environment as it finds them. It doesn't build a full tree of dependencies first. This is what pip adds for example.

Unless you have a KGS or any kind of version restrictions for everything from the start, you will always run into these problems. Managing exact versions inside setup.py doesn't work.

Hanno

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Hanno Schlichting wrote:

Tres Seaver wrote:

...

Andreas Jung wrote:

...

Please look at the getPackages() method taking the version*cfg files into account. So all versions should be pinned. However there is obviously a difference between using buildout with pinned versions and setuptools or a small undetected hole in the process.

The issue must be that one of the "pinned" dependencies (zope.publisher?) has an unpinned dependency (maybe transitively?) which requires a newer version of zope.component.

What I think is more likely to have happened here is:

An additional package (like one under development) was installed first. This depends on some zope.foo package (maybe zope.publisher) which wants zope.component 3.6. setuptools goes and fetches the latest version of all of these. Now later on the Zope2 egg is put into the environment and requires zope.component 3.5.1 - result VersionConflict.

This error is reproducible in a fresh virtualenv.

Setuptools loads packages and puts them into the environment as it finds them. It doesn't build a full tree of dependencies first. This is what pip adds for example.

Right: this is what I was calling the "incremental dependency discovery" bit in setuptlools.

Unless you have a KGS or any kind of version restrictions for everything from the start, you will always run into these problems. Managing exact versions inside setup.py doesn't work.

A Zope2-specific index supplies the same need.

Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com `

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Andreas Jung wrote:

On 16.03.2009 17:21 Uhr, Tres Seaver wrote:

...

Maybe generating indexes from the varios "known good" metadata we are already maintaining would be the right path.

By "index" you refer to a KGS or a release-specific directory containing the blessed packages under a well-known URL?

I mean an index which supplies the 'simple' PyPI interface, such that we could tell people to 'easy_install' from it, e.g.:

$ /path/to/bin/easy_install -i http://kgs.zope.org/Zope2/2.1.2

Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Chris Withers wrote:

Tres Seaver wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- I mean an index which supplies the 'simple' PyPI interface, such that we could tell people to 'easy_install' from it, e.g.:

$ /path/to/bin/easy_install -i http://kgs.zope.org/Zope2/2.1.2

But how do you then set things up when you want to use other eggs that are only available on PyPI? Surely as soon as you add PyPI as an egg source, things go belly up?

Personally, I evaluate such eggs in a sandbox, and then add them to the project-specific index once I'm sure that they work with the other software in the index: I don't use PyPI at all when building out production sites.

Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Chris Withers wrote:

Tres Seaver wrote:

...

Chris Withers wrote:

...

Tres Seaver wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- I mean an index which supplies the 'simple' PyPI interface, such that we could tell people to 'easy_install' from it, e.g.:

$ /path/to/bin/easy_install -i http://kgs.zope.org/Zope2/2.1.2

But how do you then set things up when you want to use other eggs that are only available on PyPI? Surely as soon as you add PyPI as an egg source, things go belly up?

Personally, I evaluate such eggs in a sandbox, and then add them to the project-specific index once I'm sure that they work with the other software in the index: I don't use PyPI at all when building out production sites.

That seems overly heavyweight for the average new user.

"no, sorry, you can't use Zope 2.12 with anything other than what it comes with unless you get your own egg repository running"

Who is talking about an "average new user"? I'm talking about the "my revenue stream depends on this application working" professional (me), and the best practices I use. PyPI is *great* as a tool for discovering new stuff: it sucks as the basis for a repeatable build environment.

Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Chris Withers wrote:

Tres Seaver wrote:

...

...

...

Personally, I evaluate such eggs in a sandbox, and then add them to the project-specific index once I'm sure that they work with the other software in the index: I don't use PyPI at all when building out production sites.

That seems overly heavyweight for the average new user.

"no, sorry, you can't use Zope 2.12 with anything other than what it comes with unless you get your own egg repository running"

Who is talking about an "average new user"?

We're talking about the standard was of doing things, that encompasses the average user. I don't see how the setup you describe can work unless every user runs their own egg server...

That wasn't what I said: the "Personally" part was a pretty clear signal that I was being descriptive of my practices, and not prescriptive for others'.

...

new stuff: it sucks as the basis for a repeatable build environment.

I think that's a little harsh, if you use buildout and a locked down versions section all you have to worry about is PyPI being down when you don't have the eggs in a local buildout cache.

No, you also have to worry about people removing eggs you formerly installed, or uploading new versions without changing the version number, or uploading new, backwards-compatibility-breaking versions, etc. *Nothing* about getting an egg from PyPI is repeatable.

Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com