[Grok-dev] find-links instead of eggbasket

[Jan-Wijbrand Kolman]

Martijn Faassen <faassen at startifact.com> wrote:
>  From the discussion earlier it looks like we can reliably use 
> find-links instead of eggbasket, with a caveat concerning timeout.
> 
> So, I propose:
> 
> * we devise a way to make our own index for Grok on
>    grok.zope.org/releaseinfo. This should contain basically what the
>    eggbasket contains (including windows eggs and such). So perhaps we
>     can use eggbasket. compoze is also a tool we should consider.
> 
> * we use the same structure we already had proposed for downloading the
>    versions info, though it turns out due to the extends cache we don't
>    strictly need it. I'd say: grok.zope.org/releaseinfo/1.1/eggs/ or
>    something for the eggs for Grok 1.1. We can put the versions list also
>    under grok.zope.org/releaseinfo/1.1/versions.cfg
> 
> * we then add a find-links to our standard grokproject-generated grok
>    projects that points to this. We remove the eggbasket stuff from
>    grokproject-generated projects.
> 
> * there was some talk about adding timeout facilities to buildout. This
>    doesn't sound strictly necessary to ensure Grok eggs will be fetched
>    from grok.zope.org, as find-links is reported to be followed first,
>    but we should at least explore what's optimal here for a typical
>    Grok installation.
> 
> Hopefully this strategy will make installation for beginners at least as 
> smooth as with eggbasket, but installation for users who already have 
> installed Grok more smooth as no more downloading of giant tarballs.


A couple of observations while implementing this strategy:

1) Eggs that are instrumental to the buildout process itself have never been 
part of the eggbasket tarball. They cannot be, because these eggs aren't 
dependencies of the package itself. For example:

collective.recipe.template
Paste
PasteDeploy
PasteScript
z3c.evalexception
z3c.recipe.dev
z3c.recipe.i18n
z3c.recipe.mkdir
zc.recipe.egg
zc.recipe.filestorage
zc.recipe.testrunner
zc.lockfile

2) We saw various eggs being downloaded that were not part of the eggbasket 
tarball, that one could expect to be included. For example:

grokcore.startup
grokui.admin
zope.app.apidoc
zope.app.file
zope.app.onlinehelp
zope.app.preference
zope.app.skins
zope.app.tree
zope.app.zapi
zope.filerepresentation

3) Both eggbasket and compoze __only__ include source distributions of the 
dependencies. They do not regard in anyway the binary eggs.

4) The compoze tool might be useful to us, but only in the sense that it might 
do a better job identifying the dependencies. The result of a compoze run is an 
package index structure, not a find-links structure as far as I can see. 

5) People have indeed reported that a find-links is followed first. However, if 
I do a buildout run __with__ a find-links (pointing to releaseinfo/1.0.1/eggs) 
and a non-responsive index URL, I see messages like these:

Installing app.
Download error: (110, 'Connection timed out') -- Some packages may not be found!
Download error: (110, 'Connection timed out') -- Some packages may not be found!
Getting distribution for 'grokcore.startup==0.4'.

The package then is found on the find-links location, but this seems to indicate 
that index actually is checked before the find-links as it takes a long long 
time...


A solution for observation 1) and 2) is to have a step in the release process of 
Grok, that replaces the eggbasket step. In this step we create a project with 
grokproject, and have this project use a download-cache. All the source 
distributions in this download-cache can then be uploaded to releaseinfo/
[version]/eggs.

I do not see an easy solution for observation 3).

For observation 5): I think we then do need a short timeout value after all.


I expressed this opinion to Martijn yesterday IRL already: I can't help to 
wonder if the effort we put into this is actually worthwhile (esp. since we have 
been relying on the package index for long while already, since eggbasket does 
not include everything...)


regards,
jw