[Zope-dev] "ZTK" futures: one big package?

Tue May 12 04:16:44 EDT 2009

chris
you have *exactly* mirrored my feelings as a longtime zope2 user/application
developer getting in touch with plone3/five.
now after a year digging in it I still am not free of it

thanks a lot
robert

Chris McDonough schrieb:
> I realize now that I've neglected to give sufficient reasoning for why less 
> granular packaging would be a good thing.
> 
> I've noticed that there's a common theme in Zope development, software, and 
> packages that I can only describe as "power law development" or "turtles all the 
> way down".  It's a bit of an antipattern, unfortunately.
> 
> I'll provide an example by way of Zope-3-the-appserver.  In an application that 
> uses Zope-3-the-appserver, many individual subframeworks will be used.  For 
> example, there is a traversal subframework, a security subframework, a 
> cataloging subframework, and so on.
> 
> Each of these subframeworks acts as a logical unit, and through the magic of the 
> component architecture, each can be replaced wholesale by registering some 
> adapter.  However, each of these subframeworks tend to also have settings that 
> can be configured.  For example, individual traversal steps for certain types of 
> objects can be overridden by registering an adapter that *configures* the 
> subframework.  In the case of Zope 3, we have a traversal situation where the 
> larger traversal subframework can either be replaced wholesale via an adapter 
> registration or extended piecemeal via other adapter registrations.
> 
> The problem is that the mechanism to *replace* the subframework is the same as 
> the mechanism to *configure* it (both are done via adapter registration, 
> sometimes even in the same file).  This is theoretically fine.  But in reality, 
> it's tremendously hard for someone just walking up to a complex system like Zope 
> 3 to discern adapter registrations that replace subsystems from those which 
> merely configure subsystems.  An inability to discern the difference leads to 
> situations where people just "don't get the joke" and try to wiggle wires to 
> configure-to-death a existing subsystem that's clearly suboptimal for their use 
> case instead of just replacing it wholesale with a much simpler custom policy. 
> They just don't know it was engineered to be replaced.  So they keep adding more 
> configuration code to the existing subframework to handle various 1% edge cases. 
>   Often this code makes the subframework tremendously complex, and the 
> subframework grows inappropriate dependencies along the way.  *Sometimes* the 
> situation gets so confusing for a new user, they just quit and go use something 
> else.
> 
> This is a pattern that happens over and over again in Zope development.  In my 
> personal opinion, the original error was trying to make the subframework 
> configurable at all.  Instead, the subframework should be replaceable easily, 
> but it should itself be relatively rigid.  At very least, for subframeworks that 
> really do require extra configuration (should be very few), this configuration 
> should be done via highly specialized ZCML directives (or grokkers), as opposed 
> to some very general adapter registration that can't be easily discerned from 
> other adapter registrations by a newbie.
> 
> If the subframeworks were more rigid (but replaceable), the *intent* of the 
> subframework author could be more easily discerned, and fewer people would fall 
> into the trap of adding more configuration code to a subframework instead of 
> just replacing it entirely.  And fewer people would just walk away in frustration.
> 
> What does this have to do with packaging?  Well, currently, there's a dizzying 
> number of packages that make up "the ZTK" (nee "Zope 3").  Each of these 
> packages is a pure peer of all others in a PyPI listing with no real way to get 
> a sense of their relative importance other than performing a linear audit.  Even 
> if a user *does* do a linear search of all of them, it's still awful hard to 
> discern for some new user which ones are "important", and which ones just happen 
> to exist by some inequity of history without trying to install it.  The user 
> needs to gain some holistic knowledge of the system in order to discern the 
> important bits from these historical inequities.
> 
> Most new users understandably just walk away from *all* Zope packages before 
> they gain this knowledge; it's just too hard for them to tell the difference 
> between the truly important and reusable bits and the stuff that just happens to 
> be packaged up and released but which is useless outside of some highly specific 
> context.  In effect, we just don't communicate *intent* very effectively in our 
> current packaging structure.
> 
> In my opinion, this is why a lot of Python developers who are otherwise very 
> smart have given up on trying to use Zope packages.  The time required to figure 
> out which ones are useful and which ones aren't is just too high.  It's way 
> "easier" for them to write them all off wholesale and just write what they need 
> from scratch or use somebody else's software.  Good developers tend to like 
> small, useful libraries and frameworks.
> 
> We can ameliorate the situation in a few ways:
> 
> - We can reduce the number of distributions.
> 
> - We can make each current Zope package distribution independently useful.
> 
> My suggestion is that we do the former first in order to communicate *current 
> intent* (the "state of reality right now").  We can do the latter after this in 
> pieces, hopefully aided by some new developers we've picked up by making it 
> easier to find useful stuff.
> 
> Once we deflate our current set of packages down to a reasonable number, the 
> packages listed in PyPI will immediately start to reflect "the state of reality 
> right now".  As a result, we'll hopefully be able to get some new blood in the 
> form of new developers that use the smaller bits outside Zope to help us tease 
> the truly independent pieces out of the larger pile.  If we do this, at no time 
> after the deflation will PyPI listings ever as badly advertise "the state of 
> reality" as it is advertised right now, and the community will hopefully again 
> start to grow.
> 
> - C
> 
> 
> On 5/11/09 11:11 AM, Martijn Faassen wrote:
>> Hey,
>>
>> Chris McDonough wrote:
>>
>>   >  Instead, I have argued for promoting packages that have some life of
>>   >  their own (independent of the rest of the pile) into subprojects that
>>   >  have their own release cycles.
>>
>>   >  Then outside projects such as Plone and Grok could depend on
>>   >  independent versions of such packages, giving them slightly more
>>   >  flexibility than requiring a "version of the ZTK".
>>
>> We already have that flexibility today. To me, the utility of a release
>> of version numbers in the ZTK does not at all exclude the potential to
>> evolve the packages to more independent sub-projects.
>>
>>> Given that this suggestion has been met with skepticism, let me try another
>>> tact.
>> I think that's an inaccurate description of the response you got. I'm
>> quite positive about trying to give as many packages as possible a life
>> of their own. I don't think you got anyone else arguing against this
>> point of view.
>>
>> I'm also quite positive that some packages are:
>>
>> * useful as independently distributed packages
>>
>> * only make sense in a Zope 3 or a Grok or a Zope 2 context, i.e. they
>> depend on a significant set of Zope packages.
>>
>> I'd like to get out of this paradigm:
>>
>> * the Zope packages are independent sub-projects.
>>
>> * therefore we cannot distribute a list of versions that work best together.
>>
>> And this one:
>>
>> * if we distribute a KGS of anything
>>
>> * packages in that anything aren't independently reusable automatically
>> and should be merged into a ball.
>>
>> I'd also like to get out of the following paradigm:
>>
>> * the Zope packages are not independently reusable yet
>>
>> * therefore we should distribute them all together
>>
>> We're in a grey area. Some package are here, some packages are there,
>> some are in between. Some packages build on other packages, but have
>> clear dependency structures. Some don't have clear dependency
>> structures. Some have better documentation and better focus than others.
>>
>> If there is to be a merging of code together, then I propose we continue
>> the project where the ZMI code is merged into one or a few packages. We
>> can also investigate merging 2 or 3 packages together where it seems to
>> make sense, or simply moving code between packages (some code needs to
>> go down the dependency list, some up).
>>
>>> Instead of thinking about it this way, could we think about it as
>>> *deflating* the current set of zope.* packages that do not currently have a
>>> meaningful life of their own into a single setuptools distribution named "ZTK".
>>>    This package would include most everything in zope.app*, plus things like
>>> zope.server, zope.publisher, and other things that just aren't useful outside of
>>> Zope-the-appserver, or which currently just depend on "too much".
>> -1
>>
>> This would make it a lot harder to:
>>
>> * clean up dependency relationships with the goal of creating more
>> reusable code. We'd all hide them in a sumo ball again.
>>
>> * get rid of bits we *don't want anymore*. If I need anything in a sumo
>> package I'd need *all* of it.
>>
>> * override individual packages with newer versions
>>
>> * we've done a lot of refactoring recently trying to separate the UI
>> from packages. This is done by creating a *new* package, leaving the old
>> package behind. We can do this, test this and release this
>> package-by-package now.
>>
>>> Over time, we'd tease out the dependencies of packages that live in the ZTK
>>> distribution, making them useful outside without any dependency on the ZTK.  The
>>> names of these packages could be arbitrary (they wouldn't need to retain their
>>> old "zope.*" names).  Some backwards dependency shims would be added to the ZTK
>>> to prevent old imports from breaking, and the ZTK distribution would then just
>>> have a dependency on the thing that got broken out.
>> I don't like the attempt to redefine what the ZTK means to a giant ball
>> of Python packages. That's implying that, say, zope.component is *not*
>> in the ZTK. That's wrong.
>>
>> Why generate a whole lot of work for ourselves getting from where we are
>> now to here? We've learned how to work with the current situation in
>> 2007 already.
>>
>>> I'm thinking that this would simplify things greatly for people who want to be
>>> consumers of "truly independent" Zope packages.  There'd be exactly N of them
>>> available for download (where N is much less than 100, more like 20), plus the
>>> ZTK, which would have the rest of the pile in it over time.
>> I don't see why a big package would "simplify things greatly" for you or
>> anyone else.
>>
>>> If someone wanted
>>> to use a forked version of a package that lived in the ZTK distribution, they'd
>>> either do so by teasing out the dependencies and making it "truly independent"
>>> or they'd just reroll a custom version of the entire ZTK distribution.
>> And that's easier than the current situation how? Are you really
>> proposing we destroy the dependency information we've already teased out
>> and then make people do the work again?
>>
>>> Does this make any sense?
>> Not a lot in my book.
>>
>> I think an important reason why there's so much awareness of dependency
>> issues in the Zope world now (and effort spent on it) is precisely
>> because we released our separate packages and can see the dependency
>> information clearly.
>>
>> Regards,
>>
>> Martijn
>>
>> _______________________________________________
>> Zope-Dev maillist  -  Zope-Dev at zope.org
>> http://mail.zope.org/mailman/listinfo/zope-dev
>> **  No cross posts or HTML encoding!  **
>> (Related lists -
>>   http://mail.zope.org/mailman/listinfo/zope-announce
>>   http://mail.zope.org/mailman/listinfo/zope )
>>
> 
> _______________________________________________
> Zope-Dev maillist  -  Zope-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists - 
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope )
> 
> 