[Zope-dev] zope.component test isolation (was: Zope test layers, pytest, and test isolation)

[Jim Fulton]

On Fri, Mar 25, 2011 at 4:24 AM, Wolfgang Schnerring <ws at gocept.com> wrote:
> Hello Uli,
>
> I've spent quite some time thinking (and partly coding) about the same
> issues you mention (but didn't feel ready to talk about it here, yet),
> so I'm glad that maybe we can start thinking about them together
>
> I think your email addresses two quite different topics, so I'll split
> my reply. First up: test support for zope.component. Second part:
> about the concept of test layers.
>
> * Uli Fouquet <uli at gnufix.de> [2011-03-24 01:05]:
>> Right now we have a problem with pytest integration when it comes to
>> ZCA setups [...] all tests share the same global ZCA registrations
>> and changes to the registrations in one test will affect other tests
>> run thereafter. We have a lack of test isolation.
>
> Exactly. This issue has bitten me too in various places, and as far as
> I know there are no solutions for it, yet.

The classic solution is to start tests with empty registries, or, if
you're using layers, with some baseline registries.

> What I envision to solve this issue is that test support for
> zope.component should work the same way as with the ZODB. There, we
> have a *stack* of Databases (DemoStorages, to be precise) that wrap
> each other, where each one delegates reads downwards, but keeps writes
> for itself. So you might have one database for the layer that provides
> the baseline, and each test (in its setUp) gets its own database where
> it can do whatever it wants, because it is thrown away in its
> tearDown.
>
> In principle, quite a few of the mechanics to do the same thing with
> zope.component registries are already there (since a registry keeps a
> list of base registries it will delegate to when something can not be
> found in itself). And as Hanno and Godefroid mentioned, plone.testing
> does something in this direction already. (And, it bears repeating,
> in its core has no dependencies on Plone or Zope2.)

I like the idea of stacking registries.

> But as far as I see, there are issues that plone.testing does not
> address:
>
> 1. I've been going over this stuff with my colleague Thomas Lotze, and
> we realized that just squeezing in a new registry and bending its
> bases to the previously active one is not enough for full isolation,
> since this does not cover *deleting* registrations (one, you can only
> delete the registration from the precise registry it was created in,
> and two, in the just-bend-the-bases approach, once you delete a
> registration, it's gone forever).
>
> I think to provide full isolation, we need to make *copies*. And since
> zope.component in general supports a chain of registries, we probably
> need to make copies of each of them.

Is deleting registrations important?  This seems like an odd use case.
If it's needed, I would suggest starting with a baseline (e.g. stack)
that doesn't include the component you want to test deleting, then
adding in setup.

>
> 2. zope.component has two entry points, the global site registry and
> the current registry (getGlobalSiteManager and getSiteManager).
> The current registry can be anything, or more precisely, you can call
> getSiteManager.sethook(callable) and provide a callable that returns
> the current registry.
>
> I think to provide test support for zope.component (i. e. generally,
> at the "library level"), we need to support both entry points.

Why?  Why would someone care about anything other than the current
effective configuration.

> The
> global one is not hard, but the getSiteManager one gets nasty really
> fast, because of course we have to rely on bending getSiteManager to
> return the current "test registry"

But as you point out, there's a hook for that.

> -- but anyone could at any time
> call getSiteManager.sethook to change it!

Seriously?  Nobody calls that but deep infrastructure code.

> Which means we need to
> intercept that and a) prevent our hook from being replaced and b)
> inject the new registry into the test stack somehow.

I think you're making this more complicated than it needs to be.

> As far as I understand, plone.testing sidesteps these issues by only
> dealing with the global registry, and specially munging the two known
> cases in the Zope world where getSiteManager is changed (zope.site and
> five.something).
>
> **
>
> I'd like to know what people think about this plan.
> Thomas and I have been over this quite a bit and think it's sound, not
> overly complicated, and (after we did some experiments) definitely
> doable. Please do point out stuff we missed. :-)

I think a stack-based approach is very appealing.  I think anything
more complex is likely to cause more problems than it solves.

> I'd very much like to put this functionality into zope.component
> itself, which of course raises backwards compatibility issues
> galore,

Not sure why this would have to be backward incompatible, but I'm
unconvinced that the complexity comes close to being justified by the
benefit.


> but any code for this definitely isn't wasted since we can always
> package it separately if we don't find a way to integrate it.
>
> Thomas and I taken up implementing this, but we can't devote a lot of
> time to it (about one session per week), so realistically I'm afraid I
> guess it will take a few months until we have something of substance.

OMG and then who gets to maintain it? Not it!

You and Thomas have obviously thought a lot about this and I
appreciate the effort you've put into this, but I really don't think
it's worth the added complexity.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton