[Grok-dev] z3c.testsetup and layers

Uli Fouquet uli at gnufix.de
Wed Oct 21 10:39:45 EDT 2009


Hi JW,

First of all, many thanks for the lots of work you've done!

Am Dienstag, den 20.10.2009, 13:21 +0200 schrieb Jan-Wijbrand Kolman:

> I'm trying to grasp testing layers in general and functional testing
> layers in particular - all from a z3c.testsetup perspective. Since it is
> not all that clear in my head still, let met try to write things down
> for others to comment on.
> 
> I also have some questions and suggestions for z3c.testsetup in particular.
> 
> 
> The summary of test layers:
> ===========================
> 
> * Test layers is a feature of zope.testing.
> 
> * Test layers can be used to group tests that have a shared test setup
> and teardown (the setup and teardown happen once for all the tests in 
> grouped in that layer).
> 
> * Test layers are classes and thus test layers can inherit from each
> other.
> 
> * The zope.testing.testrunner will make sure that setup functions of
> more generic layers are called before that of the specific layers and
> the other way around for the teardown functions.
> 
> * The setup and teardown functions are to be defined as classmethods(!) 
> on the test layers.

One might add, that one can add a setup function for each test, but
that's not layer related.

> * See also:
> 
> http://apidoc.zope.org/++apidoc++/Code/zope/testing/testrunner-layers-api.txt/index.html
> 
> 
> So far so good.
> 
> 
> The summary of functional test layers:
> ======================================
> 
> * Functional test layers is a feature of zope.app.testing (notice the 
> .app in there!).
> 
> * Test layers can be used to group tests that have a shared test setup
> and teardown in the context of a shared component registry that has been 
> initialized according to a *.zcml file that can be specified for 
> functional test layers (the setup and teardown and the component 
> registry initialization happen once for all the tests in grouped in that 
> layer).
> 
> * Functional test layers are classes, however, for some reason (probably 
> historical), they're still old style classes.
> 
> * Still, since they're classes functional test layers can inherit from 
> each other.
> 
> * The setup and teardown methods are just that, methods (not class 
> methods!).
> 
> * The testrunner __cannot__ on its own make sure that the setup methods 
> of the more generic layers are called before that to the specific 
> layers, nor the other way around for the teardown methods.
> 
> * You have to make sure all setups and teardown methods in the test 
> layer inheritance hierarchy are called in the correct order::
> 
>    class MyFunctionalLayer(zope.app.testing.functional.ZCMLLayer):
> 
>        def setUp(self):
>            super(MyFunctionalLayer, self).setUp()
>            # and now my stuff
> 
>        def tearDown(self):
>            # first my stuff
>            super(MyFunctionalLayer, self).tearDown()
> 
> * This difference in layer behaviour is annoyingly confusing.
> 
> 
> So far so good (albeit a bit annoying).
> 
> 
> The summary of using (functional)test layers in z3c.testsetup:
> ==============================================================
> 
> There're three "directives" that z3c.testsetup recognizes for setting up 
> test layers:
> 
> 1) :layer: <dotted.name.of.layer.def>
> 
> 2) :zcml-layer: <ZCML_filename>
> 
> 3) :functional-zcml-layer: <ZCML_filename>
> 
> Note that these layer-ish directives __only__ work in docttests!
> 
> 
> Ad 1):
> ------
> 
> This is the most "generically" useful directive. By specifying a dotted 
> name to the layer, the test is grouped in that layer.
> 
> It can point to either a "normal" layer class (i.e. of the "non 
> functional" kind). Alternatively it can point to a layer instance if it 
> is to be a functional layer. Compare:
> 
> layers.py:
> 
>    class MyLayer(object):
>      pass
> 
> 
>    class MyFunctionalLayer(ZCMLLayer):
>      pass
> 
>    my_functional_layer = MyFunctionalLayer(
>      '/path/to/testing.zcml', __name__, 'My Functional Layer')
> 
> 
> test_non_functional.txt:
> 
>    :layer: layers.MyLayer
> 
> 
> test_functional.txt:
> 
>    :layer: layers.my_functional_layer
> 
> Note that if the layer definition is an instance of ZCMLLayer (directly 
> or indirectly), the tests are suited up in a 
> zope.app.testing.functional.FunctioalDocFileSuite
> 
> Ad 2 and 3):
> ------------
> 
> The functionality of both :zcml-layer: and :functional-zcml-layer: can 
> be accomplished by using :layer:
> 
> However, in case you just want to use a common *.zcml file for a group 
> of tests and not need to provide a common setup and teardown you can use 
> these directives.
> 
> The directives' differences are subtle - and not yet completely clear to 
> me. In case of :functional-zmcl-layer:
> 
> * There're more convenience functions available in the test's globals. 
> (what extra functions are these?)

'sync', 'http', 'getRootFolder', the usual suspects from
zope.app.testung.functional.

> * Somehow the setUp and tearDown functions (what setUp and tearDown 
> functions?) are wrapped in something (in what and why?).

This is not the case any more, at least for tests registered with
[functional]-zcml-layer. When using the old test setup markers you got
your setup function be called by a function that also called
zope.app.testing.functional.FunctionalTestSetup().setUp().

Another confusion that arrived with the switch to the new markers :-/

> * Anything more?
> 
> 
> So far so... well... alright. The layer-ish directives were a source of 
> confusion for me and maybe for others too. I have couple of suggestions:
> 
> 
> Suggestions
> ===========
> 
> * The :zcml-layer: and :functional-zcml-layer: directives have a 
> slightly confusing name. They really only point to a zcml file, not to a 
> layer definition like :layer: does.
> 
> I can imagine that :zcml-file: and :functional-zcml-file: would've been 
> less confusing names.

Right. The reason for the name was that when using a 'zcml-file' or
'functional-zcml-layer' directive, you're supposed not to define any
additional layers as this does in fact define a layer for you. But
overall, I agree, it is more confusing.

> * The difference between :functional-zcml-layer: and :zcml-layer: is 
> really not all that obvious (at least to me). I wonder if we cannot just 
> have just one, with the behaviour of :functional-zcml-layer:?

Yeah, +1 from my side. Let's name it 'zcml-file'.

> Alternatively, the z3c.testsetup documentation could provides some 
> directions for choosing one over the other.
> 
> * z3c.testsetup should support the :layer: directives in unittests too.

You mean plain (non-doctest) unittests? I'd like to fiddle around with
them as little as possible. It should be possible to define a layer for
them in the code itself setting a ``layer`` attribute on the testsuite
or single tests.

> * The functional layers and "normal" layers should probably behave 
> similar. This is not something z3c.testsetup can fix. I'll post a 
> message about this on zope-dev.

Yes, it would be great if these two kinds could be more compatible.

> I hope my summary is correct and the suggestions make sense. I hope it 
> is somewhat complete as well as useful for others. I'd really like to 
> get feedback though.

Thank you very much for all the suggestions! I appreciate them very
much!

I think it's time to get rid of all the old stuff (the old markers) in
z3c.testsetup which makes things more complicated and blocks
readability. Cleaning up should lead to better understandable code and
behaviour. Doing this I could also rename the strange functional-layer
directives and reduce them to one: ':zcml-file:'.

I am still undecided about the default value of layer-teardown. Should
that be True, False or should we just use the default of zope.testing?
As there were recently some issues with it (again) I tend to reuse the
default of zope.testing.

Furthermore I am thinking about some (optional) manuel
(http://packages.python.org/manuel/) support. But that would need some
more work, maybe a redesign to make use of third-party testsetup code.

Best regards,

-- 
Uli

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Dies ist ein digital signierter Nachrichtenteil
Url : http://mail.zope.org/pipermail/grok-dev/attachments/20091021/7075a08c/attachment.bin 


More information about the Grok-dev mailing list