[Grok-dev] grokproject configures z3c.testsetup instead of grok.testing

[Uli Fouquet]

Hi there,

first thank you all very much for the encouraging notes regarding
z3c.testsetup 0.3 :-)

Kevin Teague wrote:
> The default 'tests.py' is:
> 
> import os.path
> import z3c.testsetup
> import sampleapp
> from zope.app.testing.functional import ZCMLLayer
> 
> 
> ftesting_zcml = os.path.join(
>     os.path.dirname(sampleapp.__file__), 'ftesting.zcml')
> FunctionalLayer = ZCMLLayer(ftesting_zcml, __name__,
> 'FunctionalLayer',
>                             allow_teardown=True)
> 
> test_suite = z3c.testsetup.register_all_tests('sampleapp')
> 
> This is functionaly equivalent to:
> 
> import grok
> test_suite = grok.testing.register_all_tests('sampleapp')
> 
> The only difference being that the layer_name is 'FunctionalLayer'
> versus 'GrokFunctionalLayer'. grok.testing.register_all_tests()
> takes additional *args and **kwargs as input and passes them to
> z3c.testsetup.TestCollector constructor, and will also respectfully
> allow you to override the 'zcml_config' and 'layer_name' settings.
> 
> To further confuse the situation, z3c.testsetup will fallback to
> using a package's 'ftesting.zcml' file as the default zcml_config.
> And if no 'layer_name' is specified, then it falls back to
> 'FunctionalLayer' for the layer_name. In fact, because
> grok.testing.register_all_tests respects settings overrides, this
> default name in z3c.testsetup overrides the 'GrokFunctionalLayer' one
> specificed in grok.testing! In fact, the defaults for z3c.testsetup
> are
> sufficient that one could put in tests.py:
> 
> import z3c.testsetup
> import sampleapp
> 
> # The functional layer is configured by default to read it's
> # ZCML configuration from the package's 'ftesting.zcml' file.
> test_suite = z3c.testsetup.register_all_tests('sampleapp')

I agree. You could also skip the 'import sampleapp' line here.

> And get the same thing - with the exception of the allow_teardown
> setting,
> (which normally defaults to False). This means that
> grok.testing.register_all_tests() isn't doing anything other than
> being
> another place to access the same function.

Yes, basically. grok.testing.register_all_tests sets an additional
fallback location for finding 'ftesting.zcml' (as you said). But this
could also be done in stock z3c.testsetup (and should IMO).

> So the question is, is there any point in having a
> grok.testing.register_all_tests()? Should this just be marked as BBB
> and we should all just use z3c.testsetup.register_all_tests()? Or
> should
> grok.tessting.register_all_tests() just be updated so that it creates
> the
> default functional layer with allow_teardown=True, and then the
> default
> tests.py for a Grok-based package can simply be:
> 
> import grok
> 
> # The functional layer of the test suite will use the 'ftesting.zcml'
> file
> # in this package for it's ZCML configuration.
> test_suite = grok.testing.register_all_tests('sampleapp')

What I get from the discussion here is that the most confusing points
are

* should I use grok.testing.register_all_tests or
  z3c.testsetup.register_all_tests?

* layer setup.

Let's see, what we can do here.

grok.testing or z3c.testsetup?
------------------------------

The main point behind having 'grok.testing.register_all_tests' was to
have an easier testsetup environment available with stock Grok. Import
grok and forget about complicately named package imports. I think this
is still a valid consideration.

My idea to have a (slight) different setup with grok.register_all_tests
than with z3c.testsetup turned out to be not very good. It complicates
things and should be removed. So I'd opt for making
'grok.testing.register_all_tests' an alias for
'z3c.testsetup.register_all_tests'. It then doesn't matter what you
import in the beginning, but 'grok' would certainly be easier to
remember.

Setting up Layers
-----------------

Layers are a powerful tool of the Zope testing machinery. They were not
very well supported with z3c.testsetup < 0.3. You could only define one
ZCML file or ZCML based layer which was used only for functional tests,
but all thereof. Furthermore for z3c.testsetup 'layer' alway meant
'ZCML-layer' that is setup using zope.app.testing.functional although
there are also non-ZCML layers and ZCML layers that do not need a fully
fledged ZODB running in background (okay, the latter is rare).

With z3c.testsetup I think this situation has improved: you can define a
layer for every doctest file separately, be it a functional (ZCML)
layer, a simple unittest layer or a ZCML layer that only does ZCML stuff
and does not setup a complete ZODB DemoStorage.

This raises the question: do we really need a default layer? From
looking through zope.app.testing I learned, that it can be quite
cumbersome to look in different locations for 'ftesting.zcml' and the
results might be unexpected. Sometimes it is like pure guessing.

From what I learned from developer feedback, I'd now tend to opt for
skipping the default layer and instead raise an error if one is needed
but not supplied.

In `grokproject` we could provide a locally defined ZCML-layer (as we
already do) which would be included in generated projects and (this time
_explicitly_) be used by local (functional) doctests.

I admit that this could mean more setup code but people could see how to
setup their own layer from the example code created (and the code would
be really used).

And as we are changing everything we could also switch to the new
testsetup syntax, so that you could setup a functional test like this::

  My Sample Tests
  ===============

  :doctest:
  :functional-zcml-layer: ftesting.zcml

     >>> getRootFolder()
     <zope.app.folder.folder.Folder object at 0x...>

With this registration you could use Kevins ultra short setup code::

   # tests.py
   import grok
   test_suite = grok.testing.register_all_tests('sampleapp')

or you could setup your functional tests like this::

  My Sample Tests
  ===============

  :doctest:
  :layer: sampleapp.tests.FunctionalLayer
  :setup: sampleapp.tests.setup
  :teardown: sampleapp.tests.teardown

     >>> getRootFolder()
     <zope.app.folder.folder.Folder object at 0x...>

which would require the complete layer setup beside the
`FunctionalLayer` definition in tests.py::

  # tests.py
  import os.path
  import grok
  import sampleapp
  from zope.app.testing.functional import (ZCMLLayer,
                                           FunctionalTestSetup)

  globs = z3c.testsetup.functional.doctesting.FunctionalDocTestSetup.globs

  def setup(test):
      FunctionalTestSetup().setUp()
      test.globs = globs
      return

  def teardown(test):
      return FunctionalTestSetup().tearDown()

  ftesting_zcml = os.path.join(
     os.path.dirname(sampleapp.__file__), 'ftesting.zcml')

  FunctionalLayer = ZCMLLayer(ftesting_zcml, __name__,
                              'FunctionalLayer',
                              allow_teardown=True)

  test_suite = grok.testing.register_all_tests('sampleapp')

A bit too much for the beginning, if you ask me (and not really needed
for the tests done). So I would opt for having a default tests.py like
the short one above and maybe we should add a link to the (then
finished) grok testing documentation for people that want to create more
complicated stuff.

What do you think?

Best regards,

-- 
Uli

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Dies ist ein digital signierter Nachrichtenteil
Url : http://mail.zope.org/pipermail/grok-dev/attachments/20090310/e3d0d16b/attachment-0001.bin