[Zope-dev] Test runner: layers, subprocesses, and tear down

[Marius Gedminas]

On Thu, Jul 03, 2008 at 05:22:11PM -0400, Benji York wrote:
> I'm working on making the zope.testing test runner run tests in
> parallelized subprocesses.  The option will likely be spelled -j N,
> where N is the maximum number of processes.

That's wonderful news!

> I have it basically working, but have noticed a couple odd corners of
> the test runner that I'd like to clean up.  They may be controversial,
> so I'll ask about them here first.
> 
> I'd like to 1) remove the layer tear-down mechanism entirely, and 2)
> make (almost) all layers run in a subprocess.

-1 in general.  +1 if you do that only for -j N where N > 1.

Running all the tests in a single process has the following benefits:

  * test coverage analysis produces results that are correct (well,
    correct often enough -- but it has no chance at all when the test
    runner forks a subprocess)

  * import pdb; pdb.set_trace() works

> I want to do #1 because it would simplify the test runner code and no
> one seems to be using the functionality anyway.

That's news to me.  A while ago I went through Zope 3 trunk (it was
pre-eggsplosion IIRC) and made sure all test layers defined in it
supported teardown.

Granted, FunctionalTestLayer() has allow_teardown=False as the default,
for two reasons:

  * backwards compatibility: in olden days functional test layers didn't
    support teardown

  * paranoia: it is in general impossible to determine whether calling
    CleanUp().cleanUp() will correctly clear all the global state
    (someone could easily write a custom ZCML directive that changed a
    global variable and forget to register a CleanUp hook), so
    disallowing teardowns were the conservative safe choice.

It is entirely my fault that I haven't evangelized the
allow_teardown=True option for creating new test layers.

> It also appears from
> reading the code that any tests run in a subprocess (and most are) will
> never exercise the tear-down mechanism anyway.

I guess that's fine for process state, but not so fine for external
state (temporary files etc.).

Hey, this might explain why SchoolTool's tests tend to fill up my
buildbot's /tmp without cleaning up after themselves!  I'll have to
investigate some day.

> #2 will add some process start-up overhead, but it'll only be one more
> process than is already started (and any reasonably-sized test corpus
> already starts several processes for each test run).  The one exception
> is for running with -D or with a pdb.set_trace() embedded in the code
> under test.  For that case we need a switch to say "don't start any
> subprocesses at all", I suspect that will be spelled -j0.

If that case needs to be supported anyway, what's the advantage of
spawning exactly one subprocess when you run it with -j 1?  I would also
question whether pdb-unfriendly non-performance-enhancing option should
be the default.

> For motivation, some speed comparisons: running a particular test suite
> with 3876 tests (mostly doctests, and mostly functional) without the
> patch takes 6 minutes, 42 seconds; my branch runs the same tests in 3
> minutes and 22 seconds (give or take) on a dual-core box with 3
> simultaneous subprocesses.

I know; for large test suites (by "large" I mean 40 minutes) I've been
using an ugly hack (--odd/--even test filtering) that lets me use both
CPUs if I manually run two instances of the test runner in two xterms in
parallel.

Regards,
Marius Gedminas
-- 
"Wipe Info uses hexadecimal values to wipe files. This provides more security
than wiping with decimal values."
        -- Norton SystemWorks 2002 Manual
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: Digital signature
Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080704/edfefc23/attachment.bin