[Zope-dev] KGS trunk without failures

[Hanno Schlichting]

On Sun, Aug 2, 2009 at 6:34 PM, Jim Fulton<jim at zope.com> wrote:
> 2. Some of the tests only pass if run separately, due to test
> interactions. Presumably, this means that other tests aren't cleaning
> up after themselves.  I think we need a standard automated way to run
> each package's tests separately.  I think some folks are working on
> that.

I had a similar problem with the Plone trunk tests and my solution to
that is http://pypi.python.org/pypi/plone.recipe.alltests (licensed as
ZPL).

My example usage of it is at
http://svn.plone.org/svn/plone/buildouts/plone-coredev/trunk/tests.cfg

The main idea here is to generate a "./bin/alltests" script, which
internally runs the tests on a package by package basis (using a
simple os.system call to the zc.recipe.testrunner test script). It
parses the test result and exists at the end with a status code of 0
if there where no test failures at all or 1 otherwise, making it
buildbot friendly.

Generally it will try to run the tests for all packages in the working
set and add the package to the test-path automatically. The exclude
option allows to exclude packages from being tested. For example if
those do not include their tests in the released distribution or may
not be compatible with zope.testing. The assumption here is that you
want to test everything you have with an opt-out. If some package
pulls in a new dependency that dependency will be tested too.

The other advanced option is to use a "groups" definition, which
allows to put multiple packages into one group and then run all of
them in one run (internally using ./bin/test -s foo -s bar). This is
basically to avoid test layer startup cost for similar packages which
are known to behave well when run together.

This stuff is fairly new and might be missing features. I'd welcome
feedback and patches ;)

Hanno