[Zope-dev] Automating tests of the ZTK / zopeapp package sets

[Christian Theune]

Hi,

I also looked at Hudson a few days ago and found it to be awesome. :)

On 03/04/2010 10:10 PM, Tres Seaver wrote:
> After successfully configuring the Hudson[1] continuous integration
> server yesterday to test the various repoze.* packages[2], I thought I
> would experiment with using it to drive the compatibility tests for ZTK
> and zopeapp.  Here is what I did:
> 
> In a shell on my server:
> 
>  $ wget http://hudson-ci.org/latest/hudson.war
>  $ java -jar hudson.war --httpPort=19999

That's what blew me off my chair. :)

>  ....
> 
> In the browser, visit 'http://laguna.palladion.com:19999, and configure
> the server:
> 
> - Enable security
>   - allow signup
>   - using Matrix
>   - set up 'tseaver' as having all permissions
>   - set up 'anonymous' as having 'Read' permission overall and for jobs
> 
> - Register as 'tseaver'
>   - Turn off signup (no, I don't want you guys running arbitrary
>     shell scripts on my server ;).
> 
> - Create a job, 'ztk-trunk-py26':

See below
> 
>   http://laguna.palladion.com:19999/job/ztk-trunk-py26/
> 
>   - Use SVN, pointed at
>    'svn://svn.zope.org/repos/main/zopetoolkit/trunk'
> 
>     - Set to poll SVN every 15 minutes.
> 
>     - Enable using 'svn up'
> 
>   - Added build step as a shell script::
> 
>     cd trunk
>     /opt/Python-2.6.4/bin/python bootstrap.py
>     bin/buildout
>     bin/test-ztk
>     bin/test-zopeapp
> 
>   - Ran a build manually.  The first one barfed due to a typo
>     in my script, but the second one ran, taking 21 minutes.

Right. That was my experience as well. Also a good thing is that you can
now take those as templates, or - if you like - generate those XML files
into the filesystem for new builds and ping the server to reload those.

> The test runs successfully to completion, but doesn't produce statistics
> in a form that Hudson understands.  The following changes to
> 'zope.testing.testrunner' would make the output more informative:
> 
>  - Add a flag like the '--with-xunit' flag to the Nose testrunner, which
>    dumped results into a JUnit-compatible XML file[3].
> 
>  - Add a flag like the '--with-xcompat' flag to the Nose testrunner
>    (when run with the 'nosexcover' plugin), producing a Cobertura-
>    compabtible XML file[4].

Right. However Hudson does understand failures/successes at least. I
made a job for the ZODB where the individual steps (checkout,
bootstrap/buildout, test) where separated steps so that made the UI
integration a little bit better.

> At that point, the build process would be nicely automated, with
> browsable results including coverage, pretty graphs, etc.  See the
> 'repoze' jobs for examples of those outputs.

Here's another awesome part:

  You can configure matrix build jobs which would allow us to define a
  single job for i.e. ZODB and then say "we want this tun run in the
  combinations of Unix/Windows, 32/64, Python 2.4/2.5/2.6, trunk/branchx
  /branchy" and it goes off automatically trying to find build nodes
  that can provide all combinations.

It took my about 10 minutes to get that going including the installment
of the Windows builder.

> Questions for discussion:
> 
> - I find the prettier interface and easier setup than buildbot worth
>   running a 3rd-party Java app (rather than a 3rd party Python app).
>   Would that be acceptable among the folks running our automated tests?

It would be for me. Right now I'll continue to clean up what we have
with buildbot. I really don't want to put anybody off. Getting more
coverage (however) and making it more visible and regular is my primary
goal. I don't think we have to through anything out. I'll be happy to
provide even more tests using other build systems in parallel.

> - Is it worth adding the XML support for test results and coverage to
>   'zope.testing.testrunner'?

Maybe. I'm happy hacking on the testrunner. :)

> - Would people be willing to run Hudson "permanently" on enough hosts
>   to make this a reasonable replacement for buildbot (we need Windows,
>   too)?

I'm definitely willing to run it for 32/64 bit Linux machines. I'm short
on Windows licenses but Hudson seems easy to integrate securely
distributed over multiples sites and easy enough to set up with the
JNRE/service installer.

> - Hudson seems general purpose enough to allow automating other code-
>   related tasks, e.g., production of dependency graphs.  Maybe we could
>   even automate building Windows installers?

How awesome that would be. :)

> I will try to leave the ZTK Hudson server up for a little bit, to allow
> folks to see what it can deliver already.
> 
> [1] http://hudson-ci.org/
> 
> [2] http://hudson.repoze.org/
> 
> [3] http://old.nabble.com/schema-for-junit-xml-output-td22193385.html
> 
> [4] http://cobertura.sourceforge.net/xml/coverage-01.dtd

Thanks a lot for the effort. I'm really excited about Hudson and I'm
willing to help out here.

Christian

-- 
Christian Theune · ct at gocept.com
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development