-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Too many never-resolve failures in our buildbots makes their output just noise: the amount of effort required to diagnose the cause of a failure seems to have no payoff if we don't get them each cleared up.
I'm just about done with trying to diagnose them, myself, which is making me sad, as I *want* to be pleased with the quality of the software our community produces.
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
Hello,
On Mon, 21 Mar 2011 11:56:05 -0400 you wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Too many never-resolve failures in our buildbots makes their output just noise: the amount of effort required to diagnose the cause of a failure seems to have no payoff if we don't get them each cleared up.
I'm just about done with trying to diagnose them, myself, which is making me sad, as I *want* to be pleased with the quality of the software our community produces.
I'd say revert back to the original procedure. The guilty dev has x days to fix the failure or the change gets reverted.
PS: And please please kick the guilty one's a**, not the buildbot maintainer's. It does not make much sense to disable tests just because they fail.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/22/2011 03:59 AM, Adam GROSZER wrote:
Hello,
On Mon, 21 Mar 2011 11:56:05 -0400 you wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Too many never-resolve failures in our buildbots makes their output just noise: the amount of effort required to diagnose the cause of a failure seems to have no payoff if we don't get them each cleared up.
I'm just about done with trying to diagnose them, myself, which is making me sad, as I *want* to be pleased with the quality of the software our community produces.
I'd say revert back to the original procedure. The guilty dev has x days to fix the failure or the change gets reverted.
PS: And please please kick the guilty one's a**, not the buildbot maintainer's. It does not make much sense to disable tests just because they fail.
Chameleon is an external dependency, not managed as part of the Zope repository; I presume that the authors / maintainers of the z3c.* packages chose to depend on it with full knowledge of the risks that entails. If a new release of Chameleon breaks z3c.*, then it is the z3c.* maintainers whose tails need to be motivated: they should either get the exteranl dependency fixed, or work around it in their own packages (e.g., by pinning their dependency to a "known good" version, or by updating to use the newer APIs).
Leaving the packages as "permanently failing" is obviously not doing anything to motivate those maintainers. Leaving the board red is *de-motivating* to the community at large, who have to wade through failure reports for (apparently) unmaintained packages daily while trying to diagnose stuff which might have been broken by changes made the day before.
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
* Tres Seaver tseaver@palladion.com [2011-03-22 14:18]:
On 03/22/2011 03:59 AM, Adam GROSZER wrote:
And please please kick the guilty one's a**, not the buildbot maintainer's. It does not make much sense to disable tests just because they fail.
Leaving the packages as "permanently failing" is obviously not doing anything to motivate those maintainers. Leaving the board red is *de-motivating* to the community at large, who have to wade through failure reports for (apparently) unmaintained packages daily while trying to diagnose stuff which might have been broken by changes made the day before.
I have to admit, I'm ignoring the buildbots completely at this time, (even though I think that having them in place could be very valuable), since all I perceive from them is failure-noise, and no trend of it getting less, at all, over the last few months. (That's my gut feeling, not based on research. It still makes me very much want to ignore them.) So, I agree with Tres: this situation is quite demotivating.
But I also agree with Adam: I think the value of the buildbots is that they run tests, so disabling tests because they fail feels a little like closing our eyes shut because we don't want to see we're going to be falling off a cliff.
I don't have much of an idea how to proceed, except maybe explicitly reducing the scope of the buildbots to a set of packages people around here care enough about that they want and will keep them green.
Wolfgang
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/23/2011 02:40 AM, Wolfgang Schnerring wrote:
- Tres Seaver tseaver@palladion.com [2011-03-22 14:18]:
On 03/22/2011 03:59 AM, Adam GROSZER wrote:
And please please kick the guilty one's a**, not the buildbot maintainer's. It does not make much sense to disable tests just because they fail.
Leaving the packages as "permanently failing" is obviously not doing anything to motivate those maintainers. Leaving the board red is *de-motivating* to the community at large, who have to wade through failure reports for (apparently) unmaintained packages daily while trying to diagnose stuff which might have been broken by changes made the day before.
I have to admit, I'm ignoring the buildbots completely at this time, (even though I think that having them in place could be very valuable), since all I perceive from them is failure-noise, and no trend of it getting less, at all, over the last few months. (That's my gut feeling, not based on research. It still makes me very much want to ignore them.) So, I agree with Tres: this situation is quite demotivating.
But I also agree with Adam: I think the value of the buildbots is that they run tests, so disabling tests because they fail feels a little like closing our eyes shut because we don't want to see we're going to be falling off a cliff.
I don't have much of an idea how to proceed, except maybe explicitly reducing the scope of the buildbots to a set of packages people around here care enough about that they want and will keep them green.
Yup. That is why I was asking to suspend the 'z3c.*' packages which are failing: they haven't passed in months, and nobody seems motivated to fix them.
Although they are in ragged shape at the moment, I *do* work to diagnose / fix failures in the ZTK and Zope2 tests.
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
On Mon, Mar 21, 2011 at 11:56:05AM -0400, Tres Seaver wrote:
Too many never-resolve failures in our buildbots makes their output just noise: the amount of effort required to diagnose the cause of a failure seems to have no payoff if we don't get them each cleared up.
+100 to any method of reducing that noise. Continuously failing buildbots shouldn't make the system unusable.
Hello,
On Tue, 22 Mar 2011 17:20:05 +0100 you wrote:
On Mon, Mar 21, 2011 at 11:56:05AM -0400, Tres Seaver wrote:
Too many never-resolve failures in our buildbots makes their output just noise: the amount of effort required to diagnose the cause of a failure seems to have no payoff if we don't get them each cleared up.
+100 to any method of reducing that noise. Continuously failing buildbots shouldn't make the system unusable.
Well I did some... Pinned z3c.pt and Chameleon versions. Others are welcome too.
* Tres Seaver tseaver@palladion.com [2011-03-21 16:56]:
Too many never-resolve failures in our buildbots makes their output just noise: the amount of effort required to diagnose the cause of a failure seems to have no payoff if we don't get them each cleared up.
On a tangentially-related note, what happened to the proposed new summarizer formatting? Half a year ago (or whenever), I got the impression it was going to be ready "in a few days", but it seems it never happened. What is needed to get this switched?
(I'm not saying dressing the continuous failures up nicer is a solution to the problem Tres is presenting, but I do think the noisy format contributes to the confusion about the buildbots)
Wolfgang
Hello,
On Wed, 23 Mar 2011 07:43:26 +0100 you wrote:
- Tres Seavertseaver@palladion.com [2011-03-21 16:56]:
Too many never-resolve failures in our buildbots makes their output just noise: the amount of effort required to diagnose the cause of a failure seems to have no payoff if we don't get them each cleared up.
On a tangentially-related note, what happened to the proposed new summarizer formatting? Half a year ago (or whenever), I got the impression it was going to be ready "in a few days", but it seems it never happened. What is needed to get this switched?
(I'm not saying dressing the continuous failures up nicer is a solution to the problem Tres is presenting, but I do think the noisy format contributes to the confusion about the buildbots)
You mind walking over to Theuni and asking him? ;-)
On 03/23/2011 08:45 AM, Adam GROSZER wrote:
Hello,
On Wed, 23 Mar 2011 07:43:26 +0100 you wrote:
- Tres Seavertseaver@palladion.com [2011-03-21 16:56]:
Too many never-resolve failures in our buildbots makes their output just noise: the amount of effort required to diagnose the cause of a failure seems to have no payoff if we don't get them each cleared up.
On a tangentially-related note, what happened to the proposed new summarizer formatting? Half a year ago (or whenever), I got the impression it was going to be ready "in a few days", but it seems it never happened. What is needed to get this switched?
(I'm not saying dressing the continuous failures up nicer is a solution to the problem Tres is presenting, but I do think the noisy format contributes to the confusion about the buildbots)
You mind walking over to Theuni and asking him? ;-)
Dang! I need to really change my lair now ...
Hello,
* Adam GROSZER agroszer@gmail.com [2011-03-23 08:45]:
On Wed, 23 Mar 2011 07:43:26 +0100 you wrote:
On a tangentially-related note, what happened to the proposed new summarizer formatting? Half a year ago (or whenever), I got the impression it was going to be ready "in a few days", but it seems it never happened. What is needed to get this switched?
You mind walking over to Theuni and asking him? ;-)
Oh. Right. Sure.
*walks over and back*
So. Theuni says, the code[1] (based on the current aggregator script) is probably pretty much ready for deployment. This is a script to be run by cron that scrapes the HTML list archives of the zope-test list. I haven't looked at it yet, so I don't know how ready "probably pretty much" actually means.
If I understood this correctly, the current aggregator runs courtesy of Stephan Holek. I guess it would be a good idea to get this somewhere closer to the Zope Foundation, so I'd rather we work something out with Jens Vagelpohl how/where to run this than throw it up on one of our boxes here at gocept.
Adam, do you want to tackle this? Otherwise, I can do it, but I don't know when I get enough round toits to make it happen.
Wolfgang
[1] http://zope3.pov.lt/trac/browser/Sandbox/ctheune/testsummarizer
Hello,
On Thu, 24 Mar 2011 10:51:44 +0100 you wrote:
Hello,
- Adam GROSZERagroszer@gmail.com [2011-03-23 08:45]:
On Wed, 23 Mar 2011 07:43:26 +0100 you wrote:
On a tangentially-related note, what happened to the proposed new summarizer formatting? Half a year ago (or whenever), I got the impression it was going to be ready "in a few days", but it seems it never happened. What is needed to get this switched?
You mind walking over to Theuni and asking him? ;-)
Oh. Right. Sure.
*walks over and back*
So. Theuni says, the code[1] (based on the current aggregator script) is probably pretty much ready for deployment. This is a script to be run by cron that scrapes the HTML list archives of the zope-test list. I haven't looked at it yet, so I don't know how ready "probably pretty much" actually means.
If I understood this correctly, the current aggregator runs courtesy of Stephan Holek. I guess it would be a good idea to get this somewhere closer to the Zope Foundation, so I'd rather we work something out with Jens Vagelpohl how/where to run this than throw it up on one of our boxes here at gocept.
Adam, do you want to tackle this? Otherwise, I can do it, but I don't know when I get enough round toits to make it happen.
Wolfgang
[1] http://zope3.pov.lt/trac/browser/Sandbox/ctheune/testsummarizer
I'm rather busy nowadays. But it seems like it's about bugging Stephan Holek to stop the current one and bugging Jens to start the new one, or? Unless the script is broken.
Could you run that script -- worst case we'll have 2 mails for a day -- for testing? Seems like it has the settings for gocept and I don't really have an SMTP server here handy.
Hi,
* Adam GROSZER agroszer.ll@gmail.com [2011-03-29 11:15]:
On Thu, 24 Mar 2011 10:51:44 +0100 you wrote:
So. Theuni says, the code[1] (based on the current aggregator script) is probably pretty much ready for deployment. This is a script to be run by cron that scrapes the HTML list archives of the zope-test list. I haven't looked at it yet, so I don't know how ready "probably pretty much" actually means. [1] http://zope3.pov.lt/trac/browser/Sandbox/ctheune/testsummarizer
If I understood this correctly, the current aggregator runs courtesy of Stephan Holek. I guess it would be a good idea to get this somewhere closer to the Zope Foundation, so I'd rather we work something out with Jens Vagelpohl how/where to run this than throw it up on one of our boxes here at gocept.
Adam, do you want to tackle this? Otherwise, I can do it, but I don't know when I get enough round toits to make it happen.
I'm rather busy nowadays. But it seems like it's about bugging Stephan Holek to stop the current one and bugging Jens to start the new one, or? Unless the script is broken.
Could you run that script -- worst case we'll have 2 mails for a day -- for testing? Seems like it has the settings for gocept and I don't really have an SMTP server here handy.
I'd really rather not duct-tape this... Alright, then I'll try and get it into running shape and bug people to run it. And unfortunately the code looks like it needs some work before it can be deployed properly.
Wolfgang
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 3/29/11 11:15 , Adam GROSZER wrote:
But it seems like it's about bugging Stephan Holek to stop the current one and bugging Jens to start the new one, or? Unless the script is broken.
Could you run that script -- worst case we'll have 2 mails for a day -- for testing? Seems like it has the settings for gocept and I don't really have an SMTP server here handy.
Thanks to Wolfgang's cleanup work the new script is now in place. It's running once a day at 01:00 AM Eastern Standard Time.
jens
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 4/3/11 12:41 , Jens Vagelpohl wrote:
On 3/29/11 11:15 , Adam GROSZER wrote:
But it seems like it's about bugging Stephan Holek to stop the current one and bugging Jens to start the new one, or? Unless the script is broken.
Could you run that script -- worst case we'll have 2 mails for a day -- for testing? Seems like it has the settings for gocept and I don't really have an SMTP server here handy.
Thanks to Wolfgang's cleanup work the new script is now in place. It's running once a day at 01:00 AM Eastern Standard Time.
@Wolfgang: Something is not working as seen in today's run. If I run the script with "-T 2011-04-02" I am getting correct output. But if I run it with "-T 2011-04-03" or "-T 2011-04-04" I am getting nothing. Can you test this in your sandbox?
Thanks!
jens
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 4/4/11 08:37 , Jens Vagelpohl wrote:
On 4/3/11 12:41 , Jens Vagelpohl wrote:
On 3/29/11 11:15 , Adam GROSZER wrote:
But it seems like it's about bugging Stephan Holek to stop the current one and bugging Jens to start the new one, or? Unless the script is broken. Could you run that script -- worst case we'll have 2 mails for a day -- for testing? Seems like it has the settings for gocept and I don't really have an SMTP server here handy.
Thanks to Wolfgang's cleanup work the new script is now in place. It's running once a day at 01:00 AM Eastern Standard Time.
@Wolfgang: Something is not working as seen in today's run. If I run the script with "-T 2011-04-02" I am getting correct output. But if I run it with "-T 2011-04-03" or "-T 2011-04-04" I am getting nothing. Can you test this in your sandbox?
@Wolfgang: I have checked in some fixes and it works for me now. We'll see the results tomorrow.
jens
-
Adam GROSZER -
Adam GROSZER -
Brian Sutherland -
Christian Theune -
Jens Vagelpohl -
Tres Seaver -
Wolfgang Schnerring