[cosmo-dev] steps to push new cosmo release out
travis at osafoundation.org
Thu Jan 17 13:19:11 PST 2008
> Bear, what's the current status of windmill tests performed by tbox?
> As part of validation, Adam would watch the windmill tests run to
> see if anything weird happened, and also doing a manual run-through
> of currently-not-working windmill targets like IE6. I doubt Adam's
> up for that anymore, so we either need to replace that whole round
> of work or reduce the number of windmill runs that we watch.
I'd be happy to own the windmill testing process, but "watching for
weird stuff" sounds totally infeasible. The windmill tests take at
least 30 minutes to run on firefox last time I checked and are about
as engaging as paint drying. Multiply that by 4 platforms and x number
of failed runs that require re-runs and this quickly spirals out of
control. In any case, "weird stuff" is almost impossible to nail down,
even behavior that looks normal could easily be unexpected behavior.
Windmill outputs messages to the console on test failures that I
suspect we could use to make windmill testing fully automatic. One
problem with this is that we don't have asserts properly verifying
many of the tests, but that's something we need to improve anyway. I'd
* Work on making windmill tests runs fully auto.
* Fix assert failures as we notice them missing.
* Devote some amount of time to an "assert audit" of the windmill
tests. This would be a good opportunity to acquaint anyone interested
with the test system.
More information about the cosmo-dev