[cosmo-dev] steps to push new cosmo release out

[Travis Vachon]

>
> Bear, what's the current status of windmill tests performed by tbox?
>
> As part of validation, Adam would watch the windmill tests run to  
> see if anything weird happened, and also doing a manual run-through  
> of currently-not-working windmill targets like IE6.  I doubt Adam's  
> up for that anymore, so we either need to replace that whole round  
> of work or reduce the number of windmill runs that we watch.
>

I'd be happy to own the windmill testing process, but "watching for  
weird stuff" sounds totally infeasible. The windmill tests take at  
least 30 minutes to run on firefox last time I checked and are about  
as engaging as paint drying. Multiply that by 4 platforms and x number  
of failed runs that require re-runs and this quickly spirals out of  
control. In any case, "weird stuff" is almost impossible to nail down,  
even behavior that looks normal could easily be unexpected behavior.

Windmill outputs messages to the console on test failures that I  
suspect we could use to make windmill testing fully automatic. One  
problem with this is that we don't have asserts properly verifying  
many of the tests, but that's something we need to improve anyway. I'd  
suggest:

* Work on making windmill tests runs fully auto.
* Fix assert failures as we notice them missing.
* Devote some amount of time to an "assert audit" of the windmill  
tests. This would be a good opportunity to acquaint anyone interested  
with the test system.

-Travis