[Cosmo-dev] Requirements for Windmill tests running on tinderbox
mikeal at osafoundation.org
Mon Jun 4 15:46:14 PDT 2007
> 1. Developers should be running the Windmill tests before the check
> in major functionality changes -- this is independent of Tinderbox
> 2. A Tinderbox run is schedule every time a developer commits.
> That Tinderbox run should run the Windmill tests. If that Windmill
> run fails, then something in the associated checkin is the likely
> culprit - tracking that kind of stuff down doesn't seem like
> something that ought to require managerial intervention. Besides,
> someday the person checking in might not even work at OSAF, at
> which point getting managers involved won't help much.
I know how things work on Chandler but I'm not sure that the exact
same process will work well for Cosmo.
The windmill tests are more brittle by nature than the Chandler tests
and when the tests break they require more test level changes than
the Chandler tests do. The UI layout and architecture also changes
much more rapidly than on the Chandler side and adding the burden of
fixing broken tests because of major functionality changes will load
up the cosmo UI development resource(s) with much more work than you
think. As an example, with the latest look at 0.7 Adam anticipates
that 30-50% of the tests will need to be changed, and estimates only
a day to fix them himself.
Because the tests change so dramatically through each release ( this
is in reaction to the UI code changing dramatically during each
release ) I don't think it's fair to put the burden of fixing the
tests on the developers until the end of the cycle. When checking in
"major functionality changes" the tests surrounding that
functionality need to be modified and in some cases, rewritten. Adam
is very efficient at fixing these tests and I don't think he'll be a
bottleneck for fixing them.
I agree that we don't want managerial intervention if the tests
break, but expanding something like the process we have now seems
like the best solution.
The current process is something like this;
-mde runs the tests towards the end of our release cycle when he's
fixing bugs before he checks in.
-Adam kicks off the tests every time he starts testing new build
How I see the process improving is;
-mde, travis, and bobby kick off the tests manually toward the end of
the release cycle when fixing bugs before checkin.
-Adam watches tinderbox to make sure the tests are always passing
during the entire cycle, fixing them when necessary.
For as long as I've worked on cosmo we've destabilized the entire
trunk at the beginning of every release and slowly stabilizing it
while more features and fixes drop, and QA changes many of the tests
to work in the new code base. With windmill modifying and debugging
these tests now takes about 20% of the time it did when we used
Selenium but it's still a hit and it's still easier and faster for QA
to make the larger changes. I don't think the burden of fixing the
tests should be on the developers until the end of the release cycle
when we are in bug fix mode, at that point Adam would become a
bottleneck and tests are most likely failing due to valid bugs and
not architectural or functional changes.
This is obviously a contentious issue that we'll be discussing
further. At this point I don't see anyone objecting to the task of
getting them in to tinderbox, just with the process surrounding how
we manage and handle the results of the tests being in tinderbox, and
running manually. I'm going to move forward and log a task for bear
to add windmill tests to tinderbox today and we can discuss this
process more in the coming days.
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://lists.osafoundation.org/pipermail/cosmo-dev/attachments/20070604/4b3282aa/PGP.pgp
More information about the cosmo-dev