[cosmo-dev] Windmill Test Coverage

[Bobby Rullo]

Since no one has commented, edited or added to that page let me  
explain my motivations for that post a bit more:

Cosmo needs more test coverage, and quickly more than QA can churn out  
in the time it's needed. So, Cosmo developers are going to add  
Windmill tests.

My problem is, unless it's a brand new feature, it's really difficult  
to see what has been tested and what needs to be tested - the only  
document which has this information is AFAIK is Adam's recent email  
posting, which is a good start.

I would prefer to see something easier to grab than an email,  
something at a permanent location that can be updated. This could be a  
wiki page or even a text file checked into the cosmo source tree.

Also, what Adam's email doesn't do is say exactly what is being  
validated, or how the tests work, it just mentions the functionality  
being tested. We have no idea however, the depth of testing of that  
functionality. The only thing we have to go on then is the tests  
themselves. Windmill tests don't make for easy reading (no disrespect  
to Windmill, that's just the nature of the beast), so I think we  
really need to comment all the tests so we know what they are doing.  
See for example:

http://svn.osafoundation.org/server/cosmo/trunk/cosmo/src/test/functional/windmill/canvas/event/event_status_enable_disable.json

The one thing I would've done differently in this case is put a  
comment at the top of the file describing what the file tests.

Another thing that would make it easier for developers to write tests  
is too have a list of specific tests that should be written - the more  
specific the better. Entries should look like "Ensure that when moving  
an event from one week to the next that the data has really moved" or  
"Ensure that when unstamping eventness, the event is no longer on the  
canvas". Note that these examples say exactly what should be asserted.

I'm not saying that it's not helpful for someone to point out that "we  
need more coverage in area XXXX", just that it's not sufficient. In an  
(my) ideal world, there'd be a list of these tests and devs could just  
go in and claim them as their own.

This may seem like more work for QA but it's actually less. Here's  
why: I submit that in the same time it takes Adam (for example) to  
write a windmill test, he could come up with (at least) ten specfic  
tests that need to be written. Those tests could then be written be  
developers (and volunteers! this is a great way for people to get  
involved, and giving people a list of tasks really lowers the barrier  
to entry). So in the time that it took him to write that one windmill  
test he effectively was responsible for ten getting created.

It's also less work for us devs - we spend less time rummaging through  
the tests figuring out what they do and tearing our hair out over what  
to do next.

So in summation: if you tell us exactly what to do, and what has been  
done, it's a lot easier for us to do it and everyone saves time. Also,  
comments are good. Yay!

bobby


On Nov 7, 2007, at 2:37 PM, Bobby Rullo wrote:

> In an effort to get the tests we need written and to avoid  
> duplication of work, I've created a page which we can use to  
> remember what areas need Windmill coverage. It's very crude right  
> now, so feel free to do whatever is necessary to make it more useful.
>
> see: http://chandlerproject.org/Developers/WindmillTestCoverage
>
> bobby
> _______________________________________________
> cosmo-dev mailing list
> cosmo-dev at lists.osafoundation.org
> http://lists.osafoundation.org/mailman/listinfo/cosmo-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osafoundation.org/pipermail/cosmo-dev/attachments/20071108/e9587acb/attachment.htm