[cosmo-dev] Collecting and Reporting work for cosmo windmill and protocol tests

[Mikeal Rogers]

The windmill and protocol tests both use a framework i wrote in my own  
time called functest. As our windmill test coverage has expanded we've  
given ourselves new problems that require fixing in the framework  
through different plugin mechanisms.

The first and probably most obvious one is that windmill json tests  
cannot be run along side jstests or python tests in a reliable way.  
The second obvious problem is that you can't run a single windmill  
test, you must run the entire set, or at least most of it. The third  
and slightly less obvious one is that windmill tests depending on  
their ordering leads to some tests growing too large for a single file  
and if broken in to smaller pieces would be more manageable and give  
more transparency to the amount of coverage we have.

For the last week I've been working on improving the windmill  
collector system so that we can define a custom collector that can  
pull in .json and jstest tests. This will also benefit from functests  
hierarchal dependency system. This means that tests that assume a  
certain state or set of setup/teardown steps can be grouped into  
module hierarchies. This way the tests can still be run independently  
and don't depend on any ordering, just the setup/teardown steps  
defined in the preceding modules.

This work is almost complete but as such isn't quite documented yet.  
You can however see how the pure python collector inside of functest  
works.

http://functest.pythonesque.org/trac/wiki/FunctestTestWriting#ModuleHierarchies

I've also spent some time thinking about a more distributed way for  
developers, testers and individual contributors to run tests and  
report the results. As the cosmo team has started using a more  
sophisticated branching strategy my thinking has incorporated these  
requirements.

A little while back I spent some time on a proof of concept. I  
implemented a simple reporting architecture inside of functest.

http://functest.pythonesque.org/trac/wiki/FunctestReporting

The basic concept is that you can create and register a Report class,  
which is basically an interface. The class can implement one in a  
series of methods that the functest framework will call during the  
test run and a few at the very end after all the tests have finished.  
This is only run if the --report flag is sent to functest, so the  
class can be checked in to you test repository without being run every  
single time.

With this we could define a report class inside of the cosmo windmill  
and protocol tests in svn. The class could check the `svn status` and  
`svn diff` of your current repository and traverse the directory  
hierarchy to figure out what branch it's in and report this  
information along with the test results to a web service we host at  
OSAF.

This means all you would need to do is run `functest browser=firefox  
url=http://localhost:8080 --report` in the root windmill test  
directory and the results would be reported to our web interface. This  
gives each developer a nice way to let the rest of the group know how  
stable their current work is. It can also be used to record a  
certification that the tests pass against your branch before you check  
your changes in to the trunk.

I hope this gives everyone a quick update on some of the thinking and  
work I've been doing in windmill lately to makes all our lives easier.  
Comment and suggestions are greatly appreciated.

-Mikeal