[Windmill-dev] New test collection and dependency strategy

[Mikeal Rogers]

Those of you who watch the trunk commits have probably noticed the new  
collector and dependency code I've been committing.

The largest change is to the way we collect and run json tests. I  
wrote a lot of code in functest to allow different consumers to define  
their own additions to the test collection mechanisms in functest.  
Using that framework I wrote a windmill collector for .json test files.

This means that you won't need to define a test_conf.py with a static  
list of test ordering (which is horribly unscalable as we've realized  
on cosmo). Instead you make sure that all your directories being with  
"test_" and are valid python modules ( they must include a __init__.py  
file). This may seem like more work than previously, but this also  
means that you can leverage functests setup/teardown dependency system  
which uses module heirarchies of setup/teardown for dependency  
calculation.

I've started working on a porting the cosmo tests to this new framework

http://svn.osafoundation.org/server/cosmo/trunk/cosmo/src/test/functional/test_windmill/

It's pretty ugly right now since I have copies of all the old tests in  
there, but you can start to get an idea of how things work.

At this point I'm ready to integrate the new functest framework run  
points in to windmill. This means, among other things, changing the  
meaning of certain windmill commands. The old run_json and load_json  
commands will mean something a bit different and link to the new way  
of running tests in the framework. It also means that keyword  
arguments given to the windmill binary will need to be added to  
functest.registry to avoid incompatibilities between test repositories  
written for windmill/functest.

Another big change is that json tests run like any other test library,  
they block and wait for the result of each action unless they are  
passed to the service via add_command/add_test. With functest, if you  
pass browser_debugging=True to the command line json/python tests will  
be loaded rather than block and wait for a return. The new run_ and  
load_ commands in windmill will work this way as well.

-Mikeal