[Windmill-dev] JS test suite or test setup/teardown

[Mikeal Rogers]

I'm going to explain the way the current Python/JSON tests work, and  
then lay down what I think is a comparable javascript solution.

Tests are laid out in directory hierarchies, each directory is a valid  
python module. After the module's are parsed they look something like  
this;

tests
tests.setup_module()
tests.teardown_module()
tests.test_something()
tests.test_pim
tests.test_pim.setup_module()
tests.test_pim.test_dashboard
tests.test_pim.test_dashboard.test_allday()

If you run all of the tests they execute in this order;

tests.setup_module()
tests.test_something()
tests.test_pim.setup_module()
tests.test_pim.test_dashboard.test_allday()
tests.teardown_module()

If a setup fails it doesn't bother trying to run any tests that depend  
on that setup.

If a test method, or another test namespace, exist within a namespace  
then they depend on the parents setup/teardown.

If you want to run a single test, in this case test_allday(), it will  
parse up the module hierachy and create a dependency chain so that any  
setup/teardown that are required for that test to run will be run.

tests.setup_module()
tests.test_pim.setup_module()
tests.test_pim.test_dashboard.test_allday()
tests.teardown_module()

Setup steps are only ever run once if they pass. That way a person can  
run there tests, notice a failure, make a change and run it again and  
if windmill is still up it's assumed the browser is in the same state  
and it won't need to redo the already passed setup/teardown. This  
leads to better handling of state-fullness in the setup/teardown steps  
in your tests -- meaning certain tests don't depend on any state from  
previous tests, only setup/teardown, and anything setup is town down  
properly. Having a "feature" that allows you to rerun setup steps  
every time just leads to poor test robustness.

Javascript namespaces can easily be used the same way we use python  
modules. Obviously javascript doesn't have a one-to-one mapping of  
directories to modules, so I think it's reasonable to assume that  
"collection" of tests and setup/teardown will be less than automatic  
and require some definition, like mde outlined in his first example.

But, I don't see any reason you couldn't include the same kind of  
feature we provide, for running a single test and calculating the  
setup/teardown dependency chain.

We could rename the "jstests" windmill argument to be "jsdir" which  
mounts the root test directory and sends all the javascript file urls  
to the IDE.  Another option "jsfilter" could be given to windmill,  
this option would be sent to the IDE right after it sends the list of  
javascript urls to be parsed. The js framework could then look for  
test methods that match the filter string, look up the namespace  
hierarchy to calculate their dependencies, and run them.

That's all I really have to say about setup/teardown and dependencies.  
I do want to reiterate my strong opposition to an option that runs  
setup always or once, setup should only be run once for the existing  
session no matter what.

Another thing I need the js framework to do is hold of on executing  
and running it's framework until the service tells it to. The  
framework should be able to parse all the files in the directory, take  
the filter option outlined above, and then wait for the service to  
send one last callback to kick it off. The new callback that starts  
the js framework will pass a UUID that needs to be sent back to the  
service when the tests are finished.

This is the only way to integrate the js framework in to the rest of  
windmill. It's crucial to have this work done if we ever want to run  
the js tests alongside the rest of the Python/JSON tests. We can work  
out the specifics after we're all agreed this needs to happen ASAP.

-Mikeal