[Windmill-dev] JS test suite or test setup/teardown
mikeal at osafoundation.org
Tue Dec 11 14:04:45 PST 2007
I'm going to explain the way the current Python/JSON tests work, and
Tests are laid out in directory hierarchies, each directory is a valid
python module. After the module's are parsed they look something like
If you run all of the tests they execute in this order;
If a setup fails it doesn't bother trying to run any tests that depend
on that setup.
If a test method, or another test namespace, exist within a namespace
then they depend on the parents setup/teardown.
If you want to run a single test, in this case test_allday(), it will
parse up the module hierachy and create a dependency chain so that any
setup/teardown that are required for that test to run will be run.
Setup steps are only ever run once if they pass. That way a person can
run there tests, notice a failure, make a change and run it again and
if windmill is still up it's assumed the browser is in the same state
and it won't need to redo the already passed setup/teardown. This
leads to better handling of state-fullness in the setup/teardown steps
in your tests -- meaning certain tests don't depend on any state from
previous tests, only setup/teardown, and anything setup is town down
properly. Having a "feature" that allows you to rerun setup steps
every time just leads to poor test robustness.
directories to modules, so I think it's reasonable to assume that
"collection" of tests and setup/teardown will be less than automatic
and require some definition, like mde outlined in his first example.
But, I don't see any reason you couldn't include the same kind of
feature we provide, for running a single test and calculating the
setup/teardown dependency chain.
We could rename the "jstests" windmill argument to be "jsdir" which
to the IDE. Another option "jsfilter" could be given to windmill,
this option would be sent to the IDE right after it sends the list of
test methods that match the filter string, look up the namespace
hierarchy to calculate their dependencies, and run them.
That's all I really have to say about setup/teardown and dependencies.
I do want to reiterate my strong opposition to an option that runs
setup always or once, setup should only be run once for the existing
session no matter what.
Another thing I need the js framework to do is hold of on executing
and running it's framework until the service tells it to. The
framework should be able to parse all the files in the directory, take
the filter option outlined above, and then wait for the service to
send one last callback to kick it off. The new callback that starts
the js framework will pass a UUID that needs to be sent back to the
service when the tests are finished.
This is the only way to integrate the js framework in to the rest of
windmill. It's crucial to have this work done if we ever want to run
the js tests alongside the rest of the Python/JSON tests. We can work
out the specifics after we're all agreed this needs to happen ASAP.
More information about the Windmill-dev