[Windmill-dev] Windmill JavaScript test improvements

[Matthew Eernisse]

Windmill doods,

Just a quick message to let y'all know where things are with the 
JavaScript test framework. I've been working on Cosmo tests this week, 
and in the process rounded off some of the sharp corners I ran into 
porting the new-user/login JSON tests. Here is a list of the improvements:

1. Error handling

You get more detailed errors in Output when something goes wrong. The 
framework will tell you (at least in Firefox) about syntax errors in 
your JS test files, and give you specifics about undefined/missing objects.

Caveats: This is still eval'd code, so you don't get line numbers for 
your errors. This is a problem with any system that uses eval'd code.

2. No more need to register your tests manually

This is a biggie. Now all you have to do is make sure all your test 
namespace objects and tests in your test files begin with "test_" (or 
'setup' or 'teardown,' see below), and the framework will find them.

It will run them in the order they are added to the tested app's 
window-scope. (This brings in the order-of-loading issue, which is dealt 
with below.)

This change is backward-compatible. If you include a register.js file in 
your test directory, the framework will work the way it always has, 
running the those tests (and only those tests) in the order you register 
them.

3. Setup/teardown

You can include setup or teardown for your test namespaces. These can be 
either a test function/array or a namespace object.

As you'd expect, setup will be run before any tests in the namespace, 
and teardown will be run afterward.

4. Order-of-loading, shared code, dependencies

There's now a require method you can call in your test files, like so:

windmill.jsTest.require('some_shared_code.js');

It will load and eval the code from that file before processing the code 
  in the test file.

This lets you share code properly -- you can make sure that objects 
you're trying to point to in your test code will already be loaded by 
the framework.

Caveats:

(a) The 'require' code doesn't try to do anything smart with circular 
dependencies, so it's possible to require some file that requires the 
file that originally required it. The framework won't infinite-loop, but 
it won't warn you that you may have missing-object problems.

(b) Paths for the 'require' are all currently relative to the top-level 
JS test directory you're running. Eventually this path should be 
relative to the file where the require call is made.

5. Detecting page location changes

The JS tests are loaded and run in the window of the tested app, so 
navigating to a new page wipes out all your test code. Now, if the 
tested app changes location, the test code will be reloaded and the 
tests can continue to run.

Caveats:

(a) Currently the location-change detection is a simple 
string-comparison, so even navigation within the current page triggers a 
reload of the page -- this can be annoying with anchor tags that use "#" 
as their HREF (for JavaScript onclicks).

(b) Continuing after a reload isn't possible in array-style tests, so 
don't make calls to navigate off the current page in an array-style test.

(c) If your tests include multi-page navigation, *don't* save state 
somewhere in the test code -- the test code will be reloaded, and your 
saved state will go bye-bye. Use the windmill.varRegistry Hash for that.

----

That's pretty much it. If you have any questions, please don't hesitate 
to ask.

Question: what is the policy for updating the wiki doc for all this? I'm 
sure we want to sync doc changes with actual releases, right?

I'll be talking with Adam this week about getting the reporting stuff up 
to snuff, and giving him some orientation to the internals of the JS 
test framework code.


Matthew