[Windmill-dev] Tests in native JavaScript

[Mikeal Rogers]

It's kind of hard to explain the current workflow for tests in  
windmill and how this is a departure and will have to be special  
cased, but I'm going to try because I don't think my replies below  
will make sense without it.

If you pull up the slides from OSCON.
http://windmill.osafoundation.org/trac/attachment/wiki/Miscellaneous/ 
OSCON2007-Windmill.pdf

Slide 6 shows an overview of how information comes in to the Service  
and to the IDE. The special case here is JSON tests that don't come  
in over XMLRPC, they are loaded in to the service directly.

Slide 8 shows the workflow between the IDE and the service over JSON- 
RPC.

To expand on this. Currently, a test comes in to the service, to gets  
some suite and uuid metadata attached to it and stuffed in a queue.  
When the IDE asks what to do next ( a call to next_action ), the  
service pops a test off the queue, send's it to the IDE, and stuffs  
it in another queue of objects yet to be resolved ( tests and actions  
that haven't finished running ). The IDE runs the callback it just  
got from the service and reports the results, the service pops the  
result off the yet-to-be-resolved queue, sticks some more metadata in  
to it, logs it, and stuffs it in the resolved queue.

If we tell the IDE to load a series of javascript functions and run  
them encapsulated in a "test suite" definition, and those tests call  
any function we need to log as pass/fail it breaks the workflow  
described above. So for javascript test and for JSUnit we'll need to  
special case workflow for that framework. In other words we'll have  
to write in a way for the IDE run tests it didn't get from the  
service and report them to the service, and the service will have to  
provide a way to report tests that it didn't process.


> I was thinking we could use the existing JSON syntax pretty much as- 
> is, since it's already valid JS, and the controller already knows  
> how to consume it.
>

That isn't entirely correct. Since you're calling controller actions  
outside of the regular windmill test framework ( not loading them  
through the service ) windmill doesn't know how to consume those calls.


> No need to invent yet another DSL for manipulating the UI. So,  
> something like:
>
> var arr = [
> {"method": "type", "params": {"id" : "goToDateInput", "text":  
> "05/08/2002"}},
> {"method": "click", "params": {"jsid" : "{$miniCalGotoButton}"}},
> {"method": "wait", "params": {"milliseconds" : 5000}}
> ];
>
> jum.controller.doSomeUiStuff(arr);
>
>

Windmill needs to handle these calls differently than it does the  
current JSON tests because they aren't getting loaded through the  
service and will have to report their results slightly differently.

This API could work, but it's not as nice to use as the one I  
described earlier.

When we developed the python test library we went with an API where  
you call the controller functions like regular python functions and  
debug them like regular python calls. I imagine when other people  
write XMLRPC based testing libraries they will do the same. I don't  
see why the jum.controller API couldn't do the same.

The JSON format is a hack, it's basically just a JSON string that  
translates in to a callback the service hands off to the IDE before  
it's loaded up with some more metadata from the service processing  
(uuid, suite_name, etc). I can't imagine why any other windmill test  
authoring library will write tests like RPC objects.



>> The largest limitation of all this is that you can't stager a  
>> regular windmill test and a JSUnit test one after another. They  
>> follow completely different workflows and would be loaded in to  
>> the service very differently.
>>
>
> To me the JSUnit thing is actually kind of anciallary. To me the  
> most important thing is just having a sane, structured way to write  
> Windmill tests in JavaScript.
>
> I can see an argument for wanting to go API-compatible (or mostly  
> so) with JSUnit for the native JS tests -- both because people are  
> familiar with it, and because it opens the door to the idea of  
> sharing some tests between Windmill and JSUnit.
>
> On the other hand, if it's going to sidetrack seriously the process  
> of getting native JS tests working in Windmill, maybe we ought to  
> avoid opening the JSUnit can of worms.
>
> If doing something like writing plugins is the most expedient way  
> to get it started, then that's cool -- but my suspicion is that  
> it's not a good long-term solution, and I'd like to avoid rework  
> later.
>
> Thoughts?
>

Knowing the scope of the tests you want to write, i would advise  
against using the extensions framework. That's just not what it's  
made for and debugging and maintaining them will quickly become a  
problem.


> On the other hand, if it's going to sidetrack seriously the process  
> of getting native JS tests working in Windmill, maybe we ought to  
> avoid opening the JSUnit can of worms.
>

It's funny you put the work in this order, because in my mind JSUnit  
is something we have to do anyway. Right now there isn't any way I  
know of to get JSUnit tests running inside continuous integration so  
having windmill fill this void seems like a priority to me. This is  
something we need even inside of OSAF in the short term.

Bundling the problem of writing JS tests in with the JSUnit stuff  
makes the most sense to me. First off, javascript doesn't have a  
variety of competing test frameworks like most languages do, just a  
bunch of different xUnit implementations, so we aren't stepping on  
anyone's toes saying javascript tests need to be in JSUnit. Second,  
javascript is the only language in wide use that can't use the XMLRPC  
interface in the service, because it has to run in the browser, so  
any way we slice the javascript test authoring library it's going to  
have to be special cased in windmill.

Say we didn't bundle the javascript test writing work with the JSUnit  
work. We would then need to write up an entirely new testing  
framework in javascript for it and we would need to duplicate the  
effort of creating a reporting mechanism in windmill for tests that  
don't come through the service.

-Mikeal