[Windmill-dev] js framework "exit on done" work

[Mikeal Rogers]

Well I'm glad to hear that we have consensus on getting this feature  
in to the jstests.

In terms of getting a similar feature for the service to tell the IDE  
when a suite is completed, I don't think it's actually possible.

To backtrack a little, the reason we need the jsTests to tell the  
service when they are done is because in some cases the IDE is  
launched and the jsTests run headless (nobody is watching, you still  
need a GUI and a browser obviously) and when they are completed the  
Service needs to kill the IDE and the browser completely, or move on  
to the next set of tests. In this case the IDE is a fixture created  
by the Service. The reverse case never happens, the Service is never  
launched by the IDE and is later destroyed by it.

Although I can see the utility in the IDE knowing when a suite is  
done, the Service rarely knows that state. The only time it does is  
when test objects are loaded in to the Service for browser debugging  
of the tests. It would have to fall on the XMLRPC based test  
framework to notify the service of this state change if we were to do  
it.

It's very difficult for code based test frameworks to know when  
"suites" end (suites are implemented as individual test methods in  
most cases) e. I think it's a fairly large burden on new library  
developers to have to implement this.

The only time this is easy is when you have markup based tests that  
are parsed, if you're executing code line by line it's a pain to pick  
up when you move from one set to the next and send yet another method  
to the service.

I do think that state changes should be obvious in the IDE, but the  
IDE already knows about as much as the Service does about this state.  
The IDE knows when the next suite starts, it knows when the service  
is out of test objects, that's about all the Service knows.

-Mikeal

On Oct 22, 2007, at 6:46 PM, Matthew Eernisse wrote:

> +1
>
> I actually remember asking about this before -- it's a pretty  
> standard thing to want to know when an entire suite has finished.  
> Of course that would require that we have some kind of a larger  
> concept of a 'suite.'
>
> But it would be nice to be able to call a big blob of tests, and  
> then get some kind of summary when the entire thing is done.
>
>
> Matthew
>
> Adam Christian wrote:
>> This sounds fine to me, I was also thinking that this might not be  
>> such a bad idea for the non js tests.. that way the IDE can  
>> actually display some output like.. "Suite finished running" etc  
>> so people don't have to guess on the state.. ideas?
>> Adam
>> On Oct 22, 2007, at 2:50 PM, Mikeal Rogers wrote:
>>> Trying to get the js based tests hooked up I just noticed some  
>>> problems.
>>>
>>> 1) The Service has no idea when the jstests are finished. I can  
>>> guess by waiting for the IDE to call next_action() but this is  
>>> problematic.
>>> 2) If we just add a simple "I'm done" method to the Service it  
>>> then has no way to differentiate between multiple js test runs.
>>>
>>> What I propose is to make some additions to both interfaces.
>>>
>>> Currently commands.jsTests only takes one argument, "tests" which  
>>> is a list of the js files it's recursively parsed from the given  
>>> directory. I'd like it to take another argument, "uuid" which  
>>> will be a standard UUID for the test run.
>>>
>>> At the end of that test run, the js framework should call a new  
>>> method in the Service called "js_test_run_final" with params  
>>> "uuid" and "results" (a serialized list of the test results -- i  
>>> believe you're already keeping this somewhere).
>>>
>>> This way the Service can make multiple test runs and people can  
>>> query the server to see the status of any test run. This work can  
>>> also be leveraged once we go multi-client with minimal changes.
>>>
>>> The Service will be implementing additional methods so that  
>>> remote xml/json-rpc clients can start js test runs and get the  
>>> results from the Service.
>>>
>>> If anyone else would like to add more methods or options to the  
>>> js test workflow now would be a good time. Is there anything else  
>>> going on over in JavaScript land that the Service might was to be  
>>> notified of?
>>>
>>> -Mikeal
>>> _______________________________________________
>>> Windmill-dev mailing list
>>> Windmill-dev at osafoundation.org
>>> http://lists.osafoundation.org/mailman/listinfo/windmill-dev
>> _______________________________________________
>> Windmill-dev mailing list
>> Windmill-dev at osafoundation.org
>> http://lists.osafoundation.org/mailman/listinfo/windmill-dev
>
> _______________________________________________
> Windmill-dev mailing list
> Windmill-dev at osafoundation.org
> http://lists.osafoundation.org/mailman/listinfo/windmill-dev