[Windmill-dev] Tests in native JavaScript

[Mikeal Rogers]

I'm definitely not in favor of adding another layer of abstraction in  
the current controller API to execute what essentially amounts to  
another framework. So something like this is not going to happen;

controller.load_js_module(url)
controller.js_test(name=my_javascript_function, params={})

I understand what you're saying about wanting to write tests in  
javascript and the case for JSUnit but I do think they are two  
separate problems because JSUnit test cases  as they stand now don't  
need to load a live application in the browser.

I think we can add JSUnit support without touching our fairly clean  
controller API. In fact, instead of having a declarative model where  
you define every test you could just point the windmill service at  
the directory your JSUnit tests are in and it could parse it for each  
test file, send it to the IDE and the IDE could execute them and  
report results without adding any new special case in the controller  
API. This would still get all the features of the services logging  
and continuous integration features without adding a single function  
to the controller API.

On the topic of using javascript for writing other windmill tests.  
Usually the case for writing a test in any language is done via  
XMLRPC, but the case you are talking about is different. In your case  
you want to write a test case that can't actually be facilitated by  
the current controller API and needs to be in javascript with the web  
application you're testing loaded in the browser. Well we do have the  
extensions framework already for this, you can write a new controller  
method, have it take any amount of arguments you want, it can use the  
same locator system the current controllers use, and you can already  
envoke those methods from Python/JSON/whatever.

The only issue with using the plugin system for that is if the amount  
of tests you're writing this way gets too large it would be very  
difficult to maintain.

I have another option. When i first spoke to bobby about JSUnit  
integration he was adamant that we not diverge from "standard JSUnit"  
but I'm still not convinced that there isn't something we can do here  
to make writing unittests easier and bluring the line slightly  
between unittests and functional tests -- and providing a clean way  
for you to do what you're talking about -- while still ensuring that  
regular old unittests can run in the windmill unittest framework  
without any modifications to the tests.

Since the JSUnit tests can have all the js code in the IDE we won't  
have anything loaded in the browser when we run the tests. We could  
provide an extension to unittest object that we will need to  
implement for the unittest code ( in the current cosmo JSUnit tests  
this object is 'jum' ) that is a wrapper object for the controller  
API. So;

jum.assertEquals("new", delta.getProperty("displayName"));
jum.assertFalse(delta.getProperty("body") == undefined);
jum.controller.open(url)
jum.controller.click(id='blah')

There are some issues with implementing this.
For one, the IDE never runs a test without getting it from the  
service. The service never resolves a test it hasn't received,   
processed,  and handed to the IDE. In fact implementing the whole  
JSUnit framework has this problem. What we would consider a "suite"  
would be an individual JSUnit test method, and the individual assert  
and controller calls to the jum object would be "tests", so solving  
this problem by making the IDE describe and pull out each test would  
slow down the JSUnit tests by about 10x what they currently run.

The service <-> IDE design may seem crazy at first but once you look  
at how tests come in over XMLRPC and then get resolved over JSONRPC  
and are constantly referenced by UUID it starts to make a lot more  
sense. In fact we catch a lot of bugs this way.

One idea i had was to create a new service method for resolving  
unittest's, unittest_report(), that would basically call all of the  
tail end report code we currently have without running through all  
the test resolution code we in the service.

The largest limitation of all this is that you can't stager a regular  
windmill test and a JSUnit test one after another. They follow  
completely different workflows and would be loaded in to the service  
very differently.

-Mikeal

On Aug 24, 2007, at 4:08 PM, Matthew Eernisse wrote:

> I sat down to write some Windmill tests for recurring items in  
> Cosmo, and realized that the types of asserts available with the  
> current JSON/Python tests simply weren't sufficient for the  
> specific stuff I wanted to test for. I immediately started wanting  
> the ability to write tests directly in JavaScript.
>
> I wasn't thinking of this in the conext of JSUnit integration --  
> but since there's already an established convention with JSUnit, it  
> kind of makes sense to me that any pure-JS test feature we add  
> would be compatible with JSUnit tests.
>
> That way people (including us) could reuse their existing tests.  
> And instead of running these tests against contrived test fixtures  
> set up with JS in the test environment, they can run these tests  
> against the data generated by actually maniuplating the UI elements  
> the way a user would. The tests would also run in the environment  
> our UI code runs in -- our various supported browsers.
>
> The JS code could be easily pulled down by XHR and eval'd to get it  
> into the testing environment. So, given that what we have now are  
> JSON and Python tests, that leaves the question of how to invoke  
> the pure-JS tests in Windmill, and how to report results.
>
> I'm a Web programmer, so the natural comparison that comes to my  
> mind is HTML markup and programming code in a Web app. Sometimes  
> it's more appropriate to embed small scraps of code in your markup,  
> a la PHP/ERuby/PSP/JSP. Sometimes it's more appropriate to have  
> small bits of markup in a bunch of code (mod_python Publisher, Java  
> Servlets).
>
> Ideally, I think we'd like to give people a similar choice with the  
> JS tests in Windmill:
>
> 1. Pre-load all the JS tests, and have some syntax to invoke  
> specific tests from inside JSON/Python tests.
>
> 2. Run JS tests, and have some syntax we can use inside them to  
> manipulate the UI the way the current JSON/Python tests do. (Maybe  
> some kind of adapter and JS objects in the code that look just like  
> the currently defined JSON tests?)
>
> Thoughts?
>
>
> Matthew
>
>
> _______________________________________________
> Windmill-dev mailing list
> Windmill-dev at osafoundation.org
> http://lists.osafoundation.org/mailman/listinfo/windmill-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osafoundation.org/pipermail/windmill-dev/attachments/20070825/a16dd7b9/attachment.html