mikeal at osafoundation.org
Sat Aug 25 10:34:48 PDT 2007
I'm definitely not in favor of adding another layer of abstraction in
the current controller API to execute what essentially amounts to
another framework. So something like this is not going to happen;
I understand what you're saying about wanting to write tests in
separate problems because JSUnit test cases as they stand now don't
need to load a live application in the browser.
I think we can add JSUnit support without touching our fairly clean
controller API. In fact, instead of having a declarative model where
you define every test you could just point the windmill service at
the directory your JSUnit tests are in and it could parse it for each
test file, send it to the IDE and the IDE could execute them and
report results without adding any new special case in the controller
API. This would still get all the features of the services logging
and continuous integration features without adding a single function
to the controller API.
Usually the case for writing a test in any language is done via
XMLRPC, but the case you are talking about is different. In your case
you want to write a test case that can't actually be facilitated by
application you're testing loaded in the browser. Well we do have the
extensions framework already for this, you can write a new controller
method, have it take any amount of arguments you want, it can use the
same locator system the current controllers use, and you can already
envoke those methods from Python/JSON/whatever.
The only issue with using the plugin system for that is if the amount
of tests you're writing this way gets too large it would be very
difficult to maintain.
I have another option. When i first spoke to bobby about JSUnit
integration he was adamant that we not diverge from "standard JSUnit"
but I'm still not convinced that there isn't something we can do here
to make writing unittests easier and bluring the line slightly
between unittests and functional tests -- and providing a clean way
for you to do what you're talking about -- while still ensuring that
regular old unittests can run in the windmill unittest framework
without any modifications to the tests.
Since the JSUnit tests can have all the js code in the IDE we won't
have anything loaded in the browser when we run the tests. We could
provide an extension to unittest object that we will need to
implement for the unittest code ( in the current cosmo JSUnit tests
this object is 'jum' ) that is a wrapper object for the controller
jum.assertFalse(delta.getProperty("body") == undefined);
There are some issues with implementing this.
For one, the IDE never runs a test without getting it from the
service. The service never resolves a test it hasn't received,
processed, and handed to the IDE. In fact implementing the whole
JSUnit framework has this problem. What we would consider a "suite"
would be an individual JSUnit test method, and the individual assert
and controller calls to the jum object would be "tests", so solving
this problem by making the IDE describe and pull out each test would
slow down the JSUnit tests by about 10x what they currently run.
The service <-> IDE design may seem crazy at first but once you look
at how tests come in over XMLRPC and then get resolved over JSONRPC
and are constantly referenced by UUID it starts to make a lot more
sense. In fact we catch a lot of bugs this way.
One idea i had was to create a new service method for resolving
unittest's, unittest_report(), that would basically call all of the
tail end report code we currently have without running through all
the test resolution code we in the service.
The largest limitation of all this is that you can't stager a regular
windmill test and a JSUnit test one after another. They follow
completely different workflows and would be loaded in to the service
On Aug 24, 2007, at 4:08 PM, Matthew Eernisse wrote:
> I sat down to write some Windmill tests for recurring items in
> Cosmo, and realized that the types of asserts available with the
> current JSON/Python tests simply weren't sufficient for the
> specific stuff I wanted to test for. I immediately started wanting
> I wasn't thinking of this in the conext of JSUnit integration --
> but since there's already an established convention with JSUnit, it
> kind of makes sense to me that any pure-JS test feature we add
> would be compatible with JSUnit tests.
> That way people (including us) could reuse their existing tests.
> And instead of running these tests against contrived test fixtures
> set up with JS in the test environment, they can run these tests
> against the data generated by actually maniuplating the UI elements
> the way a user would. The tests would also run in the environment
> our UI code runs in -- our various supported browsers.
> The JS code could be easily pulled down by XHR and eval'd to get it
> into the testing environment. So, given that what we have now are
> JSON and Python tests, that leaves the question of how to invoke
> the pure-JS tests in Windmill, and how to report results.
> I'm a Web programmer, so the natural comparison that comes to my
> mind is HTML markup and programming code in a Web app. Sometimes
> it's more appropriate to embed small scraps of code in your markup,
> a la PHP/ERuby/PSP/JSP. Sometimes it's more appropriate to have
> small bits of markup in a bunch of code (mod_python Publisher, Java
> Ideally, I think we'd like to give people a similar choice with the
> JS tests in Windmill:
> 1. Pre-load all the JS tests, and have some syntax to invoke
> specific tests from inside JSON/Python tests.
> 2. Run JS tests, and have some syntax we can use inside them to
> manipulate the UI the way the current JSON/Python tests do. (Maybe
> some kind of adapter and JS objects in the code that look just like
> the currently defined JSON tests?)
> Windmill-dev mailing list
> Windmill-dev at osafoundation.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Windmill-dev