[Windmill-dev] Python test suites

[Matthew Eernisse]

Robert, et al,

A very good question. Comments below ...

Robert Leftwich wrote:
> Actually I'd be interested to hear what people think about which is the
> best approach to authoring tests, python or json?
> 
> What are the pros and cons of each? Is there a scenario where one is
> clearly better than the other?

I think it will depend on:

1. How new you are to Windmill
2. What type of application you're wanting to test
3. How interested you are in learning Python
4. How technically savvy you are, or are willing to become

People who are new to Windmill, or are testing simple Web sites, might 
initially be happy with a simple, flat directory of JSON tests.

As the complexity of their needs increases, and as they get more 
comfortable with Windmill, they'll likely find a need for test 
dependencies with can't be easily done with dumb data structures like 
JSON. You really need stuff like conditional logic, which requires an 
actual programming language -- hence the Python.

I just committed a bunch of changes last night to bring the JavaScript 
tests up to parity with the Python/JSON tests -- so you'll be able to do 
similar stuff there, too.

I really think almost everyone writing tests will eventually need the 
kind of functionality that only a real programming language brings.

However, having thought a lot about this over the last couple of days, 
I'd like to add that I still think it's wrong-headed to insist that a 
directory of JSON tests be a valid Python module -- given what is surely 
a trivial amount of work to fall back to something reasonable if it's a 
just a flat directory of JSON tests.

As a new user, I'd be thinking the following:

1. If it's a folder full of plain JSON files, why is it a Python module?
2. If this mysterious __init__.py can be empty, then it's obviously *not 
really doing anything* -- why does it have to be there?

I do not dispute that eventually most people will want the kind of 
featurefulness that comes with the Python tests. But given how little 
work it would be to support the simplest use-case, it seems silly to 
force people over that conceptual hump before they are ready to go.

The argument is that it's easy to create that blank file -- the point is 
that it's one more thing to trip over for a new user, and if it really, 
truly is that easy, and not that big a deal, why can't *we* work around 
its absence for the simplest use-case?

If the argument is that then people will want dependency functionality, 
it's simple enough then to point them in the direction of the Python 
tests for that.

I don't think we should put idealogical purity over ease of use -- if 
there's not some really compelling reason to remove a barrier to 
adoption, and the amount of work is trivial, then we ought to.


Matthew