[Windmill-dev] Python test suites

[Mikeal Rogers]

I think some history brings this in to context.

The JSON format was originally developed as an easy way to save and  
rerun tests. The format in which the IDE talks to the Service is  
JSONRPC, and the callbacks translate directly to this JSON format. At  
first we figured that we would eventually get rid of it once we had  
the XMLRPC interface working for client libraries, which is what the  
Python support is written in. But after some time we found that it had  
some nice qualities.

1. If you load a single json file in windmill directly you're  
eliminating one layer of abstraction (today, quite a few layers  
actually), which is very nice for debugging.
2. Having a "test markup" is nice for people who aren't programmers.  
We certainly prefer this JSON format over a comparable XML format to  
service this use case.
3. People who live and breath windmill think in terms of these  
callbacks, so it's fairly natural for people like Adam and Matthew to  
work in these JSON formats. In contrast the language libraries, Python  
and JavaScript currently, are geared towards people who are familiar  
with programming but don't live and breath windmill internals. They  
are intended to be nice use from a programming perspective removing  
the complexity of our Service <-> IDE callback system.

I made the decision early on that we are NOT going to invent a domain  
specific language or re-invent programming features for the JSON  
markup. As the requirements for framework and dependencies for tests  
became more complex we simply started parsing the JSON files and  
wrapping them around python objects dynamically, then leveraging the  
python for framework and dependency needs.

What we are seeing more and more based on feedback is that windmill  
test cases start out small and simple, and grow more and more complex.  
It's actually rare right now to see people who just automate a couple  
things, stick them in a directory, and run them every day without  
expanding or growing more complex dependency systems. That is why once  
you start growing past the simple use case of running a single JSON  
file you start getting introduced to using the Python framework. This  
doesn't mean your tests need to be in python. If you look at some of  
the setup/teardown in our cosmo test repository you'll see how a large  
string of JSON format can be transformed into a python calling using  
some windmill internals

http://svn.osafoundation.org/server/cosmo/trunk/cosmo/src/test/functional/test_windmill/test_new_user/__init__.py

Up until 0.3 there was a difference in how JSON and Python (or any  
XMLRPC based test) was loaded in to windmill. Today, there is no  
difference except in the case that you load a single JSON test file  
from the shell. When JSON tests are in a valid Python module they are  
parsed and objects are dynamically created that actually use the  
Python test authoring system over XMLRPC. In fact you can even turn on  
pdb debugging in functest and drop into the pdb debugger on a test  
failure from a JSON test.

So today I don't see much advantage or disadvantage to either format  
in the case that you aren't a windmill core developer writing windmill  
tests :)

There are huge differences between these authoring methods and the  
JavaScript library which i will discuss below.

Today there are a couple things you should also know how the windmill  
IDE handle's it's saves. The IDE sends the JSON callbacks for a suite  
through a system in the Service called "transforms" which implements  
ways of turning those callbacks in to any supported language  
(currently only Python). This doesn't preserve any of the original  
file, it's created each time a suite of callbacks is sent through the  
transforms.

> 1. How new you are to Windmill
> 2. What type of application you're wanting to test
> 3. How interested you are in learning Python
> 4. How technically savvy you are, or are willing to become

I don't think any of these matter. The question honestly is "do you  
know python?" and "would you prefer working with python test files  
rather than a markup?".

Before 0.3 I would say some of those points mattered, but today they  
really don't, and the more feedback we get and see the way people are  
using windmill I can say with confidence that it only comes down to  
ones comfortability level with python.

> People who are new to Windmill, or are testing simple Web sites,  
> might initially be happy with a simple, flat directory of JSON tests.
>
> As the complexity of their needs increases, and as they get more  
> comfortable with Windmill, they'll likely find a need for test  
> dependencies with can't be easily done with dumb data structures  
> like JSON. You really need stuff like conditional logic, which  
> requires an actual programming language -- hence the Python.
>
> I just committed a bunch of changes last night to bring the  
> JavaScript tests up to parity with the Python/JSON tests -- so  
> you'll be able to do similar stuff there, too.

Let's not complicate things too much here. Even after 0.4 (which will  
be a release focused on the JavaScript feature set) there are big  
drawbacks and advantages to using the JavaScript tests.

If you're running in continuous integration the JavaScript tests lose  
nearly all the feature set we have for remote debugging. That's just  
an unsolvable problem, and to be honest I think this drawback is fine  
when you look at where the JavaScript tests are very strong.

The JavaScript framework shines in that it has access to well,  
JavaScript, and the entirety of your webapp and every object  
available. We have a huge range of testability in the windmill API  
available from any language, but there is a point at which there isn't  
anything we can do. If something is so dynamic and unpredictable that  
you need a good amount of code to get each id, and it's not something  
that is reusable enough to be an extension. This sounds like some way  
off use case but it really isn't. Web apps are getting incredibly  
complex, and this case is coming up more and more often.

Currently, I think debugging JavaScript tests isn't as nice as JSON or  
Python tests, but I imagine this will improve with time and use.

BUT, even if you use the JavaScript framework, if you're working with  
a continuous integration environment I would recommend you only use  
the JavaScript framework for the cases that require it.

> I really think almost everyone writing tests will eventually need  
> the kind of functionality that only a real programming language  
> brings.
>
> However, having thought a lot about this over the last couple of  
> days, I'd like to add that I still think it's wrong-headed to insist  
> that a directory of JSON tests be a valid Python module -- given  
> what is surely a trivial amount of work to fall back to something  
> reasonable if it's a just a flat directory of JSON tests.
>
> As a new user, I'd be thinking the following:
>
> 1. If it's a folder full of plain JSON files, why is it a Python  
> module?
> 2. If this mysterious __init__.py can be empty, then it's obviously  
> *not really doing anything* -- why does it have to be there?

> I do not dispute that eventually most people will want the kind of  
> featurefulness that comes with the Python tests. But given how  
> little work it would be to support the simplest use-case, it seems  
> silly to force people over that conceptual hump before they are  
> ready to go.

Creating __init__.py is "really doing something", it's making it a  
valid python module.

Here is the thing, loading a single json test file bypasses the entire  
framework. It also doesn't have a "run" case, it actually loads  
differently in to windmill than JSON tests run in the case they are in  
a valid python module.

It is NOT a barrier to entry to create an empty file. Especially when  
this decreases the amount of ways tests are run in windmill. Towards  
the end of 0.2 we were getting serious tied down by all the ways tests  
could be run in windmill. It was confusing and hard to support when  
people had problems.

You also make this case like it's a common use case, and this is in  
contrast to the feedback we've been getting. The cases we seem to see  
the most begin with saving and running a single file and extend  
quickly to larger sets of tests with some kind of dependencies, so  
it's actually a good thing to get them comfortable with doing very  
tiny python things.

These imaginary users who have a handful of tests in a directory  
without dependencies and very little programming experience just  
haven't surfaced. If they do and they give us this feedback we can  
consider changing it.

> The argument is that it's easy to create that blank file -- the  
> point is that it's one more thing to trip over for a new user, and  
> if it really, truly is that easy, and not that big a deal, why can't  
> *we* work around its absence for the simplest use-case?

Because the trade off is supporting another way tests are loaded in to  
windmill, another command line option, another shell option.

Supporting this stuff adds complexity, it adds things for users to  
trip over because they will quickly need to overstep this case for a  
more complex one, which means they then need to use another option,  
creating new files, maybe even change directory names.

Instead of giving users gradual path to using the more complex  
windmill features you're creating a "drop off point" where everything  
changes. I don't think this is a healthy approach to learning the  
product and I know from experience that it's difficult to support. You  
have to figure out what point of adoption each user is and if they've  
crossed the "everything changes" bridge yet. More often than not users  
come around with issues during this transition, and trying to figure  
out what they forgot to change over IRC or email is incredibly hard.

> If the argument is that then people will want dependency  
> functionality, it's simple enough then to point them in the  
> direction of the Python tests for that.
>
> I don't think we should put idealogical purity over ease of use --  
> if there's not some really compelling reason to remove a barrier to  
> adoption, and the amount of work is trivial, then we ought to.

It's not ideological purity, it's actually what i believe is best for  
our users based on their feedback and experience. I don't think it's a  
barrier to adoption, I don't think anyone is every going to say "i was  
using windmill but once I had to create that empty __init__.py file I  
just gave up!".

And again, the initial work may be trivial, but the work to support it  
is actually a lot, and I don't think it's actually in the best  
interest of our users.

-Mikeal