[cosmo-dev] Windmill Woes

[Bobby Rullo]

After the update to the latest Windmill (3.x) with all the new  
dependency stuff, I find myself back at the point where Windmill is  
more or less unusable for me. I ran into a number of problems:

To create my test, I needed to know what state Windmill is in during  
the running of that test. So I was told that the best way to do this  
was to just run the test, which will run all its dependencies.

So I do load_test() in the shell and give it a shot. It fails because  
I'm on the wrong page  ('/chandler' instead of '/')

Then I go to the proper starting URL and try again. It loads (takes a  
long time) and I hit run. It's taking a while so I get up to go to the  
bathroom. When I get back, the it's done! Yay! Oh wait, it's all tore  
down. Gotta start again damn.

So I clear the IDE and hit load_test(). This time load_test() behaves  
differently - only the test itself is loaded, not the dependencies.  
Damn. So I quit windmill and do it again.

This time I make sure to watch very carefully so I can hit "Stop" and  
observe the state of the app after setup. Well, the test fails. Damn.  
At this point, I've given up. There's actually more iterations of  
these sorts of things, but I didn't log every single thing - it was  
frustrating enough as it is.

Luckily, though the usability problems right now are IMO major, the  
fixes are relatively simple:

	1) Documentation: If there was documentation in the __init__.py  
telling test writers what state the app will be in after setup() this  
whole thing would've been avoided.

	2) Don't be clever (or, do what I say not what you think I want):   
The load_test() method loads a test into the IDE with all the  
necessary setup and teardown actions. Well, that's how it behaves the  
FIRST time you call it. Subsequent invocations only load the test  
itself, forcing me to restart windmill. Grrrrrr. Windmill is trying to  
be clever here, it's assuming that because I loaded the dependencies  
once, they must've ran and I don't need them anymore. Except that it's  
not clever enough - the dependencies failed, so I need to try again.  
Also, what if i didn't want to load the dependencies at all? What if I  
just wanted to load the setup? Or the teardown? There should be  
explicit methods to do each of these things that work the same way  
every time.
	
	3) Actions take too long to load.

	4) When there are several suites loaded into the IDE, give me the  
option to run one at a time.

	5) Make windmill start at the correct url


As for the tests not running after I got past all the hurdles, I'm not  
sure what the problem is.

In any event, I just want to acknowledge that the Windmill team has  
been extremely responsive to the needs of the Cosmo team, and that any  
usability bugs introduced were only because of an attempt to make  
things better for us. So, thanks you guys!

bobby