[cosmo-dev] Windmill Woes

[Mikeal Rogers]

I wanna say something before I respond.

You aren't writing simple tests, you're writing very advanced tests,  
in fact you're writing some of the most advanced tests that can be  
written with windmill, tests that can't even be written in Selenium  
and are the main reason we wrote windmill.

You're using the most advanced features of windmill but you haven't  
gone through the normal user workflow of writing simple things first,  
and through that process you usually get a feel for how the product  
works and how to get it to do what you want without a lot of headace.  
It's an organic process that you've skipped because you're jumping in  
to writing difficult tests in a fairly large and developed test  
repository.

I understand why you're frustrated but we actually have documented  
almost all of this stuff and have spent a good amount of time  
incorporating usability and user feedback in to windmill, and nearly  
all of your complaints are about how things are work now based on that  
feedback which you just don't understand yet.


> After the update to the latest Windmill (3.x) with all the new  
> dependency stuff, I find myself back at the point where Windmill is  
> more or less unusable for me. I ran into a number of problems:
>
> To create my test, I needed to know what state Windmill is in during  
> the running of that test. So I was told that the best way to do this  
> was to just run the test, which will run all its dependencies.

This actually isn't any different than before. Sure, you could write a  
test without knowing the state before hand but to actually get it to  
run you had to understand how all the test ordering stuff worked, and  
to debug a test you either had to setup the state by hand or run the  
entire suite. So if you consider the headaces of the entire process  
it's actually much better now.

>
>
> So I do load_test() in the shell and give it a shot. It fails  
> because I'm on the wrong page  ('/chandler' instead of '/')

We have the cosmo tests setup so that testing against hub and lab  
builds automatically get the right url, other testing of cosmo is  
assumed that cosmo has the default configuration, which is at / 
chandler. If this is incorrect than we can change it.

>
>
> Then I go to the proper starting URL and try again. It loads (takes  
> a long time) and I hit run. It's taking a while so I get up to go to  
> the bathroom. When I get back, the it's done! Yay! Oh wait, it's all  
> tore down. Gotta start again damn.
>

Why didn't you just remove the teardown suites from the bottom of the  
list in the IDE before you ran the tests?

> So I clear the IDE and hit load_test(). This time load_test()  
> behaves differently - only the test itself is loaded, not the  
> dependencies. Damn. So I quit windmill and do it again.

If windmill is up and it spends time setting up the state of the  
tests, it assumes that state is still there until it is stopped. This  
whole dependency chain is there for the 90% case, which is running  
tests and debugging failures, which we do far more than we write new  
tests. And this is even optimized for the test creation case which I  
cover a bit more below.

>
>
> This time I make sure to watch very carefully so I can hit "Stop"  
> and observe the state of the app after setup. Well, the test fails.  
> Damn. At this point, I've given up. There's actually more iterations  
> of these sorts of things, but I didn't log every single thing - it  
> was frustrating enough as it is.

Your test failed, or the setup failed?

>
>
> Luckily, though the usability problems right now are IMO major, the  
> fixes are relatively simple:
>
> 	1) Documentation: If there was documentation in the __init__.py  
> telling test writers what state the app will be in after setup()  
> this whole thing would've been avoided.

The names of the modules aren't obvious enough?

test_create_user/test_pim/test_canvas

Creates new user and logs in, gets windmill to the pim page, gets  
windmill to the cal canvas. That's pretty much all it does, as soon as  
you understand that it's really obvious where your test should go.

>
>
> 	2) Don't be clever (or, do what I say not what you think I want):   
> The load_test() method loads a test into the IDE with all the  
> necessary setup and teardown actions. Well, that's how it behaves  
> the FIRST time you call it. Subsequent invocations only load the  
> test itself, forcing me to restart windmill. Grrrrrr. Windmill is  
> trying to be clever here, it's assuming that because I loaded the  
> dependencies once, they must've ran and I don't need them anymore.  
> Except that it's not clever enough - the dependencies failed, so I  
> need to try again. Also, what if i didn't want to load the  
> dependencies at all? What if I just wanted to load the setup? Or the  
> teardown? There should be explicit methods to do each of these  
> things that work the same way every time.

No, you're wrong.

The normal workflow when you're running and debugging a test is to run  
it until it fails, edit it, then rerun that test. The browser should  
be in the same state it was in before windmill stopped.

If we rerun the setup it would immediately fail, because teardown  
hasn't been called. This also means that every time i make a change to  
a test and want to rerun it I have to wait through the entire setup,  
even tho I'm in the proper state for the test to run.

>
> 	
> 	3) Actions take too long to load.

We had a decision to make here, have a constant workflow between the  
Service and IDE and deal with the overhead of that communication, or  
we make the service layer transparent and queue everything into the  
IDE. The former is how Selenium handles it, and we know that doesn't  
work well for us, so we picked the first.

We're looking at improving the performance of this specific case, but  
there won't be a huge performance increase either way because the  
python side still loads tests one by one.

>
>
> 	4) When there are several suites loaded into the IDE, give me the  
> option to run one at a time.v

This is a good idea. Currently we have the option to start running the  
tests at any given point, but it would nice if the top of the suite  
had a "play" button that just ran that suite and then ended.

>
>
> 	5) Make windmill start at the correct url

umn.... it starts at the url you give it. Anything else is specific to  
the tests, which i talked about earlier.

>
>
>
> As for the tests not running after I got past all the hurdles, I'm  
> not sure what the problem is.
>
> In any event, I just want to acknowledge that the Windmill team has  
> been extremely responsive to the needs of the Cosmo team, and that  
> any usability bugs introduced were only because of an attempt to  
> make things better for us. So, thanks you guys!
>
> bobby
> _______________________________________________
> cosmo-dev mailing list
> cosmo-dev at lists.osafoundation.org
> http://lists.osafoundation.org/mailman/listinfo/cosmo-dev