[Windmill-dev] Python test suites

[Adam Christian]

An example of this user feedback from Jacob who is a relatively  
advanced user; "Why can't I just run a directory of JSON files without  
the python stuff? What if I don't want setup/teardown and I have a  
bunch of idiots working on this for me who simply want to stick JSON  
files in a directory..."

If we want to suck in a chunk of the Selenium "I don't wanna know  
anything except record, save, playback" the very simplest use case is  
important. You may have noticed that we haven't had one user in the  
channel or on the list who isn't relatively technical..  I think this  
is because Windmill still is very technical to the "I sit and do  
manual testing" user.

To appeal to this audience we need:
	- Built binaries on each platform and a working WX UI (We are very  
close on the WX and I have been using it regularly on Leopard)
	- The ability to open any directory of JSON files on the file system  
from the WX menu run / load test option and happily see them play back  
(even unordered would probably work)

It is feasible to add some logic that on failure of importing any  
python modules, goes ahead and sucks in all the .JSON files and runs  
them.

If we really want to grow an impressive user base, these are things  
that _NEED_ to be done whether it's clean and nice or not. Sometimes  
it is easy into the "developer/tech savvy box" and as I really like  
all the users in that camp because they can contribute, thats not how  
selenium has become the giant it has.

Adam

On Dec 18, 2007, at 4:05 PM, Matthew Eernisse wrote:

> Mikeal,
>
> Thanks for the response. I definitely appreciate your point of view.  
> A few counterpoints/comments below:
>
> Mikeal Rogers wrote:
>> I think some history brings this in to context.
>
> The prospective users that I'm talking about won't have any interest  
> in Windmill history. All they know or care about is what they see  
> when they come to the site.
>
>> If you look at some of the setup/teardown in our cosmo test  
>> repository you'll see how a large string of JSON format can be  
>> transformed into a python calling using some windmill internals
>
> I think there will be a non-trivial number of prospective users who  
> also have no interest at all in the Windmill internals. :)
>
>> Creating __init__.py is "really doing something", it's making it a  
>> valid python module.
>
> From the point of view of a non-Python-programmer, that is a logical  
> equivalent to nothing. :)
>
>> It is NOT a barrier to entry to create an empty file. Especially  
>> when this decreases the amount of ways tests are run in windmill.
>
> Any extra step is a barrier.
>
> The smaller API footprint is good, but I don't think that falling  
> back to the simplest use case implies a change to the API. You could  
> easily make the single method call handle both cases.
>
> The extra work would be handling both cases on the back end,  
> documenting it, and keeping it current. I'm not discounting the  
> extra work.
>
>> These imaginary users who have a handful of tests in a directory  
>> without dependencies and very little programming experience just  
>> haven't surfaced. If they do and they give us this feedback we can  
>> consider changing it.
>
> I'm glad you're willing to consider this if we get feedback that it  
> would be good.
>
> Note that we will receive exactly zero feedback from people who come  
> to the site and then just walk away because the perception is that  
> they have to use Python -- which is what "making your test  
> directories a valid Python module" implies.
>
>> Because the trade off is supporting another way tests are loaded in  
>> to windmill, another command line option, another shell option.
>
> See above. Why can't the same method call support both?
>
>> Supporting this stuff adds complexity, it adds things for users to  
>> trip over because they will quickly need to overstep this case for  
>> a more complex one, which means they then need to use another  
>> option, creating new files, maybe even change directory names.
>
> Agreed. In this case, I'd vote that making it trivial for people to  
> drop some JSON files into a directory and run their tests with zero  
> effort is more of a priority.
>
> I wouldn't mind a little more complexity down the road if the  
> initial bump is zero. By then, they will already have a bunch of  
> tests written, and will like what they see in Windmill. They will  
> also have a visceral understanding of *why* they need this increased  
> functionality.
>
>> It's not ideological purity, it's actually what i believe is best  
>> for our users based on their feedback and experience. I don't think  
>> it's a barrier to adoption, I don't think anyone is every going to  
>> say "i was using windmill but once I had to create that empty  
>> __init__.py file I just gave up!".
>
> See above. I'm not talking about existing users. I'm talking about  
> prospective users.
>
> I have a strong feeling there will be people who see their tests  
> have to include .py files, and will say, "Ah, I don't want to learn  
> Python, I already use X" and will walk away without giving us any of  
> that feedback.
>
> That is not something we should dismiss by saying, "Ah, we don't  
> need those users anyway." We need all the users we can get -- and  
> removing any excuse *not* to use the app should be a really high  
> priority.
>
>> And again, the initial work may be trivial, but the work to support  
>> it is actually a lot
>
> Granted, it does require support. I'm just putting the premium on  
> dragging people into Windmill's gravitational pull. And I think you  
> might be surprised at how low maintenance just supporting fallback  
> would be.
>
> Thanks for listening. :)
>
>
> Matthew
>
>
>
>
> _______________________________________________
> Windmill-dev mailing list
> Windmill-dev at osafoundation.org
> http://lists.osafoundation.org/mailman/listinfo/windmill-dev