[Windmill-dev] Python test suites

[Matthew Eernisse]

Mikeal,

Thanks for the response. I definitely appreciate your point of view. A 
few counterpoints/comments below:

Mikeal Rogers wrote:
> I think some history brings this in to context.

The prospective users that I'm talking about won't have any interest in 
Windmill history. All they know or care about is what they see when they 
come to the site.

> If you look at some of the setup/teardown in 
> our cosmo test repository you'll see how a large string of JSON format 
> can be transformed into a python calling using some windmill internals

I think there will be a non-trivial number of prospective users who also 
have no interest at all in the Windmill internals. :)

> Creating __init__.py is "really doing something", it's making it a valid 
> python module.

 From the point of view of a non-Python-programmer, that is a logical 
equivalent to nothing. :)

> It is NOT a barrier to entry to create an empty file. Especially when 
> this decreases the amount of ways tests are run in windmill.

Any extra step is a barrier.

The smaller API footprint is good, but I don't think that falling back 
to the simplest use case implies a change to the API. You could easily 
make the single method call handle both cases.

The extra work would be handling both cases on the back end, documenting 
it, and keeping it current. I'm not discounting the extra work.

> These imaginary users who have a handful of tests in a directory without 
> dependencies and very little programming experience just haven't 
> surfaced. If they do and they give us this feedback we can consider 
> changing it.

I'm glad you're willing to consider this if we get feedback that it 
would be good.

Note that we will receive exactly zero feedback from people who come to 
the site and then just walk away because the perception is that they 
have to use Python -- which is what "making your test directories a 
valid Python module" implies.

> Because the trade off is supporting another way tests are loaded in to 
> windmill, another command line option, another shell option.

See above. Why can't the same method call support both?

> Supporting this stuff adds complexity, it adds things for users to trip 
> over because they will quickly need to overstep this case for a more 
> complex one, which means they then need to use another option, creating 
> new files, maybe even change directory names.

Agreed. In this case, I'd vote that making it trivial for people to drop 
some JSON files into a directory and run their tests with zero effort is 
more of a priority.

I wouldn't mind a little more complexity down the road if the initial 
bump is zero. By then, they will already have a bunch of tests written, 
and will like what they see in Windmill. They will also have a visceral 
understanding of *why* they need this increased functionality.

> It's not ideological purity, it's actually what i believe is best for 
> our users based on their feedback and experience. I don't think it's a 
> barrier to adoption, I don't think anyone is every going to say "i was 
> using windmill but once I had to create that empty __init__.py file I 
> just gave up!".

See above. I'm not talking about existing users. I'm talking about 
prospective users.

I have a strong feeling there will be people who see their tests have to 
include .py files, and will say, "Ah, I don't want to learn Python, I 
already use X" and will walk away without giving us any of that feedback.

That is not something we should dismiss by saying, "Ah, we don't need 
those users anyway." We need all the users we can get -- and removing 
any excuse *not* to use the app should be a really high priority.

> And again, the initial work may be trivial, but the work to support it 
> is actually a lot

Granted, it does require support. I'm just putting the premium on 
dragging people into Windmill's gravitational pull. And I think you 
might be surprised at how low maintenance just supporting fallback would be.

Thanks for listening. :)


Matthew