[Windmill-dev] Direction of the project and plans for 0.4

[Mikeal Rogers]

On Dec 19, 2007, at December 19, 200712:20 PM, Adam Christian wrote:

> I can't handle that thread anymore.
>
> I think all of the things you have on the list for .4 are  
> important... I think the game plan should proceed as follows:
>
> 	Get js tests straightened out
> 	Get dynamic negations on assertions done

We could push this in an 0.3.x release. I don't think it needs to be  
attached to 0.4 necessarily.

>
> 	Get xmlrpc out the door
> 	Get something reasonable hacked in for running more than one JSON  
> test from a directory

I realized all of my responses to this were in emails I didn't send  
and decided to start this thread instead.

Ok, for wxWindmill I think it's a better experience to allow people to  
select multiple JSON files rather than a directory, for a couple  
reasons.

1. This can use the same, existing method of loading single JSON  
files, just doing them in succession.
2. I think Jacob said that he has to specify whether you are selecting  
a directory of file when he creates this menu option, so it means we  
get out of creating yet another menu option.
3. I think for the use case that you only have one directory of un- 
ordered tests, being able to select subsets is a nice feature.

I can change the load_json_test_file option to take a comma delimited  
list of files to support this throughout windmill.

If we get feedback that people need that option to take a directory  
then i will add it. I won't let the option recursively parse sub  
directories. And I won't change the existing load_test option to take  
a directory that is not a valid python module.

There are huge differences in how the tests are loaded in to windmill  
and run between these two methods, I can't stress enough how different  
they are and how the experience will change when they move to using  
load_test and run_test rather than load_json_test_file, which I why I  
won't stick this functionality in to those options.

>
>
> At this point I think we should do some sort of a milestone launch.  
> (0.4 super ninja)
> 	
> 	I do agree that building binaries for all platforms every time  
> there is a dot release is pure insanity. However recently I have  
> been in contact with a relatively large handful of people who would  
> pull down a binary and play with Windmill if it was built... these  
> people won't really care if it's not _the_ latest dot release. One  
> thing I have noticed is that there are so many edge cases, "Look at  
> such and such a page and try recording and playing back a click" I  
> look at it and realize that it's something completely legitimate but  
> I never would have thought of the exact scenario on my own. Having a  
> large set of people playing with it against a large set of pages  
> could yield vast improvements. (Recently I have spent some time just  
> surfing the web testing against pages and so far we are looking  
> pretty good)
>
> 	Anywhoo I am proposing a set of built binaries for each platform  
> for .4, and then after that it can be only big releases, massive  
> changes, something crucial, or user request to update the binaries.  
> For example if we had done a round for 0.3, we certainly would have  
> waited until this next release to build another round. Things were  
> changing to quickly, and many of the changes an avg joe user base  
> wouldn't have even noticed.
>
> 	Does this sound reasonable? If Mikeal can help me get setup to do  
> this on the different platforms, I volunteer to do the building,  
> updating docs and anything else that goes along with this as I do  
> feel it is important. I have had this at the back of my mind since  
> OSCON.

Ok, I think we can do Leopard and Windows XP. Not Vista (cause I'm not  
running it and don't want to), and not Tiger (for the same reason).

No Linux. If you're on linux you should be able to use the command  
line, and you can launch this UI from the command line so we don't  
need to build you a binary :) I also don't want to deal with updating  
package management systems.

Again, we have to move wxWindmill out of the main windmill module in  
to it's own project.

>
>
> I've kind of lost myself in the thread, but this scenario would  
> really scratch my itch, did I miss any anything?
>
> Adam
>
> On Dec 19, 2007, at 1:45 AM, Mikeal Rogers wrote:
>
>> On Dec 19, 2007, at December 19, 20071:12 AM, Adam Christian wrote:
>>
>>> Comments inline
>>> On Dec 19, 2007, at 12:33 AM, Mikeal Rogers wrote:
>>>
>>>> I had a couple big emails that replied to the recent thread but I  
>>>> decided instead to delete them and start fresh.
>>>>
>>>> The project has come a long way, and every time we get over one  
>>>> hurdle and see some adoption everyone starts clamoring to add new  
>>>> features and bring in more users and add complexity.
>>>>
>>>> Here is the situation we're in.
>>>>
>>>> - We have limited contributors with limited time.
>>> Because currently the people that can use the project are limited  
>>> to 'tech savvy' people, keeping any buzz from happening large  
>>> enough to expose us to large amounts of possible committers.
>>>
>>>>
>>>> - We have a couple large holes in our functionality that we all  
>>>> need plugged.
>>> These holes are SSL, concurrent browser sessions, and what else?
>>
>> More supported authoring languages.
>>
>>>
>>>>
>>>> - We have a large list of features that we consider for whatever  
>>>> reason to be "barriers to adoption".
>>>> Right now we have a lot of solid functionality, the windmill  
>>>> controller API is particularly nice and far above what anyone  
>>>> else currently has in competing products. We can't let that cloud  
>>>> the fact that we still have a long way to go.
>>>>
>>>> We have a series of half done features and a lot of areas that  
>>>> need re-factoring in order to increase the long term  
>>>> maintainability of the project. This is what we need to focus on  
>>>> _now_, because it's something that is much harder to get done the  
>>>> more we add complexity.
>>> If we are going to do these large re-factors that I have been  
>>> hearing about since 0.1 we should probably do it now because I  
>>> think 'future possibilities' of re-factoring is a bad reason not  
>>> to become user friendly to avg joe.
>>
>> What i'm talking about are the 4 areas I'm suggesting for 0.4,  
>> nothing more.
>>
>>>
>>>
>>>> We're seeing some great users coming by, yes they are "tech-savy"  
>>>> but to be frank, we can't feasibly support people who aren't  
>>>> right now. These are the kinds of users we should have at this  
>>>> stage, and should try to appeal to more instead of going after  
>>>> people with less experience that will require more support.
>>> This sounds terribly backwards to me, selenium found a large  
>>> contrib base after they had a firefox plugin so that everyone and  
>>> their brother could write tests. Maybe these tests were just to be  
>>> opened and played back over and over, but they were the gateway  
>>> drug making users want more flexibility, and those are the people  
>>> that turned into contributors.
>>>
>>> Our ability to 'support' all these users will still be an issue if  
>>> we have a large influx of 'tech savy' users. After we remove the  
>>> barriers to entrance get get users, they can help each other.  
>>> Without this happening we simply don't have the time to support  
>>> more than 5 of the most 'tech savvy' users writing to the list  
>>> about issues.
>>
>> Selenium gets out of this problem by simply not supporting their  
>> users, we know this more than anyone.
>>
>>>
>>>
>>>> In all honestly, this project can't sustain the work it will take  
>>>> to support a large "non-tech-savy" user base. We need to focus on  
>>>> appealing to more potential contributors, in other words "tech- 
>>>> savy" people, to make the projects sustainable. For this purpose  
>>>> it's important to focus on the Python and JavaScript related  
>>>> features, as the JSON format is targeted at non-developers.
>>> Again, we can't support any user base on our own really, where are  
>>> these contributors magically going to come from if this project  
>>> doesn't start to get out there?
>>
>> We're seeing a steady and healthy increase in usage. And we seem to  
>> be retaining most of that usage. I'm not talking about any magic,  
>> I'm talking about sustaining the growth we currently see and not  
>> loading ourselves up with more than we can do.
>>
>>>
>>>
>>> I disagree, and feel that the few things keeping us from being a  
>>> binary download and two button clicks away from recording tests as  
>>> 'avg joe' is just as important if not more for the community aspect.
>>
>> Again, I just don't know how we would support this person right now.
>>
>> Every user I can think of in recent memory found a bug that we  
>> fixed or a problem with the documentation. After our last major  
>> release we shipped 4 immediate bugfix releases the proceeding week.  
>> What would have happened if thousands of "average joe's" were using  
>> windmill when we shipped that? What if they were all downloading  
>> 70meg binaries? I'm not even going to go in to the issue of  
>> actually building and distributing them, updating relevant package  
>> managements systems, etc.
>>
>>>
>>>
>>>>
>>>> Here is what I would like to see for 0.4;
>>>>
>>>> 1. Completing JavaScript test authoring.
>>>> 	We cannot defer this work. Currently there are zero unit tests  
>>>> checked in for this, there is zero continuous integration  
>>>> support, and zero XMLRPC support for this framework. This work  
>>>> has to happen now, if we're going to have 3 supported authoring  
>>>> paths then they need to be in and continually supported. I would  
>>>> like Adam to take on portions of this from mde if at all possible.
>>>> 2. wxWindmill Alpha for Windows.
>>>> 	There are several caveats. Windows ONLY, we will not produce  
>>>> binaries for platforms other than windows. We will create the  
>>>> infrastructure to host and create new builds but we won't be  
>>>> pushing new binary bits for every bugfix release. And most  
>>>> importantly, wxWindmill will be moved out of the primary windmill  
>>>> module in to, more or less, it's own project, in the windmill  
>>>> trunk.
>>>
>>> I disagree, I think anybody with a Mac or Linux box who wants to  
>>> try out windmill with the 10 minutes they have before they catch  
>>> the train should be able to pull down a binary, fire it up and off  
>>> they go. I can see a lot of tech savvy people doing this and  
>>> deciding that it's pretty cool and that there is something they  
>>> would like to add, so they start digging into the source.
>>
>> We shipped 4 bugfix releases last week.
>>
>> We have a hard time keeping up with updating the code we already  
>> have and pushing it out. How do you plan on supporting 3 platform  
>> builds every time we need one of these releases? Not to mention  
>> you're opening a huge can of worms and support on all platforms at  
>> once.
>>
>> Also remember that I didn't write any of the wx code, I'm not the  
>> primary owner of it, and have no idea how to fix bugs in it. Our  
>> domain owner for this stuff, Jacob, is about to take off to  
>> Australia until the summer so I consider it an aggressive goal just  
>> to support a windows build. If we don't have an active owner there  
>> is no way we can even begin to consider what you're asking.
>>
>> I'm willing to put work in to Windows because right now installing  
>> windmill on Windows is even hard for me. There is no question it's  
>> a barrier to adoption even by very tech savy users.
>>
>> We also need to stagger the impact of supporting pre-built binaries  
>> over a longer period of time. I hope the next release will be done  
>> in January and that's just not enough time to figure out how we're  
>> going to do all of this.
>>
>>
>>>>
>>>> 3. Improved XMLRPC Support.
>>>> 	The largest barrier for people creating new language libraries  
>>>> is the XMLRPC support. It's also made features in windmill hard  
>>>> to develop as all the tests have moved to being XMLRPC based.
>>> I agree, this is another important piece of the community puzzle.
>>>
>>>>
>>>> 4. IDE UI re-arch.
>>>> 	I've heard Adam grumble about this saying it's in need of some  
>>>> refactoring. We're adding new stuff all the time and if the  
>>>> architecture becomes a bottle neck this will be very problematic.
>>> This code is ugly, however every-time we run into something I  
>>> think is going to be an impossible feat with this code the way it  
>>> is it turns out that it isn't. There is one big function here that  
>>> returns a DOM element representing an action and I have been  
>>> slowly cleaning it up as we go.
>>>
>>>> Notice that, other than XMLRPC, none of the external API's are  
>>>> changing. There are no new features in this list, just  
>>>> solidifying things we're had for a while. There are no features  
>>>> directly pointed at end-user adoption. The wxWindmill work is  
>>>> also targetted at windmill supporting itself more like an  
>>>> importable python module, with greater facilities for re-use to  
>>>> encourage adoption in other projects/frameworks.
>>> If we want to make a list of things that need to be done before we  
>>> drop the big "become available to the world" bomb, then I think we  
>>> should write it out, do it, then do the very reasonable workload  
>>> it will take to make the project available for the masses.
>>>
>>> XMLRPC, Binaries, run a folder of JSON tests. I can live with  
>>> 'sorry about SSL' for now.
>>>
>>> There is some Chandler Desktop history that we really need to  
>>> learn from here. Mass of users is important to an open source  
>>> project even if it isn't where you want it 'yet', continually  
>>> tweaking and 're-factoring' can lead to 5 years without really  
>>> putting the deserved energy into exposure, and probably most  
>>> importantly -- engineers are never 'happy enough' to pull the  
>>> trigger. As long as API's aren't changing, we can refactor under  
>>> the hood as many times as we want.. and maybe if we have 100  
>>> committers we won't have to do all that work ourselves.
>>
>> Dude, we aren't Microsoft, we don't have big "available to the  
>> world" bombs.
>>
>> We are seeing a steady increase in adoption and we are supporting  
>> users as they come along very nicely. We're seeing good steady  
>> improvements in the product in terms of features and stability.
>>
>> We don't have the time or resources to do more than we already are  
>> until we get more contributors. And i think this will happen  
>> naturally, there is no doubt in my mind.
>>
>> -Mikeal
>>
>>
>>
>>
>> _______________________________________________
>> Windmill-dev mailing list
>> Windmill-dev at osafoundation.org
>> http://lists.osafoundation.org/mailman/listinfo/windmill-dev
>