[Chandler-dev] Reviving the DESKTOP Test Automation Project

[D John Anderson]

On Sep 6, 2007, at 8:21 AM, Grant Baillie wrote:

> Hi, Aparna et al
>
> Overall, I can see why people like wx-level acceptance testing:  
> there is a certain warm fuzzy feeling from knowing that the app in  
> its entirety is behaving reasonably. (Even if you have to watch it  
> to be sure :). The concerns I have are:
>
> + How would these recorded scripts be maintained? If we changed the  
> app's layout, does someone have to go re-record them all?

The recorded tests are pretty independent of the apps layout. For  
example if you move a widget they should still work. That said, if  
you remove a widget or change what a widget does, the script will  
report that as a failure since the point of the script is to verify  
that the widget does what it did when recorded.

This will lead to situations were, when the application logic or  
features change, scripts will need to be rerecorded. But hopefully  
this can be done easier than writing some code and maintaining it by  
hand like we have done in the past.

>
> + Each script currently requires a launch and quit of the app  
> (which is reasonable, given the nasty inter-test dependencies that  
> have arisen with the current functional tests). In the scenario  
> where we really cover all the current manual test specifications  
> with recorded ones, how long will this take? Will developers be  
> required to run them all before checkins?

Starting with a fresh Chandler before each test is a good place to  
start since it minimized dependencies of one script on the others. If  
it turns out that this is a performance bottleneck we could explore  
alternatives, e.g. starting with a prebuilt repository or after each  
script is run, roll back the repository to a known state, or require  
each script remove side effect of the test (e.g. delete collections  
it created, change the view back to a known state). We could also  
have some scripts that run individually, and some that run back to back.

Today, few developers run all the tests on all the platform before  
checkin. It might make sense to have developers run a subset of tests  
before checkin but let the tinderboxes run a more complete set.

>
> + I think I remember John/Dan saying scripts currently have to be  
> recorded on Windows (something to do with keycodes). Is that likely  
> to still be the case?

Yes, today scripts need to be recorded on windows if you want them to  
playback on the other platforms. This is because windows records  
platform specific key codes which aren't recorded on the other  
platforms. These kind of problems are solvable, but it's more a  
question of it is worth the effort. I had hoped to start using the  
scripts to gain some experience before I had fixed all the problems.

I still don't have enough experience with recorded scripts to  
understand how much work it will be to script every possible user  
action. However, I think there is a reasonable chance we could have  
better than 90% coverage with a few weeks more effort.

Finally, I don't think recorded scripts need to be the only way to  
test Chandler, but I'm hopeful that they can be an significant  
improvement over our our old functional test framework.

John