[Chandler-dev] Reviving the DESKTOP Test Automation Project

[Grant Baillie]

Hi, Aparna et al

Overall, I can see why people like wx-level acceptance testing: there  
is a certain warm fuzzy feeling from knowing that the app in its  
entirety is behaving reasonably. (Even if you have to watch it to be  
sure :). The concerns I have are:

+ How would these recorded scripts be maintained? If we changed the  
app's layout, does someone have to go re-record them all?

+ Each script currently requires a launch and quit of the app (which  
is reasonable, given the nasty inter-test dependencies that have  
arisen with the current functional tests). In the scenario where we  
really cover all the current manual test specifications with recorded  
ones, how long will this take? Will developers be required to run  
them all before checkins?

+ I think I remember John/Dan saying scripts currently have to be  
recorded on Windows (something to do with keycodes). Is that likely  
to still be the case?

Possibly these all point at a more general issue, which is whether  
you see the recorded test suites as being part of the core  
development process, or something QA maintains on the side to help  
validate builds and find bugs.

--Grant

On 5 Sep, 2007, at 20:55, Aparna Kadakia wrote:

> Much in line with Katie's email, I too have been agonizing over the  
> Desktop Test Automation project lately. THe last few weeks have  
> been rather painful as we have manually gone thru the test specs to  
> validate the release. Clearly we need to make some kind of a  
> decision on the test framework for Desktop soon, esp in the light  
> of shorter release cycles.
>
> Over the last 3 years, we have seen CATS, the CPIA script based  
> test automation framework and then more recently the Script/ 
> Recording framework, both trying to fill the gap of insufficient UI  
> automated tests with easy test development. Even though CATS is a  
> full-blown fully functional test framework, the test development  
> requires adding methods to its libraries which require good  
> understanding of the internal architecture of the component under  
> test. This turned out to be not so scalable after all.
> To overcome this handicap, John initiated the effort on Script/ 
> Recording framework. It is still premature to comment on how  
> successful this is as it is far from complete. We still have a  
> number of significant things to add support for e.g. dragging in  
> cal canvas, support for modal dialogs etc. It is important to  
> remember that the framework we adopt needs to support testing at  
> the UI level and not within. This is more for functional testing  
> which covers scenarios rather than unit level testing.
>
> Since John will be back in the office next week and we will have  
> wrapped up the Preview release, I propose we re-start the  
> discussion on the Desktop test automation again.
>
> The plan is to evaluate the progress on script/recording framework  
> and if it is a viable next option. Also the idea is to analyze  
> collectively if any architectural changes are necessary to build a  
> more reliable test system.
>
> Proposed Date : Tuesday, Sept 11 2007
> Proposed Time : 2:30 PM (taking the Design Discussion slot since we  
> aren't having those)
> Conference Room : Shambala
>
> The high level goals of the automation framework remain the same:
> 1. easy UI test development
> 2. easy debugging of test failures (capturing and outputting  
> errors, stack traces in log files etc)
> 3. very clear passed and failed results from test execution
> 4. flexibility of running a single test v/s a full suite
> 5. easy installation of the framework and  execution of testscripts
>
> Let me know if that works for everyone.
>
> Thanks
> Aparna
>
>
>
>
>
>
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "chandler-dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/chandler-dev