[Chandler-dev] Reviving the DESKTOP Test Automation Project

[Mikeal Rogers]

On Sep 5, 2007, at 10:20 PM, Phillip J. Eby wrote:

> At 08:55 PM 9/5/2007 -0700, Aparna Kadakia wrote:
>> It is important to remember that the framework we adopt needs to  
>> support testing at the UI level and not within. This is more for  
>> functional testing which covers scenarios rather than unit level  
>> testing.
>
> Crossing over a thread here, I think it's important to point out  
> that "UI level" in this context presupposes an architecture in  
> which there *is* such a thing.
>
> One possible architecture refactoring is to have an "interaction  
> model" layer which encompasses a significant portion of what would  
> now be considered "UI level" code -- but *without* having any "UI"  
> code (i.e., graphics or I/O).  Such a system would be amenable to  
> automated functional testing without involving any actual GUI  
> code.  (See also my reply to John's post about transparent  
> persistence; this is the same visual/logical split I described there.)

I figured this was what you were getting at.

Removing the barrier of simulating end-user UI actions would make  
test debugging and maintenance worlds easier. I would say it makes  
test authoring easier, but if the recorded scripts framework takes  
off, it's hard to compete with recorded actions vs. sitting down and  
writing code. I can say writing these kinds of tests would be much  
easier than in the previous CATS system.

Around the beginning of the year we looked in to what automated  
testing might be accomplished around this layer. I tried looking at  
what automated testing might be feasible that was somewhere between  
the current unittests and functional tests. What i found was that the  
desktop architecture didn't seem to provide a way to break apart  
these components and reliably test them in isolation, and that doing  
anything beyond what the current unittests did wasn't feasible in the  
short term ( the unittest coverage is already blurring the line  
between functional testing and unittesting).

>
> In addition, it would eliminate the issue we've had with the  
> functional test frameworks that are distinct from the application  
> itself.  In an interaction-model design, the application *is* the  
> functional test framework, so there is no duplication of  
> functionality, documentation, learning, code getting out of sync, etc.

I think I understand what you mean here, but just to make sure.  
You're saying that the visual components of the UI will be using the  
same API that would be used for testing, so this would cut out the  
middleman "scripting" library on top of the application. This is  
somewhat analogous to what John has done with the recorded scripting  
framework as it uses the wx event system directly to record and  
playback simulated users ineractions, and the blocks system to gather  
assertion information about the data being set.

>
> Of course, that approach won't catch graphical glitches and  
> platform-specific wx behavioral quirks, but if I understand  
> correctly, we don't have a way to catch them in an automated  
> fashion now, do we?  (When I run functional tests and the screen  
> sometimes does funky things, it doesn't seem to cause any test  
> failures.)

Over on Cosmo we find a lot of value in watching our UI functional  
tests run. Because of the browser architecture it's nearly impossible  
to tell when "extra" UI shows up or if a node exists but isn't  
rendering. Adam does this at least a few times a week.

If we do keep doing higher level automated tests ( and I imagine we  
will, hopefully using the new recorded scripts framework ) there is  
some value in watching them run periodically. Especially if the sole  
purpose of those tests becomes catching graphical rendering problems  
and we start automating the application semantics testing at other  
layers.

-Mikeal