[Chandler-dev] Reviving the DESKTOP Test Automation Project

[Phillip J. Eby]

At 11:06 PM 9/5/2007 -0700, Mikeal Rogers wrote:
>On Sep 5, 2007, at 10:20 PM, Phillip J. Eby wrote:
>
>>At 08:55 PM 9/5/2007 -0700, Aparna Kadakia wrote:
>>>It is important to remember that the framework we adopt needs to
>>>support testing at the UI level and not within. This is more for
>>>functional testing which covers scenarios rather than unit level
>>>testing.
>>
>>Crossing over a thread here, I think it's important to point out
>>that "UI level" in this context presupposes an architecture in
>>which there *is* such a thing.
>>
>>One possible architecture refactoring is to have an "interaction
>>model" layer which encompasses a significant portion of what would
>>now be considered "UI level" code -- but *without* having any "UI"
>>code (i.e., graphics or I/O).  Such a system would be amenable to
>>automated functional testing without involving any actual GUI
>>code.  (See also my reply to John's post about transparent
>>persistence; this is the same visual/logical split I described there.)
>
>I figured this was what you were getting at.
>
>Removing the barrier of simulating end-user UI actions would make
>test debugging and maintenance worlds easier. I would say it makes
>test authoring easier, but if the recorded scripts framework takes
>off, it's hard to compete with recorded actions vs. sitting down and
>writing code.

Perhaps; of course, one of the main ideas behind doing something like 
this would be to make it possible to write the tests *before* the 
code exists that will be tested, as a specification of what the code 
is supposed to do.  There would still be a place for recorded 
scenarios, of course, since some tests will be added to cover 
scenarios that weren't envisioned at specification time.


>I can say writing these kinds of tests would be much
>easier than in the previous CATS system.
>
>Around the beginning of the year we looked in to what automated
>testing might be accomplished around this layer. I tried looking at
>what automated testing might be feasible that was somewhere between
>the current unittests and functional tests. What i found was that the
>desktop architecture didn't seem to provide a way to break apart
>these components and reliably test them in isolation, and that doing
>anything beyond what the current unittests did wasn't feasible in the
>short term ( the unittest coverage is already blurring the line
>between functional testing and unittesting).

Yep, this is exactly the issue.



>>In addition, it would eliminate the issue we've had with the
>>functional test frameworks that are distinct from the application
>>itself.  In an interaction-model design, the application *is* the
>>functional test framework, so there is no duplication of
>>functionality, documentation, learning, code getting out of sync, etc.
>
>I think I understand what you mean here, but just to make sure.
>You're saying that the visual components of the UI will be using the
>same API that would be used for testing,

Yep.


>If we do keep doing higher level automated tests ( and I imagine we
>will, hopefully using the new recorded scripts framework ) there is
>some value in watching them run periodically. Especially if the sole
>purpose of those tests becomes catching graphical rendering problems
>and we start automating the application semantics testing at other
>layers.

Of course.  Note, however, that in this model a functional test is a 
functional test whether the GUI is hooked up or not.  You could 
always run them with the GUI hooked up if you wanted to watch.  :)