[Dev] Proposal for new automation architecture
john at osafoundation.org
Fri Feb 10 12:04:36 PST 2006
Mikeal Rogers wrote:
> On Feb 10, 2006, at 11:20 AM, John Anderson wrote:
>> Mikeal Rogers wrote:
>>> On Feb 10, 2006, at 10:04 AM, Philippe Bossut wrote:
>>>> John Anderson wrote:
>>>>> Having written my first functional test yesterday I have some
>>>>> thoughts. The biggest problem I encountered when trying to write
>>>>> and debug tests is navigating all the layers:
>>>>> my test <-> CATS <-> CPIA Script <-> Chandler
>>>>> Fortunately I'm very familiar with Chandler, somewhat familiar
>>>>> with CPIA Script and CATS is small enough to grock without much
>>>>> effort. However, I suspect most developers would find all the
>>>>> layers daunting and trying to debug things would only make it worse.
>>>> Agree with that.
>>> One of the requirements is that the system be easy to use. Obviously
>>> there is another layer of complexity over what we do with CATS but
>>> it is still designed to be very easy for someone to pick up and
>>> start writing scripts and to see legible output. Part of the
>>> deliverables for the first version of this framework will be;
>>> -Command line python wrapper (much like do_tests, a script is
>>> imported and output is generated that is legible using a set of
>>> default parameters for the framework)
>> Yes, we've all been using variation of this for unit tests and
>> functional tests and find it useful. Also, when you get a
>> particularly tricky functional test failing somewhere deep in
>> Chandler, were a traceback isn't enough to diagnose the problem, it's
>> often handy to track it down in a debugger. So you might set things
>> up so you can attach in wing, e.g. include wingdbstub.py.
> This is probably a 2.0 feature. But, this framework has many
> advantages for debugging test cases.
> For example: Although the framework does all the reporting for you, it
> uses methods in the TestCase class which you inherit from to define
> the class for your own test. This means that in debugging you could
> write custom code and push additional reporting into the output object
> using same method the framework uses. The current CATS design requires
> you write custom code in the Logger object to do this, and the logger
> object has a bunch of logic to do fuzzy guessing as to which state the
> test is in. The idea is to have the framework do most of the heavy
> lifting for you but not keep you in such a tight box that you can't
> extend it without hacking at the framework itself.
I think I'm confused here. I was considering the case where one of my
functional tests starts failing and I don't understand the problem from
looking at a stack trace. In that situation I need to poke around in a
debugger. Are you saying that I can't do that until 2.0? Or that I can
only add more debugging code using the framework and rerun the test. Or
>>> -Sufficient Documentation ("Writting Chandler automation in 10
>>> minutes" style doc, extended OAF documentation for developers who
>>> wish to use non-default features in the system, and maybe most
>>> importantly GOOD documentation for the chandler test library that
>>> can facilitate both easy test script authoring and developer
>>> improvements to the chandler testing library itself.
>> I think much of what makes writing functional tests difficult has
>> little to do with your proposed framework, and more to do with how
>> you access the pieces of Chandler, do menu commands, click on
>> buttons, etc., i.e. the stuff that is mostly in CPIA script and CPIA.
> This is an issue that will be handled outside the framework. Again,
> the _framework_ is not Chandler specific.
> All of the QAUITestAppLib will need to be re-written to work with OAF,
> during this rewrite we will be taking care of many of the current
> issues developers have with it. I don't know if we can replace CPIA,
> after the framework is signed off on my next task is to come up with a
> plan to overhaul QAUITestAppLib.
I'm hoping that writing a new functional test for Chandler will be
almost the same as writing a CPIA script, and that a Chandler developer
needs to learn very little to write a functional test. For example, I
don't want to have to extend some QA library every time I need to drive
some new piece of the application.
> At the very least we intend to make the new library more modular and
> reuse code as much as possible. Making it easier for developers to
> alter/extend the functionality inside of inherited objects in the test
> scripts is a priority, as is making it easier for developers and QA to
> extend the library to cover new functionality in Chandler.
> The reason for all of the layers in the design is to make the
> framework non-application specific and extendable. The implementation
> will make these layers mostly transparent to the user. In the same way
> we can layer the new Chandler QAUITestAppLib to be non-CPIA specific
> in case you plan on implementing something better in the future.
"CPIA specific" means many things to many people, however, what I mean
by "CPIA specific" is simply that Chandler functional tests need to
access Chandler in a way that that is consistent with its internal
design, so I'm not sure we have the same idea of what "non-CPIA
>>> The output can be very customized using this framework, but the
>>> default output will be humanly legible and go directly to a file.
>>> Also, a -debug flag can be set, which sets all output in the
>>> framework to be processed as it comes in to the output object. This
>>> is no good for performance tests but will make debugging issues
>>> worlds easier than in CATS.
>>> To finish up, many of the extra layers that developer might find
>>> "daunting" will be transparent in the implementation, but the output
>>> that developers depend on (such as a tracebacks in the log if a
>>> failure occures) are made easy and reliable by this abstraction.
>>> I hope this alleviates your concerns.
>>>>> I think it would be preferable to make the small changes necessary
>>>>> to CPIA Script to make it appropriate for testing instead of
>>>>> adding another layer, e.g. CATS.
>>>> Improving CPIA Script to make scripting easier is indeed a good
>>>> idea. I don't think it will replace entirely a test harness though
>>>> like CATS or, better, OAF (proposed by Mikeal). There's a lot of
>>>> test functions (batch, log, data gathering and stats) that have no
>>>> place in a Chandler level scripting language. John, I suggest you
>>>> read Mikeal proposal
>>>> first. Keep in mind also that Mikeal is trying to solve a problem
>>>> that includes Chandler and Cosmo.
>>>>> Similarly, I think it's preferable to modify Chandler to eliminate
>>>>> some of CPIA Script.
>>>> What alternative to CPIA scripting do you propose? No scripting at
>>>> all? Another script mechanism? Leverage an existing one?
>>>> - Philippe
>>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>>> Open Source Applications Foundation "Dev" mailing list
>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>> Open Source Applications Foundation "Dev" mailing list
>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>> Open Source Applications Foundation "Dev" mailing list
More information about the Dev