[Dev] Proposal for new automation architecture

[John Anderson]

Mikeal Rogers wrote:
>
> On Feb 10, 2006, at 11:20 AM, John Anderson wrote:
>
>>
>>
>> Mikeal Rogers wrote:
>>>
>>> On Feb 10, 2006, at 10:04 AM, Philippe Bossut wrote:
>>>
>>>>
>>>>
>>>> John Anderson wrote:
>>>>> Having written my first functional test yesterday I have some 
>>>>> thoughts. The biggest problem I encountered when trying to write 
>>>>> and debug tests is navigating all the layers:
>>>>>
>>>>> my test <-> CATS <-> CPIA Script <-> Chandler
>>>>>
>>>>> Fortunately I'm very familiar with Chandler, somewhat familiar 
>>>>> with CPIA Script and CATS is small enough to grock without much 
>>>>> effort. However, I suspect most developers would find all the 
>>>>> layers daunting and trying to debug things would only make it worse.
>>>> Agree with that.
>>>
>>> One of the requirements is that the system be easy to use. Obviously 
>>> there is another layer of complexity over what we do with CATS but 
>>> it is still designed to be very easy for someone to pick up and 
>>> start writing scripts and to see legible output. Part of the 
>>> deliverables for the first version of this framework will be;
>>>
>>> -Command line python wrapper (much like do_tests, a script is 
>>> imported and output is generated that is legible using a set of 
>>> default parameters for the framework)
>> Yes, we've all been using variation of this for unit tests and 
>> functional tests and find it useful. Also, when you get a 
>> particularly tricky functional test failing somewhere deep in 
>> Chandler, were a traceback isn't enough to diagnose the problem, it's 
>> often handy to track it down in a debugger. So you might set things 
>> up so you can attach in wing, e.g. include wingdbstub.py.
>
> This is probably a 2.0 feature. But, this framework has many 
> advantages for debugging test cases.
>
> For example: Although the framework does all the reporting for you, it 
> uses methods in the TestCase class which you inherit from to define 
> the class for your own test. This means that in debugging you could 
> write custom code and push additional reporting into the output object 
> using same method the framework uses. The current CATS design requires 
> you write custom code in the Logger object to do this, and the logger 
> object has a bunch of logic to do fuzzy guessing as to which state the 
> test is in. The idea is to have the framework do most of the heavy 
> lifting for you but not keep you in such a tight box that you can't 
> extend it without hacking at the framework itself.
I think I'm confused here. I was considering the case where one of my 
functional tests starts failing and I don't understand the problem from 
looking at a stack trace. In that situation I need to poke around in a 
debugger. Are you saying that I can't do that until 2.0? Or that I can 
only add more debugging code using the framework and rerun the test. Or 
something else?
>
>>> -Sufficient Documentation ("Writting Chandler automation in 10 
>>> minutes" style doc, extended OAF documentation for developers who 
>>> wish to use non-default features in the system, and maybe most 
>>> importantly GOOD documentation for the chandler test library that 
>>> can facilitate both easy test script authoring and developer 
>>> improvements to the chandler testing library itself.
>> I think much of what makes writing functional tests difficult has 
>> little to do with your proposed framework, and more to do with how 
>> you access the pieces of Chandler, do menu commands, click on 
>> buttons, etc., i.e. the stuff that is mostly in CPIA script and CPIA.
>
> This is an issue that will be handled outside the framework. Again, 
> the _framework_ is not Chandler specific.
>
> All of the QAUITestAppLib will need to be re-written to work with OAF, 
> during this rewrite we will be taking care of many of the current 
> issues developers have with it. I don't know if we can replace CPIA, 
> after the framework is signed off on my next task is to come up with a 
> plan to overhaul QAUITestAppLib.
I'm hoping that writing a new functional test for Chandler will be 
almost the same as writing a CPIA script, and that a Chandler developer 
needs to learn very little to write a functional test. For example, I 
don't want to have to extend some QA library every time I need to drive 
some new piece of the application.
>
> At the very least we intend to make the new library more modular and 
> reuse code as much as possible. Making it easier for developers to 
> alter/extend the functionality inside of inherited objects in the test 
> scripts is a priority, as is making it easier for developers and QA to 
> extend the library to cover new functionality in Chandler.
>
> The reason for all of the layers in the design is to make the 
> framework non-application specific and extendable. The implementation 
> will make these layers mostly transparent to the user. In the same way 
> we can layer the new Chandler QAUITestAppLib to be non-CPIA specific 
> in case you plan on implementing something better in the future.
"CPIA specific" means many things to many people, however, what I mean 
by "CPIA specific" is simply that Chandler functional tests need to 
access Chandler in a way that that is consistent with its internal 
design, so I'm not sure we have the same idea of what "non-CPIA 
specific" means.
>
>>>
>>> The output can be very customized using this framework, but the 
>>> default output will be humanly legible and go directly to a file.
>>>
>>> Also, a -debug flag can be set, which sets all output in the 
>>> framework to be processed as it comes in to the output object. This 
>>> is no good for performance tests but will make debugging issues 
>>> worlds easier than in CATS.
>>>
>>> To finish up, many of the extra layers that developer might find 
>>> "daunting" will be transparent in the implementation, but the output 
>>> that developers depend on (such as a tracebacks in the log if a 
>>> failure occures) are made easy and reliable by this abstraction.
>>>
>>> I hope this alleviates your concerns.
>>>
>>> -Mikeal
>>>
>>>>> I think it would be preferable to make the small changes necessary 
>>>>> to CPIA Script to make it appropriate for testing instead of 
>>>>> adding another layer, e.g. CATS.
>>>> Improving CPIA Script to make scripting easier is indeed a good 
>>>> idea. I don't think it will replace entirely a test harness though 
>>>> like CATS or, better, OAF (proposed by Mikeal). There's a lot of 
>>>> test functions (batch, log, data gathering and stats) that have no 
>>>> place in a Chandler level scripting language. John, I suggest you 
>>>> read Mikeal proposal 
>>>> (http://wiki.osafoundation.org/bin/view/Projects/OpenAutomationFramework) 
>>>> first. Keep in mind also that Mikeal is trying to solve a problem 
>>>> that includes Chandler and Cosmo.
>>>>> Similarly, I think it's preferable to modify Chandler to eliminate 
>>>>> some of CPIA Script.
>>>> What alternative to CPIA scripting do you propose? No scripting at 
>>>> all? Another script mechanism? Leverage an existing one?
>>>>
>>>> Cheers,
>>>> - Philippe
>>>>
>>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>>>
>>>> Open Source Applications Foundation "Dev" mailing list
>>>> http://lists.osafoundation.org/mailman/listinfo/dev
>>>
>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>>
>>> Open Source Applications Foundation "Dev" mailing list
>>> http://lists.osafoundation.org/mailman/listinfo/dev
>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>
>> Open Source Applications Foundation "Dev" mailing list
>> http://lists.osafoundation.org/mailman/listinfo/dev
>