[Chandler-dev] Sane defaults for new Chandler testing framework

[Mikeal Rogers]

The short answer is: "because it would have taken longer."

The long answer takes a little explanation.

We examined using the Python logger for OAF (Open Automation  
Framework), the new framework for all our test tools, and came to a  
decision which I'll get to later. The reason we wrote this logger,  
and this framework for CATS was to free up our time from maintaining  
and dealing with all the current CATS issues so that we could  
actually have time to write the new framework. The Python logger has  
lots of features we don't need and saves us very little, we still  
need to gather state information which the python logger has no  
concept of, we still need to capture tracebacks and log failure/ 
cleanup state/write traceback to stdout and logfile. Everything we  
did for this framework directly translates to work we have to do  
anyway when we move to CATS with the exception of the logger so we  
didn't spend any more time on it than necessary.

The conclusion we came to for OAF was, we won't be logging to any  
medium by default but can easily support it via customized output. In  
OAF output the most important thing is integrity of the timing  
numbers, so all the output is serialized and sent to the output  
object in tuples. The output can be automatically offloaded from the  
output object (like to a central server) or stored and processed  
later. When the processing gets called the output can be pushed to  
file, stdout, SQL statements, or the python logger.

The reason for this complexity is that the new framework will need to  
handle multiple resources at once (cosmo, chandler, scooby) for  
complete end to end testing. So the test output has to be serialized  
and offloaded because PASS/FAIL is determined by the central server  
framework that is running all the tests and managing the resources.

I hope that answers your question.

-Mikeal

On May 23, 2006, at 5:15 PM, Bryan Stearns wrote:

> Just curious: why aren't we using python's logging for tests? We're  
> already using it for the rest of the app, and it's got easily  
> overridable defaults, levels, alternative streams of output (which  
> could be used for the minimal output that tbox needs to detect  
> failures)...
>
> ...Bryan
>
> Mikeal Rogers wrote:
>
>> Bear hasn't gotten back to me yet on exactly what he would like  
>> for a  format on tinderbox.
>>
>> The default we have for human use shouldn't necessarily be the  
>> same  as tinderbox. When bear integrated the functional test tool  
>> I wrote  for cosmo I believe he masked all output and just read  
>> the final  summary output.
>>
>> It's trivial to have different output setting on tinderbox and   
>> regular human use.
>>
>> -Mikeal
>>
>>
>> On May 23, 2006, at 4:03 PM, Andi Vajda wrote:
>>
>>>
>>> On Tue, 23 May 2006, Mikeal Rogers wrote:
>>>
>>>> Hiya,
>>>
>>>
>>>> All of these can be easily set manually, but the question we  
>>>> have  is "what should the defaults be when running individual  
>>>> tests and  the functional test suite?".
>>>
>>>
>>> The defaults should be whatever is used by the tinderboxes.
>>>
>>> Andi..
>>>
>>
>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>
>> Open Source Applications Foundation "chandler-dev" mailing list
>> http://lists.osafoundation.org/mailman/listinfo/chandler-dev
>
>