[Chandler-dev] Script-recording/playback as test framework

[Grant Baillie]

On 26 Dec, 2006, at 17:36, Mikeal Rogers wrote:

>> Personally, I'd vote against both asserts (which don't fire if  
>> you're running optimized, as we'd want for the performance tests)  
>> and the previous pass/fail approach (which is baroque and error- 
>> prone). Instead, why not have tests inherit from  
>> unittest.TestCase, since:
>
> I'm not against integrating with unittest, but I'm hesitant to use  
> unittest _every_ time we run a script. One thing I forgot to  
> mention was that we plan on adding a submenu in test, that has a  
> list of all the standard tests we have written, that can be run at  
> any time. A big difference with the new test approach is that we're  
> using one feature, script recording/playback, for a few different  
> facilities. If someone records a script and plays it back, they  
> don't necessarily want to run a "test", they could just be  
> automating some piece of their workflow.
>
>> 1) That has a pretty expressive API for validating state  
>> (TestCase.failUnless, TestCase.failUnlessEqual, etc)
>> 2) Tests that depend on setting up shared data can use  
>> inheritance, usually in conjunction with setUp() methods.
>
> With the way we have things now we're trying to keep people from  
> needing to write code to get a test going. In this case how do we  
> even handle tests that use shared data? This may be a feature we  
> want to build for continuous integration (tinderbox) that's outside  
> of the regular script recording/playback.
>
>> 3) Less wheel reinvention, less having developers have to learn  
>> new APIs.
>
> So there shouldn't be any api learning for people writing tests,  
> because writing tests shouldn't require any code writing. The  
> script recorder will output a test a file with a "run" method that  
> contains everything needed to run the test.

How are you supposed to verify the data without writing code? And you  
said '"run" method' -- what class is that method defined on? If  
you're spitting out a script anyway, it's just as easy to stick in  
the class definition and "def runTest(self)".

> I think for the continuous integration case (tinderbox) we'll need  
> to run the tests a bit differently that we do when selecting them  
> from a file or menu as the case is now. A good idea would be to  
> write a TestLoader class that pulls all the test methods in from  
> all the files we've saved in script-recording and assigns them as  
> methods in TestCase clases so that we can run the full test suite  
> in unittest.
>
> -Mikeal
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "chandler-dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/chandler-dev