[Chandler-dev] Script-recording/playback as test framework

[Mikeal Rogers]

> Personally, I'd vote against both asserts (which don't fire if  
> you're running optimized, as we'd want for the performance tests)  
> and the previous pass/fail approach (which is baroque and error- 
> prone). Instead, why not have tests inherit from unittest.TestCase,  
> since:

I'm not against integrating with unittest, but I'm hesitant to use  
unittest _every_ time we run a script. One thing I forgot to mention  
was that we plan on adding a submenu in test, that has a list of all  
the standard tests we have written, that can be run at any time. A  
big difference with the new test approach is that we're using one  
feature, script recording/playback, for a few different facilities.  
If someone records a script and plays it back, they don't necessarily  
want to run a "test", they could just be automating some piece of  
their workflow.

> 1) That has a pretty expressive API for validating state  
> (TestCase.failUnless, TestCase.failUnlessEqual, etc)
> 2) Tests that depend on setting up shared data can use inheritance,  
> usually in conjunction with setUp() methods.

With the way we have things now we're trying to keep people from  
needing to write code to get a test going. In this case how do we  
even handle tests that use shared data? This may be a feature we want  
to build for continuous integration (tinderbox) that's outside of the  
regular script recording/playback.

> 3) Less wheel reinvention, less having developers have to learn new  
> APIs.

So there shouldn't be any api learning for people writing tests,  
because writing tests shouldn't require any code writing. The script  
recorder will output a test a file with a "run" method that contains  
everything needed to run the test.

I think for the continuous integration case (tinderbox) we'll need to  
run the tests a bit differently that we do when selecting them from a  
file or menu as the case is now. A good idea would be to write a  
TestLoader class that pulls all the test methods in from all the  
files we've saved in script-recording and assigns them as methods in  
TestCase clases so that we can run the full test suite in unittest.

-Mikeal