[Chandler-dev] CATS3

[Mike Taylor]

On Jun 8, 2007, at 4:15 AM, Mikeal Rogers wrote:

>> I thought the current test framework already included this 
>> information within FunctionalTestSuite.  This single file method can 
>> easily be extended to include recorded tests and would also have the 
>> benefit of being a single place to run both Functional and recorded 
>> tests.  After all, are not recorded tests just another functional 
>> test?
>
> It's always been my impression that we will be replacing the "old" 
> functional tests with recorded tests. Running them from the same 
> framework wasn't really considered and isn't how they run now. There 
> were many bugs/feature requests for the old framework which we solved 
> in the new one and as far as I know we've always intended on phasing 
> out the old functional tests and replacing them with recorded scripts.

phasing out the older tests with new recorded tests makes perfect sense 
- I just wasn't aware at all that a complete replacement of the 
framework was in the plans.  but if that's what the qa team recommend 
then ok.

>> Currently we are running tests by filename - is there a plan to have 
>> it run some other way?
>
> To clarify, the _framework_ only thinks about tests in terms of test 
> modules. You can reference them by filename but it's all the same to 
> the framework.
>
>> If we are no longer running recorded tests by filename then how will 
>> they be run by tinderbox?  How will developers use the existing 
>> --recordedTest command line parameter to try and reproduce a test 
>> failure?
>
> The --recordedTest command line runs execute_frame, which uses a 
> prebuilt dictionary of tests that are collected as modules using 
> get_test_modules. Sure, the command line references them by filename 
> but it's really just treated as a dictionary key, the value of which 
> is a test module.
>
> Thanks to Jacob, we have a new dynamic menu under the Tools -> 
> scripting menu which lists all available tests ( and doesn't observer 
> platform exclusions by design ) in which you can run each test inside 
> of chandler and we think this will end being used by developers more 
> often for debugging than the --recordedTest command line parameter.

As long as one thing remains constant - if I run "rt.py 
RecordedTestName" and then run "chandler 
--recordedTest=RecordedTestName" and the same *single* test runs we are 
ok.  What would be bad is if the two runs cause different code paths to 
be run.

It sounds like the plan is to migrate to this new framework and that's 
fine as long as we get a chance to discuss the parameters of the 
framework and it's behaviour before it's phased in as a replacement.

---
Bear

Build and Release Engineer
Open Source Applications Foundation (OSAF)
bear at osafoundation.org
http://www.osafoundation.org

bear at code-bear.com
http://code-bear.com

PGP Fingerprint = 9996 719F 973D B11B E111  D770 9331 E822 40B3 CD29


-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
Url : http://lists.osafoundation.org/pipermail/chandler-dev/attachments/20070608/bb42843e/PGP.pgp