[Cosmo-dev] Re: [Scooby-dev] Issue with same tools in multiple
mikeal at osafoundation.org
Wed Jun 7 20:50:14 PDT 2006
On Jun 7, 2006, at 8:02 PM, John Townsend wrote:
> I agree with Bear on this. I think its premature to promote
> HTTPTest to a separate project and since you have a new framework
> coming, I would wait until that new work is available to establish
> it as a separate project.
I apologize for having very little documentation for HTTPTest (I'll
be writing it next week) so I'll try to explain this a bit better.
HTTPTest was written after I wrote the original proposal for the new
framework so it's written with the that framework in mind. I looked
at other test tools before I went off and wrote my own and found that
the simpler things I HAD to have weren't in them so I decided to
quickly write my own.
HTTPTest grew fast and I added a lot of features in a short amount of
time and had to stabilize it so that we could depend on the output.
HTTPTest inherits from another tool I wrote as a micro test framework
called TestObject. TestObject is simple but it is still pretty
feature rich with support for; multiple debugging levels, multiple
output masking levels, and support for a threaded stress testing
system. Even after the new framework is done I intend to keep
TestObject functional and HTTPTest will continue to support it and
the new framework.
I think I need to reiterate my main point. I don't think that we
should be certifying releases with, or bundling releases with, tools
that are not up to release quality. The main problem with homegrown
test tools is that people tend to not trust the integrity of the
testing, and sometimes look at the test scripts and tools as the
problem before believing it may be a bug in the product. The same
thing would occur with unittests if we even used an unstable unittest
Previously it was unfeasible to have the requirement that releases be
tested with release tools developed at OSAF. It's now feasible for
cosmo and scooby and I would like to move in that direction. I don't
view the issues associated with it as a burden because I think it's
the _right way_ to do things.
> One question: Where is the design proposal for this new Testing
> Framework? I don't recall seeing a design proposal on the mailing
> lists for it.
The original spec is up here (http://wiki.osafoundation.org/bin/view/
Projects/OpenAutomationFramework). Note that it is very chandler
focused because it pre-dates the existence of our cosmo and scooby
test tools and our new move towards the eco-system.
What I need to update in this spec is the major requirement that the
framework support distributed testing, which I will update as soon as
I go back to working on the framework. This framework is very
ambitious and immediate needs have superseded a lot of it's
development, but soon I'll start increasing it's priority.
> --> towns
> On Jun 7, 2006, at 7:13 PM, Mike Taylor wrote:
>> I can see the issue you are talking about but I feel another issue
>> will happen if we move the testing tool to a new location. We
>> would have to treat it like an external project and bring into
>> play all that would entails:
>> - project space (svn, docs, etc)
>> - release cycle
>> - testing across all products for each release
>> - updating Cosmo and Scooby for each new release
>> Currently I don't know if HTTPTest is ready for that and I wonder
>> if the time spent doing the above equals the time maintaining two
>> copies (knowing that it will be merged later).
>> Part of my concern is that having it as a standalone project will
>> force Cosmo's HTTPTest to worry about Scooby's HTTPTest
>> requirements and vice-versa and that may be premature.
>> Once the OSAF Eco-system Testing Tool has been defined and setup
>> as it's own project then I think we can take the two HTTPTest
>> parts and the Chandler test framework part and start figuring out
>> how to merge them.
>> On Jun 7, 2006, at 9:51 PM, Mikeal Rogers wrote:
>>> So we have a good problem, well good that we're far enough along
>>> that it is a problem.
>>> As we're nearing end to end testing across the products we've
>>> quickly started to use the same tools and frameworks in multiple
>>> products. Since traditionally our process has been to keep
>>> functional testing tools in each products svn repository we've
>>> encountered some technical and process issues.
>>> 1.) People working on a given tool or framework may have
>>> different commit privileges in one product than another.
>>> 2.) If a tool or framework is used in two products then we have
>>> to do two separate commits and maintain all the same files in two
>>> Our current issue is that a tool I originally wrote for cosmo,
>>> HTTPTest, is now being used by adam to develop a tool for scooby
>>> testing, JSONTest. Since it lives in the cosmo repository he
>>> can't commit changes to it and I can't easily merge his changes
>>> in while I'm working on the same files to extend testing for cosmo.
>>> Going forward we will be writing a very large framework that will
>>> be used by nearly every tool OSAF QA uses to test all products so
>>> this issue needs to be solved in a broader context than just our
>>> current problem with HTTPTest.
>>> I'm open to any and all ideas for solving this problem.
>>> scooby-dev mailing list
>>> scooby-dev at lists.osafoundation.org
>> Build and Release Engineer
>> Open Source Applications Foundation (OSAF)
>> bear at osafoundation.org
>> bear at code-bear.com
>> PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29
>> scooby-dev mailing list
>> scooby-dev at lists.osafoundation.org
> scooby-dev mailing list
> scooby-dev at lists.osafoundation.org
More information about the cosmo-dev