[Windmill-dev] State of functest and unit tests

[Mikeal Rogers]

On Jun 26, 2008, at June 26, 20086:48 PM, Adam Christian wrote:

> So the current state of our test coverage is pretty weak, which I am  
> about to start working on.
>
> Today I did a bunch of checkins, and went to run all of the tests,  
> browser teardown isn't working properly in FF or Safari, and about  
> 3/4 of the way through the live_tests in both Safari and Firefox I  
> get a 500 internal server error and everything pretty much stops.

I'll try to debug these, is this happening on all browsers?

>
>
> We need to build up suites of tests based on different chunks of  
> functionality, especially around the wait code because it has been  
> broken and re-broken about 10 times in the last year.

We have tests for wait, and they have failed every time over the last  
year that the wait code broke :)

>
>
> Any ideas, insight? I'd really like to get this to a state where I  
> can just RUN the tests before a checkin and know that everything  
> still works as it did.

So, I don't know why you can't "just RUN" the tests?

I run the tests all the time. They run on safari and firefox only  
because I don't usually fire up the windows VM for IE, but I can run  
them.

Over the last year I'd say that 95% of the time the tests fail because  
of breaks in functionality, and another 5% is the fault of one of the  
tests.

I'd agree that we need to improve our test coverage, but the things  
you're complaining about we do have tests for. The thing about tests,  
is that you have to RUN THEM!

We break trunk all the time, I don't necessarily think this is the  
worst thing in the world. People shouldn't be running trunk unless  
they do development on windmill. If there are fixes in windmill we  
should push a release, it only takes a minute. Whenever we push a  
release the automated tests get run, it's part of the release script.  
If we're worried about breaking trunk then we should run the tests  
before checkin, or at least run a subset of the tests, which is very  
easy to do.

In short, we have tests, you should run them more often. We should  
have more tests, but they will only help if you run them.

-Mikeal