[Chandler-dev] Proposal to deal with network errors in tests

[Andi Vajda]

On Wed, 13 Sep 2006, Heikki Toivonen wrote:

> We have a handful of tests that try real connections over the web,
> against services we have no control over. Occasionally these fail.
>
> We have modified some tests to simply ignore some network errors. This
> is of course suboptimal, because it makes it hard to find out if there
> are real network coding errors in our code.
>
> We run the tests in several different situations (tbox runs the full
> suite, individuals run suite and tests individually), so it seems
> whatever the solution it must be contained in each individual test.
>
> That leads me to the actual proposal. Wrap each network call in a block
> that tries the call a few times (try once, sleep for a minute, try
> again, sleep 2 minutes, and repeating until we still fail after 8
> minutes of sleep). I think Twisted actually provides something like this
> for tests that are based on twisted. If none of the attempts succeeds,
> report a failure.
>
> I believe this would get rid of most false errors. There have still been
> cases when flickr etc. have been unavailable for several hours, but I
> don't think we can do anything about that.
>
> Of course, the best solution would be to not make any network calls. If
> you have a test that does go over the network, see if you could change
> the test so that it does not need to do that.

+1 as long as you provide yet another command line flag to disable all such 
tests that access the network. This is important when no network is available 
or really slow. For example, running the flickr tests over dialup takes about 
10 minutes.

Andi..