[Chandler-dev] Chandler with Python 2.4 vs 2.5 perf differences

[Grant Baillie]

On 4 Dec, 2006, at 12:55, Heikki Toivonen wrote:

> Now that Chandler is runnable with python 2.5 on the branch I did some
> performance comparisons between trunk and that branch. I was expecting
> overall improvement of around 10%, but unfortunately it does not  
> appear
> that way.
>
> The first column shows the test, second column the time (in seconds)
> with Python 2.4, the third the time with python 2.5, and the last the
> difference between the two. A negative value in the last column  
> means we
> got that much faster with 2.5.
>
> I run the tests just once, but I had shut down most applications and I
> wasn't doing anything else with the machine, and this being dual  
> core I
> would expect the normal variation between test results to be under 2%.
> In other words, any difference less than +/-2% might not be a real
> indication of change.

I'm somewhat skeptical of this 2% figure. Really, the only way to  
measure "normal variation" is to run tests of the same code base  
repeatedly, under as close to the same conditions as possible, and  
look at the standard deviation of the mean.

When I do this with the performance tests, even with around 10 runs,  
I usually end up with 5-10% variations in several of the test  
results. (Like you, I was doing nothing else on the machine, and had  
shut down all extraneous applications). What often happens is that  
once in a while, a test runs anomalously slowly, probably due to some  
random task being undertaken by the OS. This is perhaps not  
surprising, but I wanted to point out that single measurements aren't  
very trustworthy.

--Grant

> The tests were run on Ubuntu Linux (Dapper Drake).
>
> Test                             2.4     2.5       %
> =======================================================
> #1 Startup                       3.98    3.96    -0.50%
> #2 New event (menu)              0.19    0.2      4.83%
> #3 New event (double click)      0.24    0.22    -8.94%
> #4 New calendar                  0.17    0.16    -5.47%
> #5 Import 3k calendar           37.08   37.52     1.18%
> #6 Startup with 3k               4.5     4.61     2.44%
> #7 New event (menu) with 3k      0.5     0.51     2.15%
> #8 New event (double click) 3k   0.35    0.37     3.97%
> #9 New calendar with 3k          0.68    0.64    -5.47%
> Switch Views                     0.23    0.25     8.01%
> Stamp                            0.19    0.21    11.30%
> Switch views with 3k             0.89    0.89     0.61%
> Stamp with 3k                    0.27    0.28     2.31%
> Scroll calendar with 3k          0.02    0.02     6.66%
> Scroll table with 3k             0.26    0.26     1.27%
> Jump calendar by week with 3k    0.29    0.29     0.28%
> Overlay calendar with 3k         1.55    1.55    -0.10%
> Switch calendar with 3k          1.27    1.25    -0.98%
> Sharing                          7.79    9.31    19.57%
>
> -- 
>   Heikki Toivonen
>
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "chandler-dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/chandler-dev