[Dev] Performance Update: Why is Chandler slow on Mac?

[Reid Ellis]

More numbers. On my dual 2GHz G5 I got 31,446. I then ran it again in  
python without quitting and got 28,736. Again: 29069.

On my 1.6GHz PowerBook: 27,778. Then I realized I was using /opt/ 
local/bin/python (v2.4.2, built with GCC 4), so I tried again with / 
usr/bin/python (v2.3.5, built with GCC 3.3) and got 24,510. Huh. The  
G5 was using 2.4.2/GCC4.

So these numbers have a fuzz factor of about 3% or so, and are highly  
impacted by build environment and/or Python version. I am assuming  
John's numbers are from using GCC 4 and Python 2.4.2.

I threw these numbers into Excel and added a "stones/GHz" field:

                      PyStone Benchmarks
        System            GHz   CPU results stones/GHz
        1.2Ghz iBook      1.2   G4   18939   15782.5
        3GHz P4 XP        3.0   P4   39463   13154.3
        3GHz P4 FC4       3.0   P4   27932    9310.7
        1.67GHz PowerBook 1.67  G4   27778   16633.5
        2x2GHz G5         2.0   G5   31446   15723.0

When you factor in the actual speed of the CPU, it looks like the G4  
CPUs in the PowerBook and iBook fare the best, with the Fedora Core  
3GHz P4 the worst. I would expect an AMD CPU to outshine the G4's.

These numbers don't really affect the final conclusion all that much  
- that Python is CPU limited. And with Apple's switch to Intel, the  
differences amongst OS X, XP and Linux should fade.

Reid


On Mar 8, 2006, at 18:50, John Townsend wrote:
> Hi John,
>
> I actually had access to an Intel iMac today so I ran your python  
> test on the machine.
>
> Intel-based iMac (running 1.83 GHz Intel Dual Core) with 1GB of RAM.
>
> The results: 31250 pystones/sec.
>
> By comparison, G4 based PowerBook (1.67GHz PowerPC G4) with 1GB of  
> RAM:
>
> The results: 22831 pystones/sec.
>
> Just thought you might be interested.
>
> --> towns
>
> On Mar 8, 2006, at 3:09 PM, John Anderson wrote:
>
>> During the sprints at PyCon I worked on Performance, focusing on  
>> trying to understand why Chandler is slower on Mac than other  
>> platforms. Preliminary results seem to indicate that OSX Macs are  
>> just overall slower than Windows and Linux PCs. Read on for the  
>> nitty-gritty details.
>>
>> Launching Chandler with a preexisting repository -- the normal  
>> user experience -- on the 1.2Ghz OSX iBook G4 I had at PyCon took  
>> 14 seconds compared to about 4 seconds on my 3Ghz P3 Shuttle  
>> running XP or Fedora Core3. Alec's XP Pentium M laptop was not far  
>> behind the 4 second mark.
>>
>> The 3.5 times slowdown in launch translated into a significant  
>> loss of productivity for me. When developing Chandler I usually  
>> run in WingIDE. On my 3Ghz P4 shuttle, launching Chandler in Wing  
>> with an existing repository takes 12 seconds and launching while  
>> creating a new repository takes 36 seconds. Multiplying those  
>> times by 3.5 made the iBook too slow for development. This  
>> probably explains why Macintosh developers avoid using Wing or  
>> even running Chandler whenever possible.
>>
>> Next I decided to run some benchmarks to see how C and Python  
>> performance compared on the different machines. Anyone who has run  
>> benchmarks on modern multitasking operating systems knows that  
>> running the same benchmark can sometimes vary as much as a factor  
>> of two on the same machine for a variety of reasons. Sometimes  
>> it's because some other task is using the CPU. Sometimes the  
>> machine saves power by slowing down the processor. I ran all the  
>> tests several times to try to get representative numbers.
>>
>> The first test I ran was Python's pystone, which I ran as follows:
>>
>> from test import pystone
>> pystone.pystones (loops=pystone.LOOPS)
>>
>> The iBook did 18,939 pytsones/sec. 3Ghz P4 XP did 39,463. 3Ghz P4  
>> Fedora Core 4 did 27,932. XP Pentium M was comparable to the 3Ghz  
>> P4 XP result. So on first glance, Python performance seems twice  
>> as slow on the iBook compared to the XP machines.
>>
>> Running the Python "make test" on the iBook vs. XP Pentium M  
>> produced a similar XP speed advantage of roughly a factor of 2.
>>
>> Since Fedora and OSX use the same gcc compiler, yet XP is 1.4  
>> times faster than Fedora on the same hardware, it looks like a big  
>> part of the difference in performance is explained by the  
>> Microsoft compiler. Not only does the Microsoft compiler produce  
>> much better code, it runs significantly faster -- Bear mentioned  
>> that builds on the iBook are about 5 times slower than Windows.
>>
>> I have a 10,000 line C program I wrote years ago that I use for  
>> performance measures on different processors. It's an interpreted  
>> language, not unlike Python, which I suspect compares to the kind  
>> of C code that makes up Chandler. The iBook did 141K instructions/ 
>> sec. 3Ghz P4 XP did 1226K/sec. 3Ghz P4 Fedora did  800K/sec.  
>> Pentium M XP did about 1000K/sec. A 1Ghz PowerBook did about 300K/ 
>> sec. The first NeXT machine did about 30K/sec. So in this test the  
>> iBook was almost 9 times slower than Intel/XP. The Microsoft  
>> compiler advantage, 1.5 times speed up, was comparable to the  
>> pystone tests -- even using -O3, the highest level of optimization  
>> for the gcc builds.
>>
>> Alec pointed out a C benchmark, GeekBench (http:// 
>> www.geekpatrol.ca/geekbench/) which has lots of tests that measure  
>> the speed of different tasks on a variety of processors. Although  
>> their tests didn't compare the same processors I tested, the most  
>> varied CPU test gave the Microsoft/Intel vs OSX/G4 a 2 to 4 speed  
>> advantage. The exciting news, however, is that the iMac Core Duo  
>> 2.0 was roughly equal to the best Microsoft/Intel machines.
>>
>> Here are my conclusions: Although these results don't explain the  
>> whole story, the XP/Intel Chandler launch advantage of 3.5 seems  
>> like it could be entirely be explained by the difference in C  
>> performance. Furthermore, a significant part of this advantage is  
>> due to the Microsoft compiler. There are more tests that would be  
>> interesting to do: profile the C code in Chandler to see where  
>> time is spent during launch and other time consuming tasks. Run  
>> tests on the Intel Macs. Measure how much IO is involved running  
>> Chandler. Construct some Berkeley DB benchmarks and measure them  
>> on the different platforms. Finally, Given Moore's law, the  
>> current state of Chandler, and its probable lifetime (assuming  
>> it's successful), performance probably won't be a significant  
>> obstacle to adoption.
>>
>> John