[Dev] Performance Update: Why is Chandler slow on Mac?

[John Townsend]

Hi John,

I actually had access to an Intel iMac today so I ran your python  
test on the machine.

Intel-based iMac (running 1.83 GHz Intel Dual Core) with 1GB of RAM.

The results: 31250 pystones/sec.

By comparison, G4 based PowerBook (1.67GHz PowerPC G4) with 1GB of RAM:

The results: 22831 pystones/sec.

Just thought you might be interested.

--> towns

On Mar 8, 2006, at 3:09 PM, John Anderson wrote:

> During the sprints at PyCon I worked on Performance, focusing on  
> trying to understand why Chandler is slower on Mac than other  
> platforms. Preliminary results seem to indicate that OSX Macs are  
> just overall slower than Windows and Linux PCs. Read on for the  
> nitty-gritty details.
>
> Launching Chandler with a preexisting repository -- the normal user  
> experience -- on the 1.2Ghz OSX iBook G4 I had at PyCon took 14  
> seconds compared to about 4 seconds on my 3Ghz P3 Shuttle running  
> XP or Fedora Core3. Alec's XP Pentium M laptop was not far behind  
> the 4 second mark.
>
> The 3.5 times slowdown in launch translated into a significant loss  
> of productivity for me. When developing Chandler I usually run in  
> WingIDE. On my 3Ghz P4 shuttle, launching Chandler in Wing with an  
> existing repository takes 12 seconds and launching while creating a  
> new repository takes 36 seconds. Multiplying those times by 3.5  
> made the iBook too slow for development. This probably explains why  
> Macintosh developers avoid using Wing or even running Chandler  
> whenever possible.
>
> Next I decided to run some benchmarks to see how C and Python  
> performance compared on the different machines. Anyone who has run  
> benchmarks on modern multitasking operating systems knows that  
> running the same benchmark can sometimes vary as much as a factor  
> of two on the same machine for a variety of reasons. Sometimes it's  
> because some other task is using the CPU. Sometimes the machine  
> saves power by slowing down the processor. I ran all the tests  
> several times to try to get representative numbers.
>
> The first test I ran was Python's pystone, which I ran as follows:
>
> from test import pystone
> pystone.pystones (loops=pystone.LOOPS)
>
> The iBook did 18,939 pytsones/sec. 3Ghz P4 XP did 39,463. 3Ghz P4  
> Fedora Core 4 did 27,932. XP Pentium M was comparable to the 3Ghz  
> P4 XP result. So on first glance, Python performance seems twice as  
> slow on the iBook compared to the XP machines.
>
> Running the Python "make test" on the iBook vs. XP Pentium M  
> produced a similar XP speed advantage of roughly a factor of 2.
>
> Since Fedora and OSX use the same gcc compiler, yet XP is 1.4 times  
> faster than Fedora on the same hardware, it looks like a big part  
> of the difference in performance is explained by the Microsoft  
> compiler. Not only does the Microsoft compiler produce much better  
> code, it runs significantly faster -- Bear mentioned that builds on  
> the iBook are about 5 times slower than Windows.
>
> I have a 10,000 line C program I wrote years ago that I use for  
> performance measures on different processors. It's an interpreted  
> language, not unlike Python, which I suspect compares to the kind  
> of C code that makes up Chandler. The iBook did 141K instructions/ 
> sec. 3Ghz P4 XP did 1226K/sec. 3Ghz P4 Fedora did  800K/sec.  
> Pentium M XP did about 1000K/sec. A 1Ghz PowerBook did about 300K/ 
> sec. The first NeXT machine did about 30K/sec. So in this test the  
> iBook was almost 9 times slower than Intel/XP. The Microsoft  
> compiler advantage, 1.5 times speed up, was comparable to the  
> pystone tests -- even using -O3, the highest level of optimization  
> for the gcc builds.
>
> Alec pointed out a C benchmark, GeekBench (http://www.geekpatrol.ca/ 
> geekbench/) which has lots of tests that measure the speed of  
> different tasks on a variety of processors. Although their tests  
> didn't compare the same processors I tested, the most varied CPU  
> test gave the Microsoft/Intel vs OSX/G4 a 2 to 4 speed advantage.  
> The exciting news, however, is that the iMac Core Duo 2.0 was  
> roughly equal to the best Microsoft/Intel machines.
>
> Here are my conclusions: Although these results don't explain the  
> whole story, the XP/Intel Chandler launch advantage of 3.5 seems  
> like it could be entirely be explained by the difference in C  
> performance. Furthermore, a significant part of this advantage is  
> due to the Microsoft compiler. There are more tests that would be  
> interesting to do: profile the C code in Chandler to see where time  
> is spent during launch and other time consuming tasks. Run tests on  
> the Intel Macs. Measure how much IO is involved running Chandler.  
> Construct some Berkeley DB benchmarks and measure them on the  
> different platforms. Finally, Given Moore's law, the current state  
> of Chandler, and its probable lifetime (assuming it's successful),  
> performance probably won't be a significant obstacle to adoption.
>
> John
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "Dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/dev