[Dev] Performance Update: Why is Chandler slow on Mac?

[Alec Flett]

Reid Ellis wrote:
>
> On my 1.6GHz PowerBook: 27,778. Then I realized I was using 
> /opt/local/bin/python (v2.4.2, built with GCC 4), so I tried again 
> with /usr/bin/python (v2.3.5, built with GCC 3.3) and got 24,510. Huh. 
> The G5 was using 2.4.2/GCC4.
>
> So these numbers have a fuzz factor of about 3% or so, and are highly 
> impacted by build environment and/or Python version. I am assuming 
> John's numbers are from using GCC 4 and Python 2.4.2.
>
It is important to note that there were some specific performance 
improvements in Python going from 2.3 -> 2.4, and that Chandler uses 
some of the new 2.4 syntax and thus requires Python 2.4. The only real 
differences in 2.4.1/2.4.2 have been security fixes, so there shouldn't 
be any noticeable performance change between those versions. Since 
Python is open-source, there isn't really a notion of the 
factory-installed Python being any "better", so in general its best to 
go off of the python that is built in Chandler.

Alec
> I threw these numbers into Excel and added a "stones/GHz" field:
>
>                      PyStone Benchmarks
>        System            GHz   CPU results stones/GHz
>        1.2Ghz iBook      1.2   G4   18939   15782.5
>        3GHz P4 XP        3.0   P4   39463   13154.3
>        3GHz P4 FC4       3.0   P4   27932    9310.7
>        1.67GHz PowerBook 1.67  G4   27778   16633.5
>        2x2GHz G5         2.0   G5   31446   15723.0
>
> When you factor in the actual speed of the CPU, it looks like the G4 
> CPUs in the PowerBook and iBook fare the best, with the Fedora Core 
> 3GHz P4 the worst. I would expect an AMD CPU to outshine the G4's.
>
> These numbers don't really affect the final conclusion all that much - 
> that Python is CPU limited. And with Apple's switch to Intel, the 
> differences amongst OS X, XP and Linux should fade.
>
> Reid
>
>
> On Mar 8, 2006, at 18:50, John Townsend wrote:
>> Hi John,
>>
>> I actually had access to an Intel iMac today so I ran your python 
>> test on the machine.
>>
>> Intel-based iMac (running 1.83 GHz Intel Dual Core) with 1GB of RAM.
>>
>> The results: 31250 pystones/sec.
>>
>> By comparison, G4 based PowerBook (1.67GHz PowerPC G4) with 1GB of RAM:
>>
>> The results: 22831 pystones/sec.
>>
>> Just thought you might be interested.
>>
>> --> towns
>>
>> On Mar 8, 2006, at 3:09 PM, John Anderson wrote:
>>
>>> During the sprints at PyCon I worked on Performance, focusing on 
>>> trying to understand why Chandler is slower on Mac than other 
>>> platforms. Preliminary results seem to indicate that OSX Macs are 
>>> just overall slower than Windows and Linux PCs. Read on for the 
>>> nitty-gritty details.
>>>
>>> Launching Chandler with a preexisting repository -- the normal user 
>>> experience -- on the 1.2Ghz OSX iBook G4 I had at PyCon took 14 
>>> seconds compared to about 4 seconds on my 3Ghz P3 Shuttle running XP 
>>> or Fedora Core3. Alec's XP Pentium M laptop was not far behind the 4 
>>> second mark.
>>>
>>> The 3.5 times slowdown in launch translated into a significant loss 
>>> of productivity for me. When developing Chandler I usually run in 
>>> WingIDE. On my 3Ghz P4 shuttle, launching Chandler in Wing with an 
>>> existing repository takes 12 seconds and launching while creating a 
>>> new repository takes 36 seconds. Multiplying those times by 3.5 made 
>>> the iBook too slow for development. This probably explains why 
>>> Macintosh developers avoid using Wing or even running Chandler 
>>> whenever possible.
>>>
>>> Next I decided to run some benchmarks to see how C and Python 
>>> performance compared on the different machines. Anyone who has run 
>>> benchmarks on modern multitasking operating systems knows that 
>>> running the same benchmark can sometimes vary as much as a factor of 
>>> two on the same machine for a variety of reasons. Sometimes it's 
>>> because some other task is using the CPU. Sometimes the machine 
>>> saves power by slowing down the processor. I ran all the tests 
>>> several times to try to get representative numbers.
>>>
>>> The first test I ran was Python's pystone, which I ran as follows:
>>>
>>> from test import pystone
>>> pystone.pystones (loops=pystone.LOOPS)
>>>
>>> The iBook did 18,939 pytsones/sec. 3Ghz P4 XP did 39,463. 3Ghz P4 
>>> Fedora Core 4 did 27,932. XP Pentium M was comparable to the 3Ghz P4 
>>> XP result. So on first glance, Python performance seems twice as 
>>> slow on the iBook compared to the XP machines.
>>>
>>> Running the Python "make test" on the iBook vs. XP Pentium M 
>>> produced a similar XP speed advantage of roughly a factor of 2.
>>>
>>> Since Fedora and OSX use the same gcc compiler, yet XP is 1.4 times 
>>> faster than Fedora on the same hardware, it looks like a big part of 
>>> the difference in performance is explained by the Microsoft 
>>> compiler. Not only does the Microsoft compiler produce much better 
>>> code, it runs significantly faster -- Bear mentioned that builds on 
>>> the iBook are about 5 times slower than Windows.
>>>
>>> I have a 10,000 line C program I wrote years ago that I use for 
>>> performance measures on different processors. It's an interpreted 
>>> language, not unlike Python, which I suspect compares to the kind of 
>>> C code that makes up Chandler. The iBook did 141K instructions/sec. 
>>> 3Ghz P4 XP did 1226K/sec. 3Ghz P4 Fedora did  800K/sec. Pentium M XP 
>>> did about 1000K/sec. A 1Ghz PowerBook did about 300K/sec. The first 
>>> NeXT machine did about 30K/sec. So in this test the iBook was almost 
>>> 9 times slower than Intel/XP. The Microsoft compiler advantage, 1.5 
>>> times speed up, was comparable to the pystone tests -- even using 
>>> -O3, the highest level of optimization for the gcc builds.
>>>
>>> Alec pointed out a C benchmark, GeekBench 
>>> (http://www.geekpatrol.ca/geekbench/) which has lots of tests that 
>>> measure the speed of different tasks on a variety of processors. 
>>> Although their tests didn't compare the same processors I tested, 
>>> the most varied CPU test gave the Microsoft/Intel vs OSX/G4 a 2 to 4 
>>> speed advantage. The exciting news, however, is that the iMac Core 
>>> Duo 2.0 was roughly equal to the best Microsoft/Intel machines.
>>>
>>> Here are my conclusions: Although these results don't explain the 
>>> whole story, the XP/Intel Chandler launch advantage of 3.5 seems 
>>> like it could be entirely be explained by the difference in C 
>>> performance. Furthermore, a significant part of this advantage is 
>>> due to the Microsoft compiler. There are more tests that would be 
>>> interesting to do: profile the C code in Chandler to see where time 
>>> is spent during launch and other time consuming tasks. Run tests on 
>>> the Intel Macs. Measure how much IO is involved running Chandler. 
>>> Construct some Berkeley DB benchmarks and measure them on the 
>>> different platforms. Finally, Given Moore's law, the current state 
>>> of Chandler, and its probable lifetime (assuming it's successful), 
>>> performance probably won't be a significant obstacle to adoption.
>>>
>>> John
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "Dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/dev