[Dev] Performance Update: Why is Chandler slow on Mac?
john at osafoundation.org
Thu Mar 9 08:12:32 PST 2006
Reid Ellis wrote:
> More numbers. On my dual 2GHz G5 I got 31,446. I then ran it again in
> python without quitting and got 28,736. Again: 29069.
> On my 1.6GHz PowerBook: 27,778. Then I realized I was using
> /opt/local/bin/python (v2.4.2, built with GCC 4), so I tried again
> with /usr/bin/python (v2.3.5, built with GCC 3.3) and got 24,510. Huh.
> The G5 was using 2.4.2/GCC4.
> So these numbers have a fuzz factor of about 3% or so, and are highly
> impacted by build environment and/or Python version. I am assuming
> John's numbers are from using GCC 4 and Python 2.4.2.
I used Chandler's latest release build of Python for all my tests.
> I threw these numbers into Excel and added a "stones/GHz" field:
> PyStone Benchmarks
> System GHz CPU results stones/GHz
> 1.2Ghz iBook 1.2 G4 18939 15782.5
> 3GHz P4 XP 3.0 P4 39463 13154.3
> 3GHz P4 FC4 3.0 P4 27932 9310.7
> 1.67GHz PowerBook 1.67 G4 27778 16633.5
> 2x2GHz G5 2.0 G5 31446 15723.0
> When you factor in the actual speed of the CPU, it looks like the G4
> CPUs in the PowerBook and iBook fare the best, with the Fedora Core
> 3GHz P4 the worst. I would expect an AMD CPU to outshine the G4's.
> These numbers don't really affect the final conclusion all that much -
> that Python is CPU limited. And with Apple's switch to Intel, the
> differences amongst OS X, XP and Linux should fade.
> On Mar 8, 2006, at 18:50, John Townsend wrote:
>> Hi John,
>> I actually had access to an Intel iMac today so I ran your python
>> test on the machine.
>> Intel-based iMac (running 1.83 GHz Intel Dual Core) with 1GB of RAM.
>> The results: 31250 pystones/sec.
>> By comparison, G4 based PowerBook (1.67GHz PowerPC G4) with 1GB of RAM:
>> The results: 22831 pystones/sec.
>> Just thought you might be interested.
>> --> towns
>> On Mar 8, 2006, at 3:09 PM, John Anderson wrote:
>>> During the sprints at PyCon I worked on Performance, focusing on
>>> trying to understand why Chandler is slower on Mac than other
>>> platforms. Preliminary results seem to indicate that OSX Macs are
>>> just overall slower than Windows and Linux PCs. Read on for the
>>> nitty-gritty details.
>>> Launching Chandler with a preexisting repository -- the normal user
>>> experience -- on the 1.2Ghz OSX iBook G4 I had at PyCon took 14
>>> seconds compared to about 4 seconds on my 3Ghz P3 Shuttle running XP
>>> or Fedora Core3. Alec's XP Pentium M laptop was not far behind the 4
>>> second mark.
>>> The 3.5 times slowdown in launch translated into a significant loss
>>> of productivity for me. When developing Chandler I usually run in
>>> WingIDE. On my 3Ghz P4 shuttle, launching Chandler in Wing with an
>>> existing repository takes 12 seconds and launching while creating a
>>> new repository takes 36 seconds. Multiplying those times by 3.5 made
>>> the iBook too slow for development. This probably explains why
>>> Macintosh developers avoid using Wing or even running Chandler
>>> whenever possible.
>>> Next I decided to run some benchmarks to see how C and Python
>>> performance compared on the different machines. Anyone who has run
>>> benchmarks on modern multitasking operating systems knows that
>>> running the same benchmark can sometimes vary as much as a factor of
>>> two on the same machine for a variety of reasons. Sometimes it's
>>> because some other task is using the CPU. Sometimes the machine
>>> saves power by slowing down the processor. I ran all the tests
>>> several times to try to get representative numbers.
>>> The first test I ran was Python's pystone, which I ran as follows:
>>> from test import pystone
>>> pystone.pystones (loops=pystone.LOOPS)
>>> The iBook did 18,939 pytsones/sec. 3Ghz P4 XP did 39,463. 3Ghz P4
>>> Fedora Core 4 did 27,932. XP Pentium M was comparable to the 3Ghz P4
>>> XP result. So on first glance, Python performance seems twice as
>>> slow on the iBook compared to the XP machines.
>>> Running the Python "make test" on the iBook vs. XP Pentium M
>>> produced a similar XP speed advantage of roughly a factor of 2.
>>> Since Fedora and OSX use the same gcc compiler, yet XP is 1.4 times
>>> faster than Fedora on the same hardware, it looks like a big part of
>>> the difference in performance is explained by the Microsoft
>>> compiler. Not only does the Microsoft compiler produce much better
>>> code, it runs significantly faster -- Bear mentioned that builds on
>>> the iBook are about 5 times slower than Windows.
>>> I have a 10,000 line C program I wrote years ago that I use for
>>> performance measures on different processors. It's an interpreted
>>> language, not unlike Python, which I suspect compares to the kind of
>>> C code that makes up Chandler. The iBook did 141K instructions/sec.
>>> 3Ghz P4 XP did 1226K/sec. 3Ghz P4 Fedora did 800K/sec. Pentium M XP
>>> did about 1000K/sec. A 1Ghz PowerBook did about 300K/sec. The first
>>> NeXT machine did about 30K/sec. So in this test the iBook was almost
>>> 9 times slower than Intel/XP. The Microsoft compiler advantage, 1.5
>>> times speed up, was comparable to the pystone tests -- even using
>>> -O3, the highest level of optimization for the gcc builds.
>>> Alec pointed out a C benchmark, GeekBench
>>> (http://www.geekpatrol.ca/geekbench/) which has lots of tests that
>>> measure the speed of different tasks on a variety of processors.
>>> Although their tests didn't compare the same processors I tested,
>>> the most varied CPU test gave the Microsoft/Intel vs OSX/G4 a 2 to 4
>>> speed advantage. The exciting news, however, is that the iMac Core
>>> Duo 2.0 was roughly equal to the best Microsoft/Intel machines.
>>> Here are my conclusions: Although these results don't explain the
>>> whole story, the XP/Intel Chandler launch advantage of 3.5 seems
>>> like it could be entirely be explained by the difference in C
>>> performance. Furthermore, a significant part of this advantage is
>>> due to the Microsoft compiler. There are more tests that would be
>>> interesting to do: profile the C code in Chandler to see where time
>>> is spent during launch and other time consuming tasks. Run tests on
>>> the Intel Macs. Measure how much IO is involved running Chandler.
>>> Construct some Berkeley DB benchmarks and measure them on the
>>> different platforms. Finally, Given Moore's law, the current state
>>> of Chandler, and its probable lifetime (assuming it's successful),
>>> performance probably won't be a significant obstacle to adoption.
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> Open Source Applications Foundation "Dev" mailing list
More information about the chandler-dev