Hands on with the Mac Pro: Testing the limits

My in-depth look at setting up and running a Mac Pro continues. In the first installment, I shared my out-of-the-box experience and ran some tests on some apps. In the second, I talked about my observations of working on the desktop, perform audio and video tasks, and use Boot Camp and Parallels Desktop to turn my Mac Pro into a Windows-running powerhouse.

In this third and final installment, I fire up my favorite games to test frame rates. I also test transfer speeds for FireWire drives and wrap it all up with a little methodology.

Gaming on the Mac Pro

Unlike the gaming section in my mini writeup, there’s no need to restrict what type of games I test on the Mac Pro. Whereas the mini lacks a graphics card of any kind, this Mac Pro is equipped with the built-to-order X1900 XT, ATI’s top-of-the-line Mac video card. Also, since I covered gaming peripherals before, I won’t go down that street again—basically, if they don’t need a driver and work on a PowerPC Mac, then they’ll work for sure. If they do need a driver, check the manufacturer’s site for an Intel-compatible version. This time, I thought it would be interesting to check in on one 3D graphics-based game that runs in Rosetta, as well as the one current game I own that runs on both Windows XP and OS X. I’ll also touch on the other games I looked at before, relative to their performance on the Mac Pro.

Quake 4: This would be the one current Windows XP and OS X game I own. I was interested in seeing how it ran on the Mac Pro under both Windows XP and OS X. After making sure I had the latest update (1.3) on both systems, I then configured them identically. I used a 1440-by-900 screen size, to match the monitor I was using, and had the quality pop-up set to “high quality.” Other than that, I left all the rest of the settings in their stock position—with one important exception. In the Advanced section of Quake 4’s preferences, I made sure that multiple CPU (or core) support was enabled. For the actual test, I made sure no other GUI apps were running, though each OS clearly as an assortment of background tasks running all the time. I then ran a time demo (timedemo in Quake-speak) in each OS, using this console command (press control-option-tilde to drop down the console):

playnettimedemo id_demo001.netdemo

At the conclusion of the demo, drop the console down again and you’ll see the average frames per second for the timedemo. So how did OS X hold up to Windows XP? In my tests, which I repeated three times, it was fairly consistent: on average, OS X scored right around 72fps, while Windows XP cranked out 90fps. So, it seems for this one game at least, the OS X “it’s a better OS” performance tax appears to be a 20 percent framerate hit. That’s a definite ouch , but it’s perhaps not quite so bad in the real world of actual gameplay. Quake 4 caps the refresh rate at 60fps in game play, and during my testing on the earlier levels, I have yet to see it drop below that when playing at the 1440-by-900 resolution.

This difference should drop sometime in the future, too, assuming Quake 4 is patched to support the new multi-threaded OpenGL feature in OS X 10.4.8 on the Mac Pros. I’m not sure when, or if, we’ll see such a patch, though. In the short term, the game is very playable on the Mac Pro, even at somewhat higher resolutions. If you want the ultimate frame rate in Quake 4 on your Mac Pro, though, you’ll want to boot into Windows XP.

Tiger Woods PGA Tour 2005: This game has not yet been updated to run natively on the Intel Macs, so it’s a good test of gaming under Rosetta. When I tested this game on the Mac mini, I found it was almost playable—frame rates were low but acceptable, though it might get a bit too slow on busier screens. On the Mac Pro, by contrast, it’s very playable. Running at 1024-by-768 resolution, I was seeing frame rates anywhere between 20 and 45, with typical averages on holes in the 30s. Rates don’t drop too much when you add some wind and clouds—but don’t turn on the display of rain in the video settings, or you’ll see a dramatic drop in frame rates (though this is true even on my G5).

To demonstrate the playability of the game, I thought I’d redo the video from the report on the mini. So here’s that same hole at the same resolution (1024-by-768 originally), except on the Mac Pro. And since Snapz Pro now runs on the Intel Macs, this capture is of much higher quality.

Click the image to load the 2.2MB movie in a new window:

As you can see, the screen refresh was relatively quick and fluid. Given that TW2005 is one of my favorite quick-play diversions, I was glad to discover that it had acceptable performance, even under Rosetta. I still have my fingers crossed for a Universal update, but even if it never comes to pass, the game is playable.

Halo, Spider-Man 2, and Unreal Tournament: I discussed all three of these games in the original mini writeup in more detail. All three are Universal, and they all work wonderfully well on the Mac Pro. I ran a time demo in Halo, with every setting maxed out (advanced pixel shaders, 9x anti-aliasing, extreme lens flare, high model detail), and scored 43fps at 800-by-600. With the anti-aliasing dropped down to 4x and the lens flare to high, that number shot up to 112fps. In actual play, at 1440-by-900 resolution, the game is amazingly smooth and glitch-free.

X-Plane: X-Plane looks and flies great; I can set very long view distances with high levels of detail without bogging down the simulator. Instead of replicating the screenshots from the mini report, here’s one of a different locale (click for a larger image):

That’s a take-off out of Telluride airport in Colorado, with some low clouds in the mountains. If you’ve ever been to the airport, then you know that this is basically the view off that end of the runway (though in reality, they typically take off in the other direction, wind permitting). Contrast the above image with the fog-shrouded view on the PowerBook from the previous report, and you can really see the payoff from the faster CPUs and video card.

Quake III Arena: Not a whole lot to say about Quake III Arena, other than 718fps. No, that’s not a typo. That was the built-in timedemo result at 1024-by-768 in the Universal Binary version of Quake III. By way of comparison, the mini managed a paltry 90fps. Even my Dual G5, with the ATI X800XT, scores only in the 285fps range. Yes, it’s meaningless for today’s games, but it shows you just how quickly the Mac Pro can dump graphics onto the screen.

Just for fun, I grabbed a Snapz movie of the entire timedemo—it’s only a few seconds long. You can watch either the full-size clip (1024-by-768, 2.4MB) or a smaller version (512-by-384, 1.2MB) if you’re on a slower connection. Note that because Snapz Pro movie capture requires a fair bit of CPU time itself, the frame rate in the above demo is “only” 538 frames per second (and I shot the movie at 15fps, to help minimize the CPU hit). I still think the video gives you a very good sense of the great speed at which the frames are rendered.)

Overall, the Mac Pro makes an excellent gaming machine. However, given its price point, it’s definitely not the most economical gaming machine you can buy. That said, if you’re a Mac user, you enjoy playing games, and your plans call for a new Mac Pro at some point, you won’t be disappointed by its gaming performance. While high-end PCs will still make better gaming platforms (thanks mainly to their ability to use multiple video cards to draw to a single screen), the Mac Pro isn’t a slouch by any stretch.

Three takeaway points

Based on my one-game cross-platform test (Quake4), it seems there’s still a bit of a performance hit for using OS X instead of Windows XP. However, it’s not a terrible hit, and it may be worth it to save the reboot time.

Don’t justify a Mac Pro purchase decision base purely on gaming capabilities; it makes no economic sense. You could buy two 60GB Playstation 3s ($599 each), two Xbox 360s ($399 each), and a couple of Nintendo’s Wiis ($250 each), and still save money over the cost of the Mac Pro. Buy the Mac Pro for all the other things it does very well.

If you do happen to buy a Mac Pro for other reasons, you’ll be more than pleased with its abilities as gaming machine.

Testing the Mac Pro

You can see the results of Macworld Lab’s Mac Pro tests in our full review.

For this report, I didn’t duplicate those tests, but instead, chose to update the tests I ran for my report on the Intel-powered Mac mini.

GeekBench benchmark: In the period of time since I used GeekBench for the mini review, it’s been updated a few times. So I re-ran the benchmark on all my machines: the PowerBook G4 (1.25GHz, 768MB RAM), the Intel mini (1.67GHz Core Duo, 2GB RAM), the MacBook (2.0GHz Core Duo, 2GB RAM), the Dual G5 (2.0GHz G5, 2.5GB RAM), and the new Mac Pro (Dual 2.66GHz Xeon, 2GB RAM).

After all the crunching was done, here’s how everything came out:

There are some interesting tidbits hiding in that table:

If you compare the “Rosetta % of Universal” columns between this test and the mini test back in March, it’s pretty obvious that Rosetta has improved (as we discuss in this Rosetta first look ). To make sure that this wasn’t just due to the Mac Pro, I also re-ran the Rosetta vs. Universal version of GeekBench on the Mac mini, and saw similar results—no longer is the average at or below 50 percent; it’s now at 62 percent on overall score, and many of the tests are at 90 percent or higher.

Although the PowerBook is well-suited to handling e-mail and Web surfing, it’s clearly showing its age in any computational tests. It’s overall score is only slight better than one-third that of its replacement in our household—the MacBook

My Dual PowerMac G5 is also showing its computational age. Although this test is far from a measure of real-world usability, there’s no doubt that the G5 CPUs are being outperformed by the Intel chips—even the lowly mini, with its Core Duo running at 1.67GHz, outscores the G5. Now, in real world usage, the G5 is still a much faster machine, thanks to its faster hard drive, higher memory bandwidth, and high-end graphics card. But the Intel Mac mini puts up a heck of a showing for a $799 machine.

Even when running in Rosetta mode, the Mac Pro is clearly the class of this benchmark suite. In native mode, it’s score is nearly twice that of its nearest competitor—the MacBook.

As I stated, this is definitely not a benchmark for measuring real-world usability—if it were, the G5 would have scored much higher than it did. What it does accurately reflect, however, is the power of the underlying processor(s) in each machine. In that sense, I find the results to be quite accurate. In my use of iMovie, for instance, I found that rendering transitions and effects took basically the same amount of time on the MacBook as it did on the Dual G5.

Cinebench benchmark: The free Cinebench benchmark uses the Cinema 4D engine to test the graphics performance of your Mac. It’s also available for Macs and PCs, so I tested everything I could. Back in March, that included my now-sold Athlon machine, so that’s still in the results table (as the benchmark hasn’t been updated, I felt it fair to leave in). I added my MacBook and the Mac Pro to the results, and with the Mac Pro, I included the results when running in Windows XP via Boot Camp—this to see if there were any platform-specific differences in the benchmark’s performance. When the dust settled, here’s how things came out (click the image for a much larger, easier-to-read version):

When I first published this chart, my old Athlon-powered PC was the champion, handily trouncing the G5 in the OpenGL tests, and basically tying it in the other areas. The Mac Pro put an end to that, and as you can see, by a wide margin. (I have no doubt that there are probably faster PCs out there—but I don’t have one available for testing). The PowerBook, as usual, is solidly in last place, as both the no-graphics-card MacBook and mini beat it out. I was pleasantly surprised to see no significant differences in the Mac Pro’s results regardless of whether it was running Windows XP (via Boot Camp) or Mac OS X—the XP box was about 2.5 percent quicker in the Open GL hardware test, and the Mac Pro was marginally quicker in all the other tests.

What that chart can’t show you is the visual difference between running the test on my Dual G5 and the Mac Pro. Though the G5 isn’t a slowpoke, the Mac Pro is over twice as fast when drawing the Open GL hardware-accelerated scene, and that kind of difference is obvious to the eye. As I’ve said many times before, the Mac Pro with the ATI X 1900XT video card is one fast machine.

Three takeaway points

There’s enough processing power in even a low-end Intel Mac to rival the Dual G5 in many CPU-intensive tasks.

The combination of the Mac Pro’s CPU power and ATI video card make for a potent OpenGL machine.

Unlike the situation with Quake4, there wasn’t a significant difference between Mac OS X and Windows when running the Cinebench benchmark.

Xbench: Although there’s a new version (1.3) of Xbench available, I was having some troubles getting what I felt to be believable figures from the program. One problem is that it doesn’t properly test for accelerated OpenGL graphics on machines (like my MacBook and mini) with a graphics chipset instead of a card. Also, I was getting very strange results on the “User Interface” portion of the test—the mini and the MacBook were both scoring about twice what the G5 was recording. Finally, the score for the Disk portion of the test on the G5 just seemed completely out of line—even testing on a newly-formatted drive, my score was roughly one-fifth of what it should have been for a stock G5.

Based on these odd results, I’ve chosen to hold off on any Xbench results until I can do some more testing with my machines (and hopefully, a new version is released that addresses some of the issues I experienced).

FireWire testing: As a new test this time, I thought I’d compare the time required to copy three sets of data (a single large file, a folder with tons of smaller files, and a small assortment of mid-sized files) to and from both FireWire 400 and FireWire 800 drives. I only tested the Mac Pro and the Dual G5, and I placed the three data sets to be copied onto a relatively clean (recently formatted) partition on each machine, then copied them from there to the FireWire drive. When copying the files back, I copied them back to a single partition on the Mac Pro (not the RAID 1 boot drive). Here’s what I found—the top figure in each cell is the number of seconds to complete the task, and the number below is the transfer rate in megabytes per second:

This is obviously a far-from-complete test, and I didn’t do everything necessary to control all the variables as would be necessary in a real performance evaluation. Still, there are some interesting results to consider:

Whether copying to or from the FireWire 400 or FireWire 800 drives, the Dual G5 had serious issues with my iChat folder—the one with over 10,000 files and folders in it. I tried this test multiple times, from multiple different internal drives, and the results were basically the same. For whatever reason, it seems the Mac Pro doesn’t choke as badly on the huge folder as does the Dual G5. I welcome explanations here, because I really can’t figure out what’s going on. The fact that I saw similar results on both FireWire interfaces rules out any sort of drive-specific cause, too.

When copying to the FireWire drive (400 or 800), there wasn’t a huge amount of difference between the two machines. The Dual G5 was faster in three of the four remaining tests, but the difference was roughly 5 percent, which isn’t all that significant given my hand-timing work.

When copying from the FireWire drive (400 or 800), the Mac Pro was the quicker machine in the four non-huge-folder tests—though again, the differences weren’t all that significant (other than the high-def video clips on the FireWire 800 drive, which took 26 fewer seconds on the Mac Pro).

Overall, FireWire seems to work about as well on the Mac Pro as it does on the Dual G5—at least when copying to or from internal drives. I’ve heard a report that copies between FireWire drives on the various buses are slower on the Mac Pro, and I plan on doing some testing in that area for a future report.

Three takeaway points

The Mac Pro appears to handle large numbers of files in copies to/from FireWire drives much better than does the Dual G5. I can’t say that this is a global issue, but it was certainly repeatable in my testing, regardless of the source and destination disks.

FireWire 800 drives are faster than the first-generation FireWire 400 drives, but not twice as fast. I was seeing improvements of anywhere from 10 percent to 30 percent.

When working with FireWire 800, the Mac Pro proved faster in five of the six test cases.

Rosetta and Intel transition issues

As I look back on what I wrote in the spring in this section of the mini’s report, I still think my general conclusions drawn back then are still true today—just a bit less so (in a good way!) in some regards.

Regarding Rosetta, I wrote:

For most typical users, who spend their time in e-mail, on the web, and working on simple letters and spreadsheets, I don’t think Rosetta is going to be noticed much, if at all. If, on the other hand, you’re a power user who works on massive Excel spreadsheets, huge image-laden Word documents, or monstrous 20MB advertising layout TIFFs in Photoshop, you’re going to want to wait for native versions of your applications before you switch.

That’s definitely still a true statement today, but the speed of the Mac Pro means that even a non-native Photoshop isn’t an excessively slow Photoshop—as long as you’re working with reasonably sized images. For most remaining Rosetta apps, unless they’re doing something special with graphics, you probably won’t even notice that they’re running in Rosetta. Apple realized early on that Rosetta would be the key to a successful Intel transition, and it seems to me like they’re gotten it pretty much right. I’m still astounded that something like Tiger Woods PGA Tour even launches in Rosetta, much less runs at a semi-decent frame rate.

Regarding that Intel transition, back in March I wrote (about the mini):

At the end of the day, the mini is really just another Mac, regardless of what’s powering it. If nobody told you, you wouldn’t know there’s anything different under the hood. The machine is a Mac, just one with a different engine powering it than we’re all used to.

Of course, we’re now completely transitioned to Intel, and some models are already into their second generation. I think anyone who has used an Intel Mac will tell you that it’s still got the soul of a Mac. It still runs OS X. It still works with peripherals as well as it always has. Sure, some things have changed under the hood, but the machines are still Macs. Back in March, I lamented the lack of both Snapz Pro and SideTrack, but now, both are available for the Intel Macs. In March, we had about 1,000 Universal applications to pick from. Now that figure is well over 4,000.

The combination of Apple’s industrial design and Intel’s speedy chips has created some of the fastest, most expandable, and least expensive Macs that we’ve ever seen (more on that in a minute). If I had to grade the Intel transition, I’d probably have to give Apple an A- or a B+, as there’s not much to fault them on. Sure, a native Adobe suite would’ve been nice to have, but beyond that, I feel the transition has gone much better than I expected. For my takeaway points, I’m going to repeat two from the spring, and add one new one.

Three takeaway points

Most typical users won’t find Rosetta to be an intrusive, slowing technology, despite the benchmark results.

Native applications tend to be quite fast, and Java applications are already native by default.

The added CPU and video horsepower on the Mac Pro make it even harder to use “but I’ll have to run things in Rosetta” as an excuse not to upgrade to an Intel Mac.

Conclusion

As I write this, I’ve had the Mac Pro for almost exactly one week. It’s been a busy week, with lots of shuffling between the two machines (they’re on opposite sides of the desk at the moment), and lots of file copying (and recopying) and system updating (and re-updating). Through all of that, I’ve been impressed with the Mac Pro’s quietness and speedy operation. Assuming Macworld bought everything on my machine through the Apple store, it retailed for $3,126 (2.66GHz model with 2GB of RAM, Bluetooth, AirPort, one 250GB drive, and the ATI X1900 XT video card). While that’s not an insignificant amount of money, I would argue that it’s an amazing value on two different fronts.

First, there’s just the raw value of the machine. Compared to my Dual G5, prices have dropped tremendously (as they generally do in computing). I ordered my machine on June 23, 2003, in this configuration:

Dual 2GHz PowerPC G5, 512MB DDR400 SDRAM

160GB Serial ATA – 7200rpm

SuperDrive (DVD-R/CD-RW)

ATI Radeon 9800 Pro (256MB)

Keyboard, mouse, internal modem, and an “accessory kit”

For all of that, I wrote Apple a check for $3,299. Adjusting for inflation, that would be $3,638 in today’s money. So for $500 less, I have a machine with twice as many processors (and those processors are much faster), a larger and faster (SATA 2) hard drive, a much better video card, a faster SuperDrive, room for three more (instead of one more) internal drive and a second optical drive, Bluetooth, AirPort… and no modem. Clearly today’s Mac Pro contains a lot more technology for a lesser cost.

Second, there’s the “two mints in one” value of the Mac Pro. It’s not just that that Mac Pro is an excellent OS X box. It’s also a very capable Windows XP machine. If I had a “real” job, this one machine could easily take the place of two—consider a graphic designer who has to use some corporate Windows XP-only application, for instance. So instead of having two huge boxes sitting on their desk, the designer would have just one, and use either Parallels Desktop or Boot Camp for when they needed access to Windows XP.

So there you have it, my in-depth look at the Mac Pro. And now it’s time to actually finish my data migration, which I’d set aside to complete this report. I can’t wait to make the switch, and move the Mac Pro into its permanent spot on my production line. The Dual G5 will become some sort of home server, though I’m not quite sure in what regard just yet—perhaps that project will be the subject of a future report.

[ Rob Griffiths is a senior editor at Macworld.]