Virtualization: Performance testing

Last year, when I did my comparison of virtualization apps for OS X, many readers told me they wished I’d done more extensive performance testing. So this year, when I decided to compare the latest versions of those apps ( Fusion 3.0.1, Parallels Desktop 5 for Mac build 9308, VirtualBox 3.1.2), I did. Macworld Labs ran (or attempted to run) the WorldBench Windows benchmark test in each of these programs. And I ran my own suite of benchmarks to gauge other aspects of performance.

One thing I found out in doing that testing: Benchmarking virtual machines is hard. There are so many odd variables to account for that it makes it hard to get reliable results. For example, some tests base their measurements on a system’s internal clock. But sometimes the operating system you’re virtualizing manages that clock poorly. In one of my tests, the score for one virtual machine was five times higher than those of the other two; yet, it had taken longer in real-world time to complete the test.

That’s why I hand-timed all the tests I ran. To minimize the effects of human error (a slow thumb, for example), I rounded all times to the nearest full second, and ran each test at least twice; if the results differed measurably between those two runs, I’d run them twice again, to get a more representative score.

I tested each virtualization app with four variations of Windows: XP Pro (two CPUs) and Windows 7 (one CPU, two CPUs 32-bit, two CPUs 64-bit). Even with that relatively small sample, I still ended up running 90 benchmark tests on each of three virtual machine applications—270 total tests—at least twice.

Except for the testing done at Macworld Labs, all of those benchmarks were run on a 2.66GHz Mac Pro with 8GB of RAM and a fresh install of Mac OS X 10.6.2; I ran OS X without any third-party startup items or background apps, and each was configured to with 1GB of RAM per guest operating system. Newer desktops should match or exceed these speeds; some (I’m looking at you, Core i7 iMac) should greatly exceed them.

I would have liked to run these tests on more versions of OS X and on more hardware configurations. But it was simply impractical to do so; the testing was time-consuming enough as it was.

Testing results

It’s amazing how much things can change in just a year. Last year, I described slow and jerky video playback, cautioned readers against using anything other than Windows XP Pro, and issued numerous scary caveats.

While they still have their areas of weakness, these programs have improved remarkably in one year. Last year, none of them could really handle a 720p high definition Windows video. This year, I tested with 1080p Windows video and (for the most part) was quite pleased with the results. I used Windows 7 with Aero effects enabled for video playback and gaming tests, just to further stress the systems; again, they did well.

So, without further ado, here are the results of that testing, followed by a brief overview of what each test entailed and some comments on the results. (If you wish, you can download the original testing spreadsheet for a detailed look at the results.)

WorldBench score: We ran the PCWorld WorldBench benchmark in all three virtualization programs, using Windows a 64-bit Windows 7 Professional virtual machine set to use 2GB of RAM and two CPU cores. The virtual machines were running on a 20” 2.66GHz iMac running 10.6.2 with 4GB of RAM. The WorldBench tests were run using the latest Parallels build (9310), but my own tests were done using build 9308, due to some issues I had in benchmarking the newest version (see the behind the scenes article for more on those issues).

Cinebench: The total time required to run the OpenGL test and the Render tests in Cinebench. Although the results seem to show a one-CPU Windows 7 box outperforming all the rest, that’s because the one-CPU box can only run two tests (OpenGL and one-CPU Render). The two-CPU machines ran those two tests, plus the two-CPU render. You can really see the impact OpenGL acceleration has by looking at the results for Fusion running Windows 7 and VirtualBox running 64-bit Windows 7, which both lack it—they took five to seven minutes longer than the competition to finish the test.

PerformanceTest 7: The PassMark PerformanceTest measures CPU, 2D and 3D graphics, memory, and disk performance. The results in this section include all tests except for the 3D tests; VirtualBox was unable to complete those, so they’re split into their own section.

PerformanceTest 7 3D: The time required for the 3D tests in PerformanceTest 7. Only Fusion and Parallels were able to complete these tests; as you can see, their times were similar in each OS.

Copy via shared folders: I created a 2.5GB folder containing three large Windows executables, along with 29 photographs. I then timed how long it took to copy the folder to and from Windows using each virtualization program’s shared folders feature. (Open the shared folder in Windows, point to the source folder on the Mac.) What stands out here is the speed of Parallels compared to the other programs—in some cases, it’s over twice as fast as the others.

Copy via drag-and-drop: Using the same set of test files as above, I copied them via drag-and-drop to and from the Mac, with a window open in the Mac OS and in the Windows OS. VirtualBox doesn’t support this feature, so I tested Parallels and Fusion only. Again, Parallels was quickest—but not by the same margin as it was when copying between shared folders. Fusion performed similarly regardless of the way I did the copy.

Unzip large archive: How long it took to unzip a 543MB game demo that contained a total of eight files (one large, and seven small). With the exception of Windows XP Pro, Parallels was again notably faster than the competition.

Boot/sleep/wake/shut down: The times in this category are the total time required to: suspend and wake the virtual machine with nothing running; suspend and wake the virtual machine with Microsoft Excel and Word running and with documents open in both; and shut down the virtual machine with nothing running. Once again, Parallels was fastest, but it can’t suspend and resume 3D applications. (See the Parallels review for more detail.)

If there’s one obvious conclusion from the performance testing, it’s that Parallels 5 is the fastest virtualization solution, at least in this set of tests. It won or tied every single category except for one, and won many by a wide margin. These differences aren’t just measurable, they’re also noticeable in regular use: Regardless of the OS I was running, Parallels 5 felt the fastest at most typical tasks.

One interesting note: You can see the advantages of Windows 7 in 64-bit mode in CPU-intensive tests such as Cinebench and the PerformanceTest 7 non-3D tests. But that advantage vanishes when doing things like copying files, expanding zip archives, or changing the state of the virtual machine. If you’ve got a particular app that’s CPU intensive and runs in 64-bit Windows, you’ll probably see a performance boost by using Windows 7. Otherwise, using Windows 7 in 64-bit form probably won’t noticeably improve your day-to-day Windows work.