The Ratings Game

When you bill yourself as “The Mac Product Experts,” you’d best be prepared to prove it—every month, in our case. Or, to paraphrase the mission statement of occasional photojournalist Peter Parker, with great claims comes great responsibility. When a new product arrives for our review, we’ve got to make sure that our testing dots every i, crosses every t, and double-checks every feature.

But our readers don’t just expect accuracy—they also want speed, especially in the Internet era. After all, in an age where instant information is just a click away, even the most thorough product review isn’t very valuable if you have to wait around forever for it to appear.

That’s the balancing act we have to perform every month—we have to be fast, but we also have to be right. And we faced that conflict head-on this month, when it came time to evaluate Apple’s fourth-generation iPods (see “Rock and Scroll,” page 58).

What’s the Buzz

The new iPods arrived a little less than a month before this issue went to press. So I spent a long weekend putting a 40GB model through its paces to turn around a review that could appear both in print and online. But just as we were putting the finishing touches on our articles, reports began circulating on the Internet that some new iPods gave off a strange buzzing noise through the headphone port—especially when their hard drives activated to load more music into the iPod’s memory.

Most of the iPods we bought for testing sounded fine. But Senior Editor Jonathan Seff did hear the buzzing sounds from his 40GB iPod, through his headphones. So what were we to do? Soldier on with our mostly positive review, or write a bad review based on a quirk that seemed to exist only on some iPods?

We decided to report the issues some people were having with their iPods, but not to change the overall tone of our review. Every mass-produced product is going to have its share of lemons; if the share is relatively high, that’s a fact we can’t ignore (which happened when we reviewed the original 15-inch aluminum PowerBook G4 in December 2003). But if we were to give every product a terrible review because a few of the models that rolled off the assembly line were broken, we would wind up giving every product a terrible review.

Reliable Sources

In our iPod review, we concluded that if you get a normal fourth-generation iPod—and all but one of the ones we examined fit this category—you’re going to have a good experience. (As for Jonathan Seff, he went, incognito, to an Apple Store with his buzzing iPod and reported the problem. An Apple Genius took a listen, agreed that there was an issue, and immediately offered him a replacement. You can read about Jonathan’s iPod odyssey at find.macworld.com/0055.)

Sadly, it’s impossible for us to get a clear idea of how reliable a product will be in the long term. So if we see major flaws in a large enough sample, as we did with the PowerBook G4s, we’ll write about it and rate the products accordingly. If we don’t, we’ll assume that the product we tested is typical of most of its kind.

But Macworld is committed to finding a way to better track the reliability of Mac hardware products—not only those from Apple, but also those from the companies that make the printers, cameras, and other hardware you buy. It’s a tough undertaking, and I can’t promise that we’re going to figure out how to do it overnight. But filling in that missing piece of the product-review puzzle is something we’ve got to do if we want to be know as the Mac Product Experts.

1 2 Page 1
Page 1 of 2
  
Shop Tech Products at Amazon