Sciencamatiffic: Bad iPhone surveys

Bad methodologies lead to meaningless results.


Today's Best Tech Deals

Picked by Macworld's Editors

Top Deals On Great Products

Picked by Techconnect's Editors

Remember iPhones? Seemed like everyone used to have one. But now? Nope.

According to TNW, “Tons of iPhone users are joining the dark side by switching to Samsung.” (Tip o’ the antlers to Alan.)

It seems Apple fans are finally shaking off the iPhone mania…

Yes, after… [pretends to check calendar]… 12 years, the fad is finally dying. No one is buying iPhones anymore.

…compared to last year, fewer people are trading in their old iPhones for the latest models.

Now, remember that no one has weighed these people who are ditching the iPhone so it is possible that, combined, they do weigh “tons.” It is also unknown if these are metric or imperial tons.

And, if you look at the methodology of the survey this claim is based on, it is further unknown if these people actually exist.

iPhone retention this year is down 15.2 percent in contrast to March 2018, according to data CNET obtained from phone-swap service BankMyCell.

The Macalope simply adores the use of “obtained.” Just as we do not know the total combined weight of these former iPhone users, we also do not know how many Bothans died to get TNW this information. Only Mon Mothma knows and she ain’t tellin’.

This information certainly didn’t come from an email blast to any and every tech web outlet that BankMyCell thought would publish it. This is top secret information and not in any way designed to get BankMyCell’s name into the press.

Weird thing, though: six paragraphs in there is a late-breaking newsflash: this is probably all garbage.

Here’s an important disclaimer though: the survey is based on a pretty limited data sample…

Oh. Oh! Ohhhhhhhhhh.


… – only 38,000 people. Considering this is only a tiny fraction of all iPhone users, you ought to take the results with a grain of salt.

That’s certainly not the smallest sample set The Macalope’s seen people try to fashion Close Encounters-style into Mt. Apple Doom (no, you’re mixing metaphors). But more than the size of the sample, there are other problems with these results.

First, the population comes only from “phone swaps BankMyCell facilitated in the period since October 2018.” Let’s just posit that customers of a service The Macalope had never heard of before are not necessarily the same as those who frequent carrier stores or Apple Stores.

But that’s not the biggest problem. The biggest problem is that in making its claim that iPhone loyalty dropped 15 percent, BankMyCell isn’t comparing its results this year to its results last year. It’s comparing its results this year to Consumer Intelligence Research Partners (CIRP)’s results last year. The company even admits it has no idea what the demographic makeup of CIRP’s study was.

…our online audience 62.4% millennials with a 60/40 gender split in favor of females, which may differ from CIRP; however, they did not mention demographics in their release.

Somewhere Edward Tufte suddenly felt intensely angry, although he did not know why.

You can’t compare results for different demographies and declare them meaningful. You’re not controlling for anything. “We surveyed people outside Faneuil Hall about the New York Yankees and compared them to a survey done last year outside Grand Central Station and, well, people have really soured on the Yankees!” The only constant here is the gullibility (or culpability) of the technology press. If BankMyCell were interested in meaningful results, it would have compared its own results over two years if it couldn’t get CIRP’s demographic breakdown. But it’s clearly not. Which leads The Macalope to conclude that the point of this exercise was less science than advertising.

Macworld’s Michael Simon also takes the report credulously, saying:

You can quibble with Bank My Cell’s methodology all you want, but the numbers are significant.

That’s not how statistics work. If the methodology is crap, then you can’t trust the numbers. It’s like playing pin the tail on the donkey in a room full of coked-up baboons. Apple still may have a loyalty problem and, to Simon’s overall point, maybe it should sell a mid-tier phone, but this survey is like throwing an astrological chart into an argument for why humans should colonize Mars.

SCIENTIST 1: “I think we should colonize Mars to ensure humanity’s long-term survival!”

SCIENTIST 2: “I think we should colonize Mars to grow human knowledge!”

ASTROLOGER: “I think we should colonize Mars because Jupiter is in Venus this month!”

SCIENTIST 1: “WHO LET KENNY IN HERE? Kenny, take your spirit crystals and your night guard and go back to bothering people in the public library!

Apple will announce its second calendar quarter results next week (sans unit sales, of course), but during last quarter’s conference call it noted:

Across our stores, we had an all-time record response to our trade-in programs and with more than four times the trade-in volume of our March quarter a year ago.

Truth be told, however, iPhone sales seem to be simply stagnant, not rocketing up, not rocketing down.

And somehow the company seems to survive.

Note: When you purchase something after clicking links in our articles, we may earn a small commission. Read our affiliate link policy for more details.
Shop Tech Products at Amazon