Analyst Gene Munster gives Apple’s Siri a grade D for accuracy

Piper Jaffray analyst Gene Munster has put Siri to the test, asking his iPhone 1,600 questions in total to find out how accurate the 4S’s personal assistant is.

Fortune reports that Munster split the questions into two halves, asking 800 on a busy street and 800 in a quiet room. Munster then published his results in a note to clients on Thursday.

When testing Google, the search engine understood 100 per cent of the typed in questions and replied accurately 86 per cent of the time, earning a B+ from the analyst.

Siri understood 83 per cent of the queries in noisy conditions, and 89 per cent in a quiet room. Siri gave an accurate reply 62 per cent of the time on the busy streets and 68 per cent in a quiet environment. This earned Siri a grade D for accuracy, according to Munster.

“In order to become a viable mobile search alternative, Siri must match or surpass Google’s accuracy of B+ and move from a grade D to a B or higher,” wrote Munster.

The release of iOS 6 this autumn and the decrease in reliance on Google could help Siri catch up, said Munster, who believes that the iPhone’s assistant is two years behind Google in its learning curve.

But Google is fighting back with its own smart assistant, Google Now, which was announced at the company’s annual I/O developer conference on Wednesday.

“Breaking down Siri’s reliance further, Google provides 100 per cent of navigation results, 61 per cent of information results, 48 per cent of commerce results and 42 per cent of local results,” wrote Munster. “Among other result aggregators, Yelp provided the most local results (51 per cent) and commerce results (51 per cent), while WolframAlpha provided 34 per cent of information results.”

With the release of iOS 6, Apple’s new maps service will provide all of the navigation results, Yahoo Sports will be added for sports information, Open Table will provide detail about restaurants, Rotten Tomatoes will give movie show times and Fandango will allow ticket purchases. How these changes will affect Siri’s accuracy is not yet known.

Munster highlighted several types of mistakes made by Siri in his test, the most common error being a response with the answer to the previous query.

When asked where Elvis was buried, Siri said “I can’t answer that for you,” mistaking ‘Elvis buried’ for the name of a person.

The pin would sometimes drop in the wrong place when Munster asked Siri “Where am I?”

Several other examples showed how Siri can pick up on particular words in a query, providing related information that does not answer the question asked.

Apple co-founder Steve Wozniak recently said that he thinks Apple ruined Siri, calling the voice recognition assistant “poo-poo”. Charming.

How have you found your experience with Siri? Let us know in the comments section below.