Should businesses use collective intelligence and the wisdom of the crowds?

Amazon reviews are just as likely to give an accurate summary of a book’s quality as those of professional newspapers, according to a study from Harvard Business School.

Professor Michael Luca and his co-authors analysed the top 100 reviews from 40 media outlets, including the New York Times, and the Washington Post, between 2004 and 2007 for their paper. The academics used data from reviews aggregator, which summarises professional reviews and then awards ratings, if not given, based on content. They also looked at Amazon reviews for each title.

Although the study points out that there is “virtually no quality assurance” in Amazon’s consumer reviews, which can also be “gamed” by publishers or competitors submitting false reviews, they found that, nevertheless, experts and consumers agreed in aggregate about the quality of a book.

Another piece of research looking at the reliability of information shared through social media. Earlier in the year, a report was published of researchers who looked at mining Twitter to predict the success of movies. The study was not so positive in this case:

Overall, the study found no clear evidence that shows a direct link between Twitter hype, ratings and box office sales.
“The most surprising finding was that Twitter data may not be representative enough of the total population, so it is somewhat risky to use the site for forecasting,” Sen said. “More sophisticated techniques may be needed to understand the applicability of such data sets, such as the metrics we developed to understand the extent of the difference between Twitter users and other online rating side users.”

Others, like The Economist magazine, still see potential:

search-volume forecasts will help spot consumer trends of this sort with increased precision. But the improvements they bring will be incremental. Sophisticated methods based on natural-language analysis of tweets, blogs, or Facebook pages, by contrast, hold greater disruptive potential. As users of social media grow accustomed to sharing highly personal information, apparently unfazed by market-research outfits like WiseWindow watching their every step, the feelings and intentions of hundreds of millions of people are there for data-hungry computers to see.

Really, the reliability of crowdsourcing has to be looked at in context. Structured or designed crowdsourcing sites like , Amazon’s Mechanical Turk and others are seeing success. Competitions and gamification are popular technique, perhaps hinting at the value of using social media and Web 2.0 to achieve scale or breadth of participants rather than the wisdom of the crowds as such. Crowdsourcing is also useful where no other viable method exists to solving the problem, because there is nothing to lose in those situations. Opinion and research on the reliability of Wikipedia is also an ongoing story, but what is perhaps more interesting is the fact that Wikipedia has an article discussing its own reliability – transparency is critical when evaluating the reliability of social media. “Experts” on the other hand typically don’t like to be challenged.

What about crowdsourcing inside businesses?

All the caveats above apply, particularly beware that the size of the pool may affect the quality of outputs. This is why is makes sense to extend crowdsourcing to business partners and customers. But crowdsourcing doesn’t need to be about making decisions or prediction markets, it can simply be about sharing information widely, getting tasks done too, solving small problems, and gathering feedback. “Working outloud” is another way to tap into collective intelligence about what is going on and who is doing what, which has the potential to be mined (tools like Jive and Atlassian Confluence for example are already doing by telling users about popular content or making recommendations on relevant people and content).

Finally, lets not forget the role of the user, as information literacy is critical. We can’t blame the technology for all its hits and misses.