Computer deciphering of word meaning is not always accurate and tone can be completely missed. Even the leading vendors acknowledge that the data is 70-80% reliable. For example, we may know that the phrase “quite interesting” means one thing in America, another in Britain, but the computer would see the same meaning. Note some of the long-standing issues with voice recognition technology.

There are questions about how robust or representative the data is. Are a brand’s tweeters the key WOM influencers or are they just a small vocal segment?

Some brands and products may be under the radar for this technology. Yes we love to chat about Apple but do we also regularly, enjoy blogging and tweeting about Charmin or business insurance?

There are conflicting approaches, metrics and offerings; over time a common Microsoft, Google, Nielsen type platform may emerge.

The notion of accurate sentiment analysis is very intriguing, but, as Randall notes, it is far from a finalized technology.

On the one hand, we now have access to an unprecedented about of data about people’s opinions that is in constant flux, constant evolution, and is constantly being updated. In business (and, I would argue, life) the key is lessening your information gaps, reducing the information asymmetries you face. Often times this can be accomplished by finding a way to take the private information people hold (e.g. opinions about a product or brand, their preferences and priorities, etc.) and making it visible. This is the essence of market/consumer research. The current environment makes the collection of that data much easier, especially at high volumes, and more cost effective.

However, the only way to derive usable, reliable information from this ocean of data is to properly code it. If we can develop reliable technology that overcomes some of the current shortcomings we will be in a position to literally visualize the collective mind, and do so real time. That is a very exciting prospect, but one that will be difficult to achieve.

As I was watching the video I thought about the potential to leverage such a platform for large-scale coding of qualitative data. Coming from the social sciences, often we find the need in large scale research for the massive coding of data, whether it is language from a speech, the tenor or sentiment of quotations (or newspaper articles in media studies), the nature of cases (i.e. did country A make a threat to country B, did country B back down as a result, etc.), or the responses from an open-ended survey. Coding is an issue whether you conducting qualitative or quantitative analysis–especially where you have captured large amounts of data. Often times the data is not inherently numerical and needs to be translated so that quantitative analysis can be conducted. Likewise, with a qualitative approach one still needs to categorize various data points to allow for meaningful comparisons.

The interesting thing about a service like Crowdflower is that it can leverage a ready group of workers globally who are ready and willing to conduct the coding at a reasonable price. Additionally, Crowdflower utilizes various real-time methods to ensure the quality of the coding. Partially this is achieved through the scoring of coders relative to their past performance, how they fair on tasks that are “planted” by Crowdflower (i.e. salting with tasks where the correct answer is known ahead of time), and how much agreement there is between coders on various items.

The final method comes up quite a bit in social science research when you have to determine how to categorize a given piece of data. The level of agreement is crucial to confidently coding a particular case. I would imagine that a platform such as CrowdFlower could make that task easier and more robust by quickly tapping into a larger pool of coders.

Has anyone used a service like CrowdFlower in this way (i.e. coding data from qualitative research)? Would be interested in your perspective.