Recently at the SES London Jim Sterne discussed the hot topic of his new book Social Media Metrics. One of the things that stood out from his keynote was Sentiment measurement, or rather the inaccuracy of it. Jim basically said it sucks, which Iâm delighted about because someone needed to say it.
Sentiment or attitude analysis is about understanding the mood of the person making a comment or post about a topic.
Itâs been my opinion for some time that you canât measure sentiment with any real confidence in the results, at least when youâre relying on the results of tools. Jim brilliantly illustrated this by showing some sentiment analytics from Twitrratr.
As you can see from the red strikeouts over 50% of the comments here could be categorized as falling into a different category. Â Many of the negative comments about Obama here are actually positive but because they talk about âloseâ or âwonâtâ (as in might lose or wonât win) they have been categorised as negative sentiment.
They clearly arenât negative, they are actually positive âTod sez Obama could still lose pa, ohio and fla and still get to 270, thatâs a favourable playing fieldâ is a positive slant saying Obama could win without those states. It was listed as negative. Twice!
The point here is that even humans canât agree in many cases on whatâs negative and whatâs positive in terms of sentiment so how do we expect a machine to do it?
My advice is to treat âbuzzâ or âsentimentâ analysis in a qualitative way. In other words manually analyze whether itâs good, bad or indifferent! Kwantic for instance developed its own in house tools to randomly select comments around key terms.
Our tools randomly select comments and then we use one of our market research professionals to score the context of this random sample. The tool collects enough comments to ensure statistical significance is there and our analysts do the rest.
To me this is the only way to analyze sentiment correctly because only a human can put it into context, get the language right, understand slang and give an accurate picture of what people are discussing.
Comments as always welcome!