Don’t Get Sentimental About Tools When Measuring Attitude

Recently at the SES London Jim Sterne discussed the hot topic of his new book Social Media Metrics. One of the things that stood out from his keynote was Sentiment measurement, or rather the inaccuracy of it. Jim basically said it sucks, which I’m delighted about because someone needed to say it.

Sentiment or attitude analysis is about understanding the mood of the person making a comment or post about a topic.

It’s been my opinion for some time that you can’t measure sentiment with any real confidence in the results, at least when you’re relying on the results of tools. Jim brilliantly illustrated this by showing some sentiment analytics from Twitrratr.

twittratr_screengrab

As you can see from the red strikeouts over 50% of the comments here could be categorized as falling into a different category.  Many of the negative comments about Obama here are actually positive but because they talk about “lose” or “won’t” (as in might lose or won’t win) they have been categorised as negative sentiment.

They clearly aren’t negative, they are actually positive “Tod sez Obama could still lose pa, ohio and fla and still get to 270, that’s a favourable playing field” is a positive slant saying Obama could win without those states. It was listed as negative. Twice!

The point here is that even humans can’t agree in many cases on what’s negative and what’s positive in terms of sentiment so how do we expect a machine to do it?

My advice is to treat “buzz” or “sentiment” analysis in a qualitative way. In other words manually analyze whether it’s good, bad or indifferent! Kwantic for instance developed its own in house tools to randomly select comments around key terms.

Our tools randomly select comments and then we use one of our market research professionals to score the context of this random sample. The tool collects enough comments to ensure statistical significance is there and our analysts do the rest.

To me this is the only way to analyze sentiment correctly because only a human can put it into context, get the language right, understand slang and give an accurate picture of what people are discussing.

Comments as always welcome!

Steve is a well known analytics specialist, author and speaker. A pioneer since 2002, he established one of the first European web analytics consultancies (Aboavista), later acquired by Satama (now Trainers’ House) in 2006. In 2008 he wrote his first book Cult Of Analytics published on May 14th 2009. He currently serves as CEO at Quru and has presented and keynoted web analytics topics across Europe. These include The Internet Marketing Conference (Stockholm), The Search Engine strategies (Stockholm), IIH (Copenhagen), the IAB Finland (Helsinki), Media Plaza (Amsterdam), Design For Conversion (Amsterdam) The eMetrics Summit (London, Munich, Stockholm), Divia (Helsinki) in addition to sitting on dozens of panels.

Posted in General, Web 2.0
11 comments on “Don’t Get Sentimental About Tools When Measuring Attitude
  1. Steve,

    Nice post. Human interaction will always be required and whilst I’m sure that some clever sod will eventually develop a tool which will give more accurate sentiment. More than a glancing eye will be required in developing the insight from social media.

    What I am seeing that there is lots of interest in investing in social media – whether this be twitter feeds, facebook fan pages or blogs. But there is little or no interest in understand the behaviour of people on this sites or the halo / direct affect on revenue (or for that matter the influence these have in the purchasing process).

    Great to see you at #seslondon and hope to see you again for another beer soon.

  2. Phil Dearson says:

    I’d say (and indeed I do) that most automated sentiment analysis tools are in their infancy unless you can spank 50k on something like Cymfony. Until AI and natural language processing becomes more advanced that’s likely to remain the case. However, it’s worth mentioning that systems like Get Satisfaction are good ways to measure sentiment when you have people engaged enough to tell you how they feel.

  3. Bill Porter says:

    I can understand your frustration with many sentiment analysis tools. The problem with them is that they make no real attempt to “understand” the meaning in comments – they don’t understand sentence structures, phrasing, can’t disambiguate, etc. But with deep semantic technology (I make no apology for directing you to http://www.expertsystem.net, more specifically our product Cogito Monitor http://tinyurl.com/yzscnsf), not only can greater accuracy be achieved, but deeper insight. So the brand or product manager is informed about what it is that the authors of on-line opinion like or dislike about brands/products/services.

  4. Tom Miller says:

    I think that your point: “…even humans can’t agree in many cases on what’s negative and what’s positive in terms of sentiment so how do we expect a machine to do it?” really sums up the problem. As someone that does Social Media monitoring, I have found that automated sentiment analysis tools, even on a very basic good/bad/neutral rating scale, are only 25-50% accurate. If you consider the complexity of our written speech and then the added transformations that we make to written speech online, well, then 25%-50% actually isn’t too bad!

    I think that automating sentiment rating to a high accuracy threshold (>90%) via software is probably impossible given our language complexity. However, it is possible to automate processing by organizing human raters. By being well-organized and deploying something like Amazon’s Mechanical Turk (or some iron-willed interns!), you can get results that have a high level of accuracy.

  5. Fully agree here and it is good to raise this. Sentiment analysis is the “hot” features that business stakeholders are after when it comes to SM monitoring & analysis. But it’s a trap :-)

    I have been evaluating different “pro” social Media monitoring tools and all have “automated” sentiment analysis features. And all sucks (60% accuracy is far too low)! But many vendors admit it – and recommend to do sentiment analysis manually (when possible).

    Note that progress are made in that area but there is a long way to go.

    Finally, automated sentiment analysis on Twitter is a non-sense. How can anyone think it is possible to do sentiment analysis based on 140 characters? No way!

    Michael

  6. Blackbeak says:

    @Miles
    Good to see you too. I think your point is spot on, social media at the moment is the hot topic but businesses don’t understand how to measure it or use it.

    It reminds me of 1995 when everyone wanted a website because “the jones’s” had one, no strategy, just get a brochure and put it online. While I think social media will mature faster I still think we’re a year or 2 away from seeing strong social media strategy being developed.

    @Phil
    I agree but even tools like Cymfony need human analysis to get it right. Get satisfaction is a great concept, thanks for sharing. Crowd sourcing feedback for products and services is a great idea but again requires an analyst.

    @Bill
    I am convinced you can get deeper understanding with tools like Cognito or Cymphony as Phil mentioned, I’m sure your tools have great value. However what I am saying is trust tools about as far as you can throw them when you’re talking about sentiment! If it says 90% positive check it out, don’t rely on the tool being right.

    @Tom
    Spot on! :)

    @Michael
    I agree. If you want to try us out on sentiment analysis I will do you a good deal with guaranteed deliverables! :) Drop me a line if you’re interested.

  7. Mikko Kotila says:

    Automated sentiment analysis is a joke. What’s worse, vendors are doing their best to make sure that customers are looking for it. Take the Oscars for example, how useful is Firth 62% vs Clooney 58%? Especially with 20% error margin that vendors publicly state!

    “Ah, that’s nice”

    There is so much more we can do, directions that are going to actually be useful in driving commerce.

  8. Mikko Kotila says:

    More about why automated sentiment is all wrong here: http://ow.ly/1eXm3

  9. @Mikko;
    That’s a really great question you posed on your blog. Why is knowing sentiment useful? How does it drive commerce? It’s a fair point because directly it doesn’t.

    However done correctly and analyzed well it helps answer the “why” question in my opinion and knowing why people do things, what they’re happy about and what they’re unhappy about allows a business to take decisions.

    At Kwantic we view sentiment analysis as just another form of market research. Market research never has a directly attributable impact on sales because something else then has to happen to make the investment in research possible to monetize.

    One of my clients for instance conducted market research on one of their products and 60% of their customers said they wanted a specific application to work with the device they were selling. The client listened, put the app on the device and sales rose. Was it the fact that they put the app on the device that caused the increase? Yes it was and that is the directly attributable part. But it was research that drove the action.

    There are pointless ways to use sentiment analysis. Your post excellently demonstrates this with the example around Clooney and Firth, I agree with you that I can’t think of a single reason why that information could be used in a business sense.

    I hope though my point about the research aspect answers your question about why we find it valuable.

  10. Mikko Kotila says:

    Thanks for the reply Steve. I do understand that research information is a vehicle for better decision making. I do admit to being a big fan of market research, even though I do see quite a few problem with the methods. For example questionnaires: expensive, slow turn around to results and relies on response. We can do same much more effectively using social media. But only when real humans design the methodology (what to search, where to search, etc.) and analyze the results. Just like in market research, or like you do at Kwantic.

    I can totally see how +95% accurate sentiment net value combined with other data (for example sales data) over long period of time can be very insightful.

    For example: If you do one press release and use sentiment change as a gauge for success, you’re not doing a favor for anyone. On the other hand if you have historical data of 1000 press releases and key message relevant sentiment changes (not brand level changes) we’re already talking about some pretty cool analytics :)

    The problem is that whenever there is a report that talks about brands/topics, net sentiment is always a ‘key’ metric. This puts a lot of people in a completely wrong mind.

  11. Mike Layton says:

    Sentiment analysis is challenging enough in traditional media. Applying it to social media is a futile exercise at best. You are absolutely spot on, human analysis is the *only* way to qualify sentiment. Love your example.

    Thanks for sharing, Mike.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Subscribe

Enter your email address:

Delivered by FeedBurner

Recent Tweets
Site Sponsors

Find out how we're trying to make our clients €1bn, click below!