Drawing on A/B testing experiences

An old friend of mine Sean D’Souza was in Chicago recently watching Tom Leung (Google Website optimizer team) do a presentation about the benefits of website optimizer. Tom thought that Sean had some Chutzpah when he offered to show him what he was doodling during the presentation. Sean however wasn’t doodling, he was actually putting one of his many talents to use to describe what he was seeing.

My favorite is this cartoon;

Testing Overkill
Too much testing can make things messy! Don’t try to juggle too many variables at once, do simple tests that are designed with a simple purpose in mind and then if it’s improved things, move onto the next test. If it hasn’t improved things then go back to the beginning.

When Sean showed me this article I just thought it was cool! It gave me a chuckle and shows simplistically what people should be thinking about when doing A/B testing.

Looking at the rest of Sean’s drawings I had slightly different thoughts to Tom, though I’m sure Tom was relating to his own seminar speech and putting the correlating discussions into the context of Sean’s cartoons. However I thought I’d share what i thought when I saw Sean’s work.

Be patient
Be patient. Long term positive trends are what you’re looking for.

A/B testing

This is a great one. The slow and careful tester will often get better results than the tester who tries to test everything at once. It’s true. You have to be extremely careful not to “overdo it” when doing multivariate testing. Take it easy. Do a tortoise impression.

A/B testing

Simple tests with optimizer such as A/B testing can be very effective. A/B testing is like comparing one thing to another and seeing which is best. A bit like comparing to girlfriends! :)

Bold testing

This is another cool one. Do bold changes. I would advise this on sites which are really awful at converting their traffic and need to do something radical. I wouldn’t advise sweeping changes on websites generally because you can never tell what has worked if you do too much at once. However being bold doesn’t mean sweeping changes in my opinion, it means doing things differently and trying to test an idea or a concept with something that stands out from your control (your original site).

And finally…..

Bad landing pages

I completely agree with Tom on this one and have often been in this position. On the one hand I see a massive bounce rate and think “that’s a huge opportunity for improvement” and on the other I think “God that is awful! how can they be so bad at the basics!”

Combining web analytics data with testing is a great way to decide where to make your changes and I’d start by looking at the highest bounce rate or the page with the highest number of exits from your site.

Oh and just in case I forgot to mention, just like Google Analytics, Google Optimizer is free. So this won’t cost you the earth.

Steve is a well known analytics specialist, author and speaker. A pioneer since 2002, he established one of the first European web analytics consultancies (Aboavista), later acquired by Satama (now Trainers’ House) in 2006. In 2008 he wrote his first book Cult Of Analytics published on May 14th 2009. He currently serves as CEO at Quru and has presented and keynoted web analytics topics across Europe. These include The Internet Marketing Conference (Stockholm), The Search Engine strategies (Stockholm), IIH (Copenhagen), the IAB Finland (Helsinki), Media Plaza (Amsterdam), Design For Conversion (Amsterdam) The eMetrics Summit (London, Munich, Stockholm), Divia (Helsinki) in addition to sitting on dozens of panels.

Posted in General, Web Analytics
4 comments on “Drawing on A/B testing experiences
  1. Lars says:

    I love it! I hope to see a lot more of these instantly thought-provoking comics.

    This is also a good example of visualization. đŸ˜‰

  2. Sean says:

    Thanks Steve. I had fun doing this. And yes, you were right. I had my own thoughts when doing these cartoons–and Tom had his own. And you have your own :) They co-relate, no doubt, but are still different.

    My favourite is: the Landing Pages one-because it’s plain funny. And the Multi-variate testing one. Because it was hard to bring out a topic like multi-variate.


  3. Sean says:

    And some that didn’t make it. Not all the cartoons made it.

    Here are some that didn’t.


  4. Ward says:

    I was a tester of software applications for over 20 years and what you say is spot on. The biggest mistake is to change too many things at once. How will you know which changed got you closer or further from your goal? Change only 1 thing at a time and then you will know exactly how it affects your site. Love the cartoons!

2 Pings/Trackbacks for "Drawing on A/B testing experiences"
  1. […] incentive for improvement, and multivariate testing will help you raise the bar. Image Source: Blackbeak BlogNot only should you employ multivariate testing on web pages, but other marketing materials such as […]

Leave a Reply

Your email address will not be published. Required fields are marked *



Enter your email address:

Delivered by FeedBurner

Recent Tweets
Site Sponsors

Find out how we're trying to make our clients €1bn, click below!