A/B Testing: The science of marketing

Did you just read ‘science’ and vomit a little bit in your mouth? That’s OK, I did too. But Hopefully I can explain the nuts and bolts of A/B testing in the least scary way i.e. with the help of Ron Burgundy, Ryan Gosling (you’re welcome) and a Crafty-Homeless-Guy. A/B testing is actually far less hideous than it sounds and it is imperative to understand it if you want your online marketing efforts to pay off.

First thing you need to know: Content Matters!


Thanks Ron.

Seems obvious yeh? But how do you know which content is best? A gut feeling? Aesthetics? Industry experience? Ron’s mustache told you? Unfortunately, most marketers tend to judge the efficacy of their online content based on intuition or what looks best, however, none of these techniques necessarily work (except for maybe Ron’s mustache). To know what elements of your online marketing tools work best marketers need to do an A/B test.

What is A/B Testing? A/B testing is where you compare two variants of online material – whether that be a website, an email, a social media post, an advert or a landing page (nearly anything which alters viewer behaviour can be tested) – to determine which one works best. For example, when testing your website you might want to know if blue font (Variation 1) works better than your current black font (Variation 2). To do this you would split the traffic visiting your website into two and for a specified period of time half your visitors would view Variation 1 (blue font) and half would view Variation 2 (black font). To define what ‘best’ looks like a measure of success must be chosen. For example if we were a clothing site this could be a visitor seeing the confirmation page following a sale. After a period of time Variation 1 (blue font) is compared to Variation 2 (black font) in terms of this measure of success (seeing the confirmation page following a sale). The variation with the biggest score in terms of the measure of success (i.e. going through with a sale) wins and is therefore the better version that should be used in the future, until a better version (as determined with an A/B test) is found.

Here’s a Crafty-Homeless-Man A/B testing religion


If, like this man, you feel the urge to immediately A/B test something at any point whilst reading this blog feel free to click here for a step-by-step guide to developing your own A/B test.

Why use A/B tests?

A/B tests are critical because our intuition can often be wrong – I’ll show you an example in a second. By using A/B testing you remove both your conscious and unconscious bias in decision making for an objective measure of what ‘best’ looks like for the measure in question. What this means is that your marketing works better without working harder and your bottom line improves.


Just to show how faulty our intuition can be which landing page do you think resulted in the most conversions? Post your answers in a comment to compare what other people thought.

Version A:                                               or     Version B:

Screen Shot 2015-08-18 at 2.14.12 PM          Screen Shot 2015-08-18 at 2.14.27 PM

If you thought Version B, you were with the majority. However it was actually Version A (the one with the image removed) that got a 24% increase in form submissions! This shows how important it is to A/B test to differentiate your intuition from what is true. For more surprising A/B test results click here.

A/B testing has revealed some astounding insights into how small changes to call-to-action buttons can have big impacts for marketing’s effectiveness. I’ve included a few interesting ones below.

Changing copy on payment page button

Screen Shot 2015-08-18 at 2.30.33 PM

In this scenario including the words ‘get started’ added an extra benefit, as consumers were often in a rush to start writing. This resulted in a 31% increase in conversion.

Is bigger better

Screen Shot 2015-08-18 at 2.37.40 PM

Not necessarily, especially when it comes to call-to-action buttons as can be seen above.

I hope I have dispelled some of your fear regarding A/B tests. They are less scary and easier to create than you would think and most importantly they are indispensable in today’s marketing environment where information overload is the norm.


Don’t forget to comment and say which version of the landing page you intuitively would have picked. 


3 thoughts on “A/B Testing: The science of marketing

  1. I too went with version B of the landing page..
    This is an awesome way to discover your what works best for your site, with your bias eliminated! It would be great to know the sample sizes for some of these statistics (just for my own curiosity).
    You’ve discussed the A/B testing in such a simple way and made it very easy to understand. Big fan of the Ron burgundy post as well!
    As you mentioned on my blog, it is definitely worthwhile for a music producer, or anyone wanting to promote and market their own sort of ‘brand’.
    Very well written blog! Well done!

    Liked by 1 person

  2. I guess I join the masses with faulty intuition- I assumed version B of the landing page would be most popular. To me the picture makes the page seem more welcoming. However, as you pointed out, it’s important to eliminate your own bias when designing an appealing website for a business. A marketer needs to cater for it’s customers’ (or users’) preferences, not their own.
    After reading Smashing Magazine’s guide to A/B testing (http://www.smashingmagazine.com/2010/06/the-ultimate-guide-to-a-b-testing/) I was keen to give it a try on my own website, but was a little confused how to go about it. The concept is so simple, but I see myself as scoring pretty low in the tech savy department. I checked out your link to the quick tuition about how to set up an A/B test. I’ll definitely be using this tool for my own website later this semester. Thanks for breaking it down 🙂

    Liked by 1 person

    • I thought the same about version B too. It shows how important these tests are. I massively understand how daunting the thought of trying an A/B test is though (am not the most tech savvy myself), but there are some amazing tools online to guide you and the potential for website optimisation, like you said, is too good to miss.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s