Five ways to screw up your A/B testing efforts

A/B testing and Conversion Optimisation are often hailed as a silver bullet to your website’s conversion woes. It’s often too easy to get carried away with the hype that you end up making mistakes that will cost your business (or your client’s business) dearly. Bare in mind that Optimisation when done right can have an amazing impact on your revenue and profits. But use it wrongly and you will more than likely make a wrong decision based on the “signals” you get from the results.

In this blog post. we will cover the top five ways you can mess up your optimization efforts.

No strategy

An A/B test got a 400% uplift on Which Test Won? Surely, you can do the same and reap the rewards. Right?

Wrong! Which Test Won and blog posts are notorious for showing you all the positive results but really none of the processes that go into it. More on that later.

Seeing everyone have this “success” with optimization colours your view of the underlying forces at work that make a test successful.

You then end up running “easy” tests where you test the colours of your CTA button or long pages vs short pages and where the conversion lifts you achieve are mostly illusionary.

The first step is to understand the business needs and customer needs. After all that’s what you’re trying to achieve with CRO aren’t you? – Helping your business make more money by aligning it with your customer needs.

The next step of a strategy is to have a repeatable and scalable process that you can rely on to come with strong ideas and hypotheses.

No process

A/B testing is not Conversion Optimization. In fact, it’s only one part of the Optimization process.

Strategy and process go hand in hand. If you don’t understand your overall reasons for optimizing and the strategy behind it, your process will also suffer.

Before jumping the gun and deploying test after test, you need to take a step back and look at your data. Mine your data for insights. Segment it to understand why certain types of customers are behaving differently from others or not converting as well. This will help inform your hypotheses. A solid hypothesis will help create a strong foundation for the test.

Michael Aagaard and Craig Sullivan came up with a hypothesis framework that summarises what a strong hypothesis should contain.

Changing (element tested) from _____ to _____ will increase/decrease (a conversion metric) for ________ (customer segment)

Your process must be a cyclical process that starts at Observation then ideation then hypothesis then start testing after which you analyse the results to gain insights and that takes you back to step one.

No Quality Assurance checks

This one is where most tests fall flat and cause CROs to misinterpret their results. Trust me on this. I’ve been there myself.

A/B testing technology at this moment relies on Javascript. Tools like Optimizely allow you to inject and manipulate code via a Javascript library called JQuery. If you use their visual editor, you will see the changes you make will be interpreted as code in their code editor. It is this code that gets injected into the variation.

But, users may land on that variation via a multitude of different devices and herein lies the problem. Different browsers interpret the code differently and can show a (slightly) different experience to different users.

This can result in breaks in the CSS styles or layouts.

As such this pollutes the data and can affect the conversion rates of the variation.

Unless you QA the test you would never know whether such an error was responsible for the conversion rates or not.

We have a more detailed post on Quality assurance coming up soon.

Calling your test too early

Hey! We hit significance! Our testing tool says its a significant uplift.

We have seen this happen before. Tests that were running for just two-three days were stopped because they were deemed to be statistically significant. Let’s just say you called the winner at that stage and decided to implement the result sitewide. What happens when the results flipped but you weren’t able to measure it correctly.

The standard time a test should run for that most CRO experts agree with is 14 days – 2 business cycles so it can truly capture the fluctuations in the week/weekend trends.

Don’t stop your test too early just because your testing tool says so.

Granted that some testing tools are now using Bayesian stats or multi-armed bandit algorithms that allow you to reach results faster, you still want to adhere to the rule of testing it for a minimum of 14 days (7 at least if you’re strapped for time)

No Post test analysis

Testing tools make it easy for you to look at the outcome of your experiments but if that’s all you look at then you’re missing out on some crucial insights which can inform your decisions.

When setting up your A/B test, make sure you integrate with your analytics tool. Most testing platforms like Optimizely already integrate with Google analytics so you can set it up in a few simple steps.

Once you get your test into Google analytics, thats when you can dig deeper into the data and break down the results based new vs returning visitors,channels, events and even your own custom segments.

Now you can not only see the goals you have set in your testing platform but also other interactions those users have taken on your website.

Ton Wesseling from Online Dialogue also suggests about taking the integrations one step further (something you can also do if for whatever reason your testing platform doesn’t integrate with GA) using event tracking.

window.ga('send', 'event', 'Optimizely', 'exp-XXXXXXXXXX', 'Variation1', {'nonInteraction': 1});

Now, you can create segments which can be used to analyse your test results further.

Do you know of other ways you can mess up your optimization efforts?