Choosing the right conversion optimization objective

Conversion rate optimization can be a very powerful process, but it’s easy to get carried away by the methodology. Despite the best intentions, many conversion optimization initiatives fail to produce valuable results, resulting in misleading and unsubstantiated insights. Fortunately, the biggest blunders can be prevented by working methodically.

A conversion optimization process can fail for a variety of reasons. One of the biggest considerations when running experiments revolves around choosing the right optimization objective. Generally speaking, conversions are measured when a visitor executes actions that are defined as valuable to your business. Conversion can be measured as a click on an element, completion of a goal, or any online action which has a direct impact on revenue. Whether you’re trying to improve display ad clicks using CTR-based optimization, generate more leads using goal-based optimization, or increase sales using revenue-based optimization, choosing the right objective for your optimization initiatives can make a huge difference between their failure or success.

Having a specific, measurable objective in mind, however, is not enough, and that’s the first thing you need to realize before planning a test. Many other variables are involved, such as the type of experiment you’re running, the audience sample size, experiment length, conversion attribution, the number of content variations that are being tested, and how prominent the changes being tested are. It’s not just about measuring the results – it’s about optimizing the data as effectively as possible using the right optimization method.

Below is a table that illustrates this point. Use it to get a better idea of what the optimization objective of your experiments should be, based on sample size and estimated length of the experiment. If you don’t know what the sample size is, use a sample size calculator. Of course, there is no silver bullet, and this table can only serve to guide you towards making the right decision.

How to Choose the Right Optimization Objective:

Time is of great concern, and it’s one of the main factors influencing the decision of choosing the right objective for your experiments. If you’re running an eCommerce site and intending to get fast results for an experiment you’re just about to launch, remember this: Revenue-based optimization requires greater purchase cycles than CTR or goal-based optimization.

With revenue-based optimization, it’s far more complicated to attribute the actual conversion to a single experiment variation. It takes more time for a random visitor exposed to an experiment to complete a purchase than it takes to complete a click on an element. Some customers compare prices, some are just slow buyers.

In fact, many things can happen between the first exposure to a live experiment and the final destination of completing a purchase. So as a general rule of thumb, if you’re looking for fast results, forget about optimizing for revenue and stick with shorter and simpler conversion cycles, such as click-based optimization or goal completion optimization.

Avoid peeking until the completion of the test

Drawing conclusions too quickly is also a major issue. It can be difficult to keep your cool and ignore an (allegedly) obvious winner a few days into a test. While results vary across industries and sample sizes, waiting for the optimization process to accumulate enough data to achieve statistical significance is absolutely crucial. Even if your testing tool forecasts incredible uplift, keep the test running for at least a few weeks or a few purchase cycles before starting to analyze results.

While some experiments reveal major uplifts early in the test process, results can vary dramatically over time, causing many marketers to draw the wrong conclusions and even lose money over time if they implement changes based on the initial results.

Prioritize tests accordingly

Every test is fallible. If you can’t send enough traffic and generate conversions to all variations, your efforts will go unrewarded. Just because you can test a page element, that doesn’t mean you should. In fact, the larger the sample size, the more data becomes available for the system to optimize. Make sure you test noticeable changes that are big enough to have a real impact on the bottom-line business. Without sufficient data (e.g. traffic or conversions), your experiments will never reach the statistical significance minimum of 95% and thus will be bound to fail.

Every test requires considerable resources (budget, time, and people), so stick to the elements you know make a difference, such as messaging, hero banners, and the call-to-action (CTA). Start testing minor features and you’ll find yourself lost in data you can’t use. There’s no point in running short-term optimization initiatives for low-traffic sites. If the sample size is low, the system will struggle to optimize and gain real uplifts. It would take a lot of time and may eventually even fail.

As with any tool or methodology, a single experiment will not revolutionize your business, so don’t expect the test to produce a “Silver Bullet.” However, integrating testing over time and rolling out insights across your website will introduce incremental improvements that can (and do) reach a tipping point. Take a granular approach, choose the right objective, and run your optimization initiatives intelligently and methodically. Plan for small wins, and they’ll add up to a big difference in the long run.