Understanding conversion attribution scoping in A/B testing

The buyer’s journey in today’s fragmented marketing environment is nonlinear, and it often consists of multiple touchpoints. As a result, understanding the statistics of A/B tests and personalization campaigns, and being able to attribute conversions and revenue uplifts to specific variations, has become a huge challenge for marketers.

If John Wanamaker were still with us today, he’d likely declare, “I know half my optimization effort is wasted, I just don’t know which half.”

Attribution in A/B testing and personalization

Attribution is the process of assigning credit to different touchpoints or engagement actions along a consumer’s conversion path. In A/B testing and personalization, it helps us figure out if we should give credit to specific experiences or variations for revenue or conversion events. It allows us to get a better understanding of the influence of our actions on buyer decisions to ultimately serve the highest converting content.

For example, take a standard A/B test. What happens if a variation is served to a user, and this user makes a purchase the next day? Should we attribute this conversion to this variation? Maybe. Maybe not. It depends on the content and context of that specific experiment. This is why moving beyond strict conversion attribution models to flexible attribution is vital to helping marketers know what is truly influencing conversion so they can optimize accordingly.

To make sure you are correctly associating conversions to a given experiment or personalized experience, you first need to understand the different conversion attribution configurations:

• Session-Level Attribution Scoping – When a user is shown an A/B test variation or a personalized experience, all subsequent revenue and goal conversion events are only attributed to that variation, if a conversion occurs in the same session.
• User-Level Attribution Scoping – When a user is shown an A/B test variation or a personalized experience, all subsequent revenue and goal conversion events from that user are attributed to that variation, for as long as the experience is active.

Which attribution level should I use?

To illustrate this further, let’s look at a common use case: triggering an overlay to get the user to subscribe to your newsletter. In order to not bombard your users with pop-ups, you are limiting the appearance of the overlay to once a week.

Imagine a user visits the site on Sunday and sees this overlay. She quickly closes the overlay to continue browsing and ends the session without subscribing. Three days later, the user returns to the site and during the course of this session, enters her email and becomes a loyal newsletter reader. Would you really want to attribute the overlay to this subscription? Probably not. Limiting the attribution window to session will help you filter out conversions that are unrelated to the variations. Reducing this noise also means that we can reach a conclusion faster to determine what actually influenced the conversion.

Now, consider the purchase of a high-value item such as a washing machine which almost always happens across multiple sessions. The user may first browse products on a website or mobile app, think about it, explore competitive offers, swing by the retail store with his partner to make sure they both feel the item has that certain je ne sais quoi element about it and eventually come back to the website to purchase. Looking only at attribution that happened in a single session might result in low conversion counts that greatly undervalue the impact of specific personalized experiences. In this case, go with user attribution.

For this reason, marketers use different conversion attribution windows when experimenting or personalizing experiences.