Challenges Publishers Are Facing Today with A/B Testing

Marketers have long seen A/B testing as the holy grail for selecting and displaying the best performing site layouts and content. Rather than basing decisions on human intuition, a scientific method is relied on to evaluate data and determine how to best maximize marketing ROI. The practice has allowed marketers to reduce the presence of ego and bias when rolling out changes on digital properties, an approach which could greatly benefit publishing and news organizations struggling to keep readers engaged in today’s hyper-saturated content marketplace.

With busy news cycles and shifts in audience attention, prioritizing automated processes into decision making in regards to what content to surface for individual readers could be the relief publishers are looking for. However, rarely do we see such optimization occur, leaving reader experiences compromised and money on the table.

So, we started asking ourselves:

• Why are publishers averse to automated optimization technologies?
• Is A/B testing simply not suitable for publishers?
• Can publishers, despite their initial resistance, still benefit from A/B testing?

There are several unique challenges media companies and publishers face when implementing A/B testing. And for some, out-of-the-box solutions won’t always cut it. Let’s dive into some of these and explore solutions that will help gatekeepers of information maximize returns, no matter the desired outcome.

Automation vs. editorial control

The fact of the matter is, publishing operates very differently than the eCommerce world, where the goal is to capture attention, not move products. Publishers are only able to monetize content once they’ve leverage strategies that drive traffic, and at the end of the day, what ends up being pushed to the top of a homepage is typically a decision made by the editors – whose role is to curate interesting, relevant, and newsworthy content. Naturally, introducing an automated solution to streamline the process would create tension – it poses a threat to an individual’s editorial expertise and the function of publishing and journalism as we’ve come to know / rely on it as a society.

This sentiment is an outdated one, though, curbing publishers from optimizing the impact of their content. In fact, many publishers can (and do) greatly benefit from A/B testing. Used in tandem with manual decision making, insights surfaced from algorithms can help inform editorial processes so publishers can ensure they are striking the best balance between front page news and content that interests individual readers. Editors can place the “burden” on automated, data-supported results without having to sacrifice the integrity of the news in order to serve users with the best user experience possible.

Important consideration:
In a fast-paced sector, traditional A/B testing simply can’t keep up or address issues such as limitations in testing scope due to periods of low site traffic. Only by employing an advanced personalization solution that leverages sophisticated machine learning algorithms can a publisher overcome these challenge in order to improve the user experience and start seeing positive results in short periods of time.

To favor audiences or individuals

Publishers have one larger goal dictating all of their decisions: surface content readers or viewers are seeking out – requiring them to remain laser-focused on delivering the most relevant, high-quality content to their audiences. To do this they must be able to collect and analyze audience data and behavior to make informed decisions that, in turn, better monetize their content.

But in an attention-driven business, consistently identifying content that can be served as a “winning” variation to a more general audience with the highest total interest in a topic, is a balancing act. For instance, just because a smaller, sub-audience doesn’t share the same preference, doesn’t mean they aren’t worthy of surfacing different content options. If your reader is highly interested in national politics and Washington, D.C. sports teams, but less interested in science and football, that gap needs to be addressed.

While there’s no one-size-fits-all approach, the right solution can dynamically allocate traffic to multiple high-performing variations to audiences. This capability, better known as a Contextual Bandit algorithm, works to keep readers happy on an individual level based on context and personal interest.

Important consideration:
The nature of an audience’s visit to a publisher site will change dramatically per reader – some will come for politics, some art, sports, or finance. This difference in intent is what can produce unfair results when testing or trying to compare categories, a strategy we suggest avoiding. Instead, test content variations within individual sections and categories to determine what pieces of content are generating clicks, subscriptions, and retention.

Sensational stories over in-depth coverage

It’s no surprise that readers love a good story. Editorial teams know this, which has led to the rise of sensational news and clickbait headlines over long-form, in-depth content. Testing in the world of publishing tends to compare variations of titles and images with a core goal of tracking higher CTRs. This has led to a rise in stories featuring the buzziest news stories or topics can tend to beat out investigative pieces, cultural content, and less flashy stories – a point of contention for editors, especially those that believe there is value in news integrity, as well as in-depth, high-quality content.

When navigating a homepage or a section page, an editor may want to prioritize a piece of content they deem newsworthy and have invested time and energy in, but an A/B test points out that a different piece of content will drive more clicks. Media companies must tackle this issue and strike a healthy balance. Rather than pitting content against each other, editors can carve out inventory on the homepage and across sections designated strictly for high-impact, quality news, and place buzzy, click-driving stories in its own reserved areas of the homepage and throughout the publication.

Important consideration:
When considering other strategies for optimizing your content placement, you can A/B test headlines, descriptions, and images, as well as the layout of your homepage and sections. Multivariate testing allows you to test different combinations of content elements to find the best variations in real time. In order to properly deploy the most relevant content recommendations to an audience, you need a testing engine that can determine accurate results quickly, in real time to maximize CTR. This means, your algorithm must be more aggressive and less statistically strict in order to quickly and effectively prioritize content, especially on your homepage where content is changing on an hourly basis. Not only will this drive up content performance, but it also gives editors the ability to track content trends and tweak their overall strategy accordingly.

Using our Front Pages feature, publishers can test combinations of content elements, including titles, descriptions, and images, to optimize content performance.

Curating content recommendations in a fast-changing news environment

Just how my favorite retailer suggests styles I may like, publishers should recommend content I am interested in. Using recommendation technology, audiences can be served tailored experiences when arriving on a publisher’s homepage. However, there are a few rules of thumb when it comes to effectively deploying content recommendations.

Evergreen content – When recommending content to an audience, be sure to identify evergreen pieces to use as variations. Outdated content will lead to a poor user experience, and relevance is key to keeping readers happy and engaged.

Fresh content – Publishers should minimize exposure to content that’s already been consumed. If someone has read an article or watched a video already, chances are, they won’t want to read it again unless specifically seeking it out. Be sure to implement targeting rules that deploy fresh recommendations based on each individual’s user behavior and content they have previously engaged with.

Complementary content – Another effective method for increasing the output of content recommendations is to implement a “Read Next” function. Once a user is on a publisher’s site, they should prolong a visitor’s stay by recommending more content the user may like. This, in turn, leads to more page views and exposure to third-party advertisements, a common source of revenue for most publishers.

With fresh news stories being churned out hourly, an automated engine will ensure that you are consistently serving readers with the best possible recommendations and experience possible.

Important consideration:
Personalization doesn’t end with what pieces of content are served — it can also help you identify what types of content to display. Some users may showcase an affinity for video or audio content, while others prefer long-form written pieces. With an automated solution, you will be able to expose your audiences to content of interest in the formats they prefer.

The balance of free and subscription-based content

Publishers face the challenge of timing. As media companies struggle to monetize content, they are consistently looking for new strategies to drive new subscriptions. Publishers must establish a working balance between offering free content and encouraging readers/viewers to pay for their content. While users will always prefer free access to any service or product, publishers have the important task of differentiating the free user experience from a paid one to drive sign-ups. They must ensure that readers understand why the experience for subscribers is significantly better and more valuable than their free offerings.

With an AI-powered platform, you can track user behavior to identify trends in the customer journey to inform when you should deploy sign-up notifications, when to target users with subscription offers, and key moments for acquiring new customers.

Trading in impressions for engagement

As tempting as it may be to adopt industry standards, be sure to identify what KPIs best suit your business and your marketing goals before implementing an automated solution. While CTRs tend to be a typical metric to track, invest more time in identifying meaningful insights that can inform your greater marketing strategy. With so many external factors impacting how publishers create and prioritize content, testing combinations of content, images, titles, and site elements using an automated, algorithm-driven AI solution can be the game-changing strategy publishers have been looking for.

Don’t waste precious time on testing elements that won’t help you measure success. Analyze your existing processes to determine what areas of your site will benefit the most from automated testing, and then, get to it!