Test and Learn: When Testing Isn’t about the Offer or Buy Button

It’s hands-down the most basic testing concept. You have an idea—maybe it’s a way to generate more revenue for your business, improve engagement, or maximize some key performance indicator (KPI). To take it to the next level you’ve got to test it “live.” Real people, real scenarios, real-time feedback. It’s a focus group or, maybe, a small slice of your audience. Here’s the idea, now go.

That’s the root of the test and learn approach, which many brands integrate in high-stakes, real-world situations. Starbucks famously employs this method, testing—and learning from—the “Trenta,” a 31-ounce drink, as well as integrating alcoholic options during “happy hour.” For these and other similar experiments, the brand segments representative populations and unleashes the test. Successes are monitored and lessons are garnered while Starbucks assesses viability at a national level.

Test and learn in the digital realm

The implications are clear for retailers and national chains like Starbucks, but it’s also the perfect testing format for digital. Think about it: massive audiences, tons of data, and real-time reporting paired with 24/7 consumer access that starts the minute you say “go.” It’s a dream platform, especially when it comes to optimizing against revenue-oriented KPIs. For some, digital has actually become the perfect test (and learn) bed before the “real” rollout.

Test and learn also adds another unique layer to the digital testing universe, since there are so many nonquantifiable points in the interactive space. Understanding behaviors and experimenting with different customer experience elements can shed important light on how to improve it. So whether we’re taking a quick peek over someone’s shoulder through user testing or analyzing heat maps to know where visitors are focusing their attention, digital audiences offer plenty of learning opportunities. How do we quantify or even monetize it? We’re certainly used to A/B testing for quick wins, but not everything is the red button versus the blue button. Sometimes it’s testing the next big idea, or reducing the risk associated with a large-scale user experience or website rollout—and that might not be a simple A versus B ruling.

In fact, often knowing even what to test—and which audiences should be part of that test—can be elusive. That’s where using machine learning can come in handy. Using automated personalization as part of a test and learn strategy can be an effective way to make informed go-forward marketing decisions. The result of using AP in this fashion is a wealth of insights that can be used to shape marketing strategy (as well as gain some attractive short-term conversion lift in the process).

Overcoming obstacles in test and learn

Despite its unique benefits, many marketers still struggle with organizational intolerance for anything that doesn’t drive immediate, measurable results to the bottom line. But you’ve got to take calculated risks to excel—you can’t argue with that. In Adobe’s Digital Roadblock Survey this year, more than half indicated ideal marketers should take more risks, and 45 percent hope to, themselves. Test and learn tactics can be that runway, where new ideas percolate and marketers experiment, learn, and innovate, without overwhelming risk or irreparable consequences. Some of the most optimizationally mature organizations I’ve encountered do just that—learning is core to their individual and overarching goals. It makes sense—marketers are endeavoring to influence total consumer experiences. How can you do that when you can’t test and learn? Hundreds of A or B/this or that/red or blue moments won’t provide the same consumer experience view quite like a test and learn approach.

Keeping in mind the potential resistance, I recommend starting modestly and evangelizing wins while sharing the indisputable effectiveness of this type of testing. Try the “hard” wins first—increased conversion rates, average order value, or something that has both inherent learning benefits to the marketers and solid KPI wins for stakeholders. This heightens credibility and perhaps gives you more runway to extend your testing to those more learning-oriented or strategic initiatives.

The risks and challenges

In addition to stakeholder pushback, some critics also lament the cost and coordination. Whether it’s training staff, bringing in consultants and outside experts, or simple must-haves like new signage, displays, branding, or even product lines, there can be substantial resource allocation toward a test and learn approach, that could be better controlled through more traditional A/B or multivariate testing. There’s a real need to get granular, and that’s when the costs start to rise even for basic must-haves like tracking mechanisms and comprehensive analysis.

Also, often, there’s fear around aligning “real” traffic with tests, especially as we enter the holidays. That shouldn’t surprise anyone: a test and learn approach, unlike other seasonal testing rollouts, requires a good amount of meaningful traffic and high-potential challengers be integrated, in the name of learning and future optimization. The more data-savvy, yet cautious, might want to consider experimenting in a “laboratory setting” by modelling offline data to identify key (and potentially unknown or untapped) audience segments, and determine the correct algorithmic approaches before going live with them.

Determining the right test and learn approach

A full-blown test and learn strategy potentially requires a significant carve out. For brick and mortars that could be a set number of stores in a critical or representative market; for online it might mean trying an entirely new creative direction with an important traffic segment—a strategy not entirely without risk, as we’ve already discussed, but one that will yield tremendous learning insights. Daunting? Maybe. But certainly an effective way to learn, validate, and realize upside. Start small, says Korn/Ferry Senior Client Partner Caren Fleit, and “learn something and then adapt it as you go to bigger scale before you throw all the money behind it.” And don’t be scared to fail—as Forbes’ John Ellett explains, simply, “Inherent with testing and learning is realizing that tests don’t always work, that there are times when you test and fail.” What’s more, in a test and learn environment there’s really no true “fail”—learning the challenger is a dud is a positive. You definitely won’t go down that rabbit hole now!

But regardless of your testing objectives, the benefits of test and learn—be they traditional wins, valuable takeaways, or, conversely, proving the hypothesis wrong—far outweigh the downside of potentially losing out on short-term wins. It’s also a great approach if you just want to test to learn—imagine that! Maybe by observing user engagement you can develop a meaningful story and subsequent dialogue with consumers, ultimately unearthing what your organization needs to grow and evolve. The gains, no matter what they look like in the immediate, will provide a powerful series of next steps that, once implemented, retested, or completed reconsidered, will steer your organization toward ongoing success thanks to unparalleled consumer insights.