From a One-Person, One-Department Team to Enterprise-Wide Optimization

Image source: Adobe Stock / Paul Lemke.

When Steve Rude was hired to test and optimize the website for the legal division of Thomson Reuters, he may have felt just a little in over his head — especially when he discovered he was the only person doing testing in the entire Thomson Reuters organization. At that point, he decided that he could either panic and quit or dig in and make it happen. Fortunately for both him and Thomson Reuters, he chose the latter.

That was seven years ago. Thomson Reuters is now expanding its optimization program throughout the organization, and Steve is stepping into the role of senior manager of optimization and testing for the entire company. He has earned that role through enthusiasm for his work, a love of learning, and in no small part from being a super likable guy with a great sense of humor. (Just watch his Adobe Summit presentation to see that humor in action and learn how to calculate your testing ROI.)

Steve recently shared with me how both he and the Thomson Reuters optimization program evolved to their current expanded state. He provided valuable tips and advice, along with some personal experiences of learning those lessons and putting them into practice.

Tip 1: Get buy-in for the program by showing its value

Like most optimization professionals, Steve will tell you that the first priority of any program is to get buy-in for testing from executives and business stakeholders. Without it, it’s difficult for a program to gain traction. Steve was fortunate that early on, he was given the freedom to learn and explore testing — leadership of the legal division already believed it was important. However, Steve never took that support for granted. With some successful projects under his belt, he started doing “road shows” in which he presented capabilities the optimization program offered, experiences tested, and successes. Those road shows generated excitement and helped new team members and stakeholders brainstorm about all the possibilities for optimization and personalization.

Steve also realized that he needed to demonstrate a return on the program’s efforts to maintain that support. That’s where his degree in marketing and economics became useful. He devised calculations that put a dollar value on the results of his winning experiences — his Probable Annual Revenue Impact (PARI) reports. These reports enabled him to show how a single test increased subscriptions by almost a quarter and generated an estimated $400,000 to $1 million in revenue over a three-year period. Proof like this, with calculations that were understandable, really helped bolster that all-important buy-in.

Tip 2: Go “old school” to develop processes and test ideas

Steve says his approach to developing his optimization processes and test ideas may be “old school,” but it works for him. Anytime he hears something interesting about optimization or personalization, he jots it down in a notebook. If he hears it again, he highlights it. Each week he reviews his list and scratches off ideas that don’t feel like a good fit for the business or its customers. As he puts processes into practice, he finds aspects of them that make sense to keep, while others don’t, and he refines and evolves his processes accordingly. Steve emphasizes, “I don’t think there’s a perfect process for an optimization program that you can simply take and apply to another program without some modification. Start by looking at what others are doing, and then refining that through your own experiences or understanding of your business and its customers.”

Testing involves a lot of processes from start to finish — from researching and coming up with test ideas, to analyzing data, QA-ing tests, monitoring tests, concluding tests, conducting post-test analysis, and communicating and archiving the results. Steve suggests having some work going on within each process area at all times. This lets you continue moving from one part of the process to the next, which enables your program to continuously deliver value.

Tip 3: Be aware that words matter when working with stakeholders

A barrier that Steve encountered when he first started testing was the perception of business stakeholders such as marketers that his testing was going to prove their ideas wrong. He discovered that saying things like, “Let’s test your marketing campaign idea” made marketers feel like he was actually testing them. He learned to instead talk about optimizing campaigns to improve their ideas.

In one of Steve’s first big testing wins, he optimized the form on a landing page for a paid search campaign in an effort to get more people to submit the form. By changing the form from its original eight fields arranged in a single vertical column to four fields in two side-by-side columns, form submissions increased by 140%. Tests like these helped marketers realize how optimization could help them gain even more from their campaigns — and make them look good.

Tip 4: Promote the idea that every test is a winner

It’s easy to focus only on winning tests. Steve and many other optimization professionals will tell you that really, every test is a winner. He explains that even if you’re not gaining revenue or increasing conversions, you’re learning what your customers do and don’t prefer and what improves or worsens the website or customer journey. You’re also protecting the business from losing sales and revenue from a poor experience that might have been launched without first testing it — one of the most underrated values of testing.

Steve suggests that actively promoting the concept of every test as a winner and showing that a short-term loss can actually be a long-term gain can help senior leadership see the value of a losing test, an idea that may otherwise be difficult for them to accept.

Tip 5: Clearly define personalization and set a measurable goal

Steve found that as personalization became increasingly recognized as valuable, many people in the legal division wanted to try it. His marketing partners were frequently asking him for ideas for personalizing. Yet, for some reason, they couldn’t get started. That’s when he realized that many of them viewed personalization as a one-to-one tactic — more along the lines of, as Steve says, “Hi Bob, welcome to our website.”

Steve decided that they needed a definition of personalization, which he defined as “dynamic modification of any content, task, or specific asset that’s targeting a specific audience.” The definition showed them that while personalization can be one to one, it can definitely be one to many when targeting at the audience level. Steve also recommended that they define a measurable goal — for example, to increase visitor engagement or conversions.

Creating that definition and advising defining a measurable goal really helped unblock them from coming up with ideas. Last year, the legal division alone ran around 700 personalized experiences on their website. Steve wants to expand that to the other divisions this year.

Tip 6: Focus on test quality over test quantity

Steve has noticed a familiar conversation topic whenever testing and personalization professionals get together: testing velocity. Everyone asks how many activities you should run per month or year. He explains that when he started off, he might have run 35 to 50 tests simultaneously. Unfortunately, he later discovered that the results were useless because visitors saw multiple experiences from overlapping tests. He had no way to draw conclusions from the results.

When it comes to testing velocity, Steve believes in running as many tests as it takes to positively impact the customer and the business. That could be one test per year, or hundreds. He currently launches six to 15 new tests per month, a number he expects will grow significantly as the program begins serving additional divisions in Thomson Reuters.

Tip 7: Test the limits by trying to break things

Although it sounds counterintuitive, trying to technically break things on the website through testing gave Steve confidence about what tests he could and couldn’t run without causing issues, or which tests needed to allow time to build out something differently to use Adobe Target on it. For example, if he set up a test experience on part of a page, he’d test to make sure the page would still load, that scripts would fire, and that tracking would work. He recommends really digging into the tool to find out its capabilities, looking at how your website is built, and figuring out how the two can comingle.

He strongly cautions that when trying to break things, always ensure those experiences are only seen internally. He also advises communicating internally what you’re experimenting with and where so that team members and stakeholders aren’t alarmed when the site looks or functions differently. He learned that lesson the hard way when he ran an internal test that put images of Smurfs and Judge Judy on the website, causing his manager a slight panic. Although he and his manager were the only people who could see the experiences, she didn’t know it!

Plans for the near future

As Steve steps into this new role, he’s excited to expand the program throughout the organization. He also sees great opportunity to use Adobe Target in new ways — to optimize their offline sales interactions, test external ads that drive people to the site, and deliver and measure the impact of highly personalized content suggestions. As always, we look forward to seeing what Steve does next.