Multivariate Testing Rules of Thumb

In my previous post, I hope I helped you understand the value of multivariate testing (MVT). I even asked you to think of a page on your site that could benefit from it. But before you launch your first test, I’d like to suggest three general rules to follow:

  1. Make sure you have enough traffic

MVT requires more traffic than an A/B test, which probably makes sense given that instead of testing two or three experiences, you’re often testing all the possible combinations of the elements. (I’ll explain why I said “often” next.) This can add up quickly to many experiences.

In our example from the previous post, which is shown below, we wanted to test three headlines, two videos, four information boxes, and two call to action (CTA) buttons in a particular page.


If you run a multivariate test, you’ll have to test 48 possible different experiences:

3 Headlines X 2 Videos X 4 Info Boxes X 2 CTA Buttons = 48 Combinations

That also means that you have to split traffic for testing among 48 separate experiences, which requires a lot of traffic to gain meaningful insight. It also requires a lot of time to get that traffic for most pages. The takeaway here is that you have to be reasonable about the number of locations and offers within those locations that you’re testing. Don’t go overboard; the sample size calculator embedded within the Adobe Target Premium or Standard MVT workflow will help you estimate an acceptable number of experiences based on your traffic level.

  1. Verify compatibility of the options for the elements you combine

I said that you often test all possible combinations of the locations and their offers. Sometimes, however, not all elements are compatible. Verify that the individual elements that you are testing combine to form experiences that make sense. For example, a headline that says “Free shipping” does not go well with a call to action button that says “Save $2 on shipping.” Also, if you have the same offer as a candidate in several locations you would want to make sure that that same offer is not shown in more than one location at a time.

In Adobe Target, you can view all the possible combinations and eliminate those that don’t make sense to test before launching the test.

  1. Use MVT in the right places

A/B testing handles testing larger layout changes better than MVT. Use MVT for testing options for elements within a layout. Some of the most common elements that marketers test with MVT include:

In my final post on MVT, I’ll debunk a couple of misconceptions about this optimization tactic.