Testing: When to use What

Dutch HollisSo, you’ve decided you’d like to optimize your homepage (or email campaign, or PPC creative, etc.) and you’re not sure where to start or what your test should look like. Perhaps you want to use Multivariate testing because you need it, or because you want it. Or perhaps you are going to use A/B testing because it’s what you know.

Neither approach is wrong here, but neither is an informed, defensible approach, either. Like many things in life these are both valid solutions for particular jobs. Sometimes you need a 3-wood, sometimes a putter. Sometimes a claw hammer will do, sometimes you need a sledge. So here are some brief scenarios and tips to help you pick the right tool.

When you want to test a single variable: A/B or A/B-N testing is excellent at testing one particular aspect of your creative – email subject lines, for example. In fact, A/B-N testing is valid only when you are testing exactly one variable of your creative. When you test more than one variable of the creative, you are creating an invalid test with unreliable results, despite the fact that it may “feel” exactly the same in process.

When you need speed to market: With the caveat above, A/B or A/B-N tests can be very quick and easy to setup. Test design is simple. Create 1, 2 or “N” versions and test each over equally distributed, random sets of your total population. For web content you may want to consider the statistical significance of the differences you see before declaring a winner, while email decisions are more often driven by time constraints.

When you want to test multiple variables: As mentioned above, A/B testing is invalid over multiple variables. However, this is what multivariate testing is designed to handle. So test your model photography, and the color scheme, and the headline, and the offer…all at once. Multivariate testing allows you to get smarter about several factors at the same time.

When you want to know the interaction effects: This is another situation where multivariate testing shines. Not only can you test multiple variables with one effort, but you can gauge the effects the variables have on each other. Even with multiple A/B tests for multiple variables you’ll never get to see the interaction effects that are caused by changing multiple factors at the same time. Additionally, with the right software and test design, you can avoid testing all variations equally while concentrating on those combinations that are most likely to yield results.

Happy testing.

One Trackback/Pingback

  1. […] definition A/B/N tests need to be kept simple (see earlier discussion on when to use multivariate testing). Remember that you are simply testing one variable and can’t know how multiple variables will […]

%d bloggers like this: