Testing is not just a good idea, it is paramount to everything you do. Regardless of whether you are trying to build an audience, build circulation, change direction or change focus you need to test to make sure you head in the right direction.
However, you need to make sure your test will give you meaningful results and there are some rules that should be followed.
1) When testing, only test one element at a time. As soon as you put in another variable, the results of the test will no longer be valid. You will be tempted to test more than one element at a time, but if you do, whatever the results of the test are, you will not know which element triggered the result, so you will have learned nothing.
2) To ensure the results are statistically valid, you need to test at least 5,000 names; anything less will not give you a response that you can accurately project. In this day and age of virtually instant response from email blasts there is a great temptation to test “a couple of hundred names to see what happens.” This is not going to work and any results you get, and follow, will lead you in a false direction.
3) If your test consists of 5,000 names, then a response in excess of 50 to 100 orders is what you really need to be able to draw conclusions from that test.
4) When you conduct a test, you cannot assume the results of that test can be extrapolated across other platforms. For example, if you get a 10 percent response using one channel, you cannot assume a 10 percent response on another channel to the same market segment. The only real assumption you can make is that your promotion is worthy of a test across that platform as results suggest a positive outcome.
We all like to think we know our customers, but in a world that is changing faster than ever before, the chances of messing up are greater than they have ever been. Far better to mess up and annoy 5,000 people on a test than send an offer to your whole file and have it blow up in your face.
Many of us have met the editor who cannot understand why we need to send more than one renewal reminder. More and more magazines are now owned by companies that are not print-centric and the editors have been joined by other people of influence who also believe one renewal is enough. Are they right? Most of us know the answer to that one, but that is why we test—to prove or disprove ideas, both good and bad.