Lies, Damned Lies, and Statistics
Posted on Wed, Sep 29, 2010
By Brandon Hemel, Director of Analytics
Mark Twain attributed the above phrase to Benjamin Disraeli. Whether or not he ever actually said it is irrelevant. BUT whoever did say it definitely had issues with using numbers to prove their point.
Since I spend the majority of my waking hours (and some of my dreams) thinking about numbers, I can truly understand both Twain’s and Disraeli’s anxiety regarding Statistical Analysis.
However, as direct response marketing professionals (I am including you, dear reader, in this illustrious fraternity), we sometimes rely on faith that our tests will be statistically valid and our margin of error will be low.
Yet as a confirmed agnostic, I know faith can only get you so far, so here are some quick pointers regarding how make sure your results are valid and repeatable.
1. Figure out what you are trying to test before you test it. If you are looking to increase average gift, and yet your results show only a slight variance in gift amounts, but high variance in response rate – something else happened and your test may be invalid.
2. You need enough people in your control and test (and an equal number of people, too!) to lower deviation to within an acceptable number at the highest confidence level possible. If you have a huge statistical deviation in response rate – you should think about a retest.
3. Regarding #2, is your control and test groups a random sample of your overall population? If your universe was arranged in some arbitrary way (alphabetical by name, chronologically by gift date, ranking order by lifetime value, the type of diet soda they prefer, etc.) and then you took a slice of the data, this isn’t random.
4. Give people enough time to respond and the returns to be processed before you declare success or failure. Beware of impulsively shouting “EUREKA!” when your test wins or rashly committing Hari-Kari over horrible results. Sometimes you need to be patient and wait for the campaign to be fully complete.
5. Were there external variables that affected your test? What if a world-wide spam outbreak made people wary of opening your email? What if the truck with your test packages got stuck in the snow and entered the mail stream five days later than your control?
Finally, after all of your testing and analysis are done, be sure to properly word your results. You can never be 100% sure 100% of the time. Just because something was a winner once, doesn’t mean it will win again.
These five items are not the be all and end all to ensuring validity to your tests – what else can you do to prevent your Statistics from becoming Damning Lies?
