Creative testing in a physical network
Creative testing in digital media is cheap because the same audience can be split and the results arrive quickly. In a physical store network the split is by store rather than by person, the results arrive through sales data, and the test needs designing rather than switching on.
It is entirely possible and it is rarely done, which means most in-store creative is chosen on judgment alone.
How is the split made?
By store. One execution in one group of stores, another in a matched group, with assignment randomised within strata as with any store-level test. The comparison is between store groups rather than between individuals.
That means the test needs enough stores to detect a difference between two executions, which is usually a smaller difference than between running a campaign and not running one. Creative tests therefore need more stores than lift tests, which is the main reason they are avoided.
What can a creative test settle?
Which of two executions produced more of the measured outcome in the stores where it ran. That is a narrower claim than which creative is better, and it is the claim the design supports.
It cannot say why, and it cannot generalise far beyond the categories and stores tested. Treating a single creative test as a general rule about what works in the channel is the overreach to avoid.
What should be varied?
One thing, if the aim is to learn something transferable. Two executions differing in every respect produce a winner and no explanation, which is fine for a decision and useless for the next brief.
Varying a single element — the headline, the product shot, the call to action — produces a result that informs future work.
What is the practical alternative?
Sequential testing, where one execution runs and then another, with the comparison against the same stores over time. It is weaker because the periods differ, and it is considerably cheaper in stores.
Where the store count cannot support a parallel test, a sequential design with seasonal adjustment is better than no test at all, and it should be labelled as the weaker design that it is.
What about qualitative testing?
Showing executions to shoppers and asking is fast, cheap and answers a different question: comprehension and clarity rather than effect. For a medium with a two-second glance, comprehension testing is unusually valuable.
An execution nobody can read in two seconds does not need a sales test to be rejected.
How should results be used?
As one input to the next brief rather than as a rule. A winner in one category, one season and one store set is evidence, not a finding about the medium.
Accumulating several such results is what builds a genuine view of what works.
What should be agreed in advance?
The measure, the window, the store assignment and the decision rule. Without a decision rule the test produces a discussion, and the discussion is usually won by whoever preferred an execution beforehand.
Frequently asked questions
How many stores does a creative test need?
More than a lift test, because the difference between two executions is usually smaller than the difference between running and not running. Sizing it properly frequently reveals that the available estate cannot support it, which is worth knowing before spending.
Can I test more than two executions?
Only with enough stores for each arm, and the requirement grows quickly. Two arms is usually the practical limit in this channel.
Is comprehension testing worth doing?
Yes, and it is the cheapest useful test available. For a medium read in a glance, whether the message is understood at all is a prior question to whether it performs.
What outcome should be measured?
The one the campaign is for, agreed before the flight. Measuring sales for an awareness execution will show nothing and prove nothing.
Can the network run the test?
Many can handle the delivery split; the analysis should be agreed independently. A test designed and analysed entirely by the seller is a weaker basis for a decision.
How long should a creative test run?
The same considerations as any store-level test: long enough for power, with a pre-agreed window. Stopping when one execution is ahead inflates the false positive rate substantially.
Does creative performance transfer between store clusters?
Not reliably, which is an argument for testing within the clusters that matter rather than assuming a single winner.
What about testing localised creative?
It is one of the more interesting tests available in this channel and it needs the store groups to be comparable other than in the localisation. Matching on everything except the variable is the whole design.
What is the cheapest useful starting point?
Comprehension testing before flight and a sequential comparison afterwards. Together they cost little and they are considerably better than judgment alone. Industry standards work is discussed by bodies such as the IAB.
### How should creative learning be accumulated?
In a record of what was tested, in which categories and stores, and what the result was. A single test settles one comparison; a file of them starts to describe what works in the channel.
That record is also what stops the same comparison being run twice, which in a channel where tests are expensive is a real saving.
Can the network contribute creative learning?
It sees more campaigns than any single advertiser and its observations are not a controlled comparison. Treat them as hypotheses worth testing rather than as findings.
### How should a test be staged within a programme?
After the channel has shown an effect and before a large always-on commitment. Testing creative in a channel whose effect is unestablished confuses two questions and answers neither.
Once the channel is established, creative testing is where the remaining improvement sits, because store selection and weight have usually been optimised first.
Can two clusters be used as the two arms?
Only if the clusters are comparable, which is unlikely by construction since clusters are built to be different. Randomise within clusters instead.
How should the winner be used?
As the default with the runner-up retained for rotation. A winning execution run alone for a year will exhaust itself, which undoes the gain the test identified.
Preparing creative? See the NRS Digital Media creative specs or contact the team.