Sample size for a store-level media test
The most common outcome of an in-store media test is an inconclusive result read as a negative one. That outcome is usually decided before the test runs, by a store count chosen for convenience rather than by a calculation.
Sizing the test is the step that converts it from an exercise into an answer.
What determines the required number?
Three inputs: the variability of store sales in the category, the smallest effect that would change a decision, and the confidence required. Those three produce a number, and the number is frequently larger than expected.
Each input is available. Variability comes from historical store data, the effect size from the commercial decision, and the confidence level from convention.
Why is store sales variability so important?
Because it is the noise the effect has to be detected against. A category with wide store-to-store variation needs many more stores than a stable one to detect the same proportional effect.
Small-format retail is variable, which means tests here generally need more stores than an equivalent test in larger, more homogeneous outlets.
What effect size should be used?
The smallest one that would change what you do. If an effect below a certain size would not justify continuing the programme, the test does not need to detect anything smaller.
That framing usually reduces the required sample, because businesses can act on a larger effect than the smallest one that is statistically interesting.
How does the flight length interact?
Longer flights accumulate more transactions per store, which reduces noise in the measured outcome. A longer test with fewer stores and a shorter test with more can have similar power.
That trade is worth calculating, because store availability and flight length have different costs.
What happens when the sample is too small?
The test produces a point estimate with an interval wide enough to include both a meaningful effect and no effect. That result is honestly reported as inconclusive and is routinely reported as a failure.
Knowing the detectable effect size in advance prevents that, because it tells everybody what the test can and cannot settle.
Can a pilot substitute for a powered test?
A pilot can establish feasibility, execution quality and data availability. It cannot settle the effect size unless it happens to be powered, and it should not be presented as though it could.
Running a pilot to learn how to run the real test is a reasonable use of one.
What should be reported alongside the result?
The detectable effect size at the achieved sample. A null result accompanied by that figure is interpretable; one without it is not.
What if the store count is fixed?
Then compute what effect the test can detect and decide whether that is useful. If the detectable effect is larger than any plausible campaign effect, the honest conclusion is that the test should not run.
That conversation is uncomfortable and considerably cheaper than running it anyway.
Frequently asked questions
How many stores does a typical test need?
More than intuition suggests, and the number depends entirely on category variability and the effect being sought. Any figure quoted without those inputs is a guess, and it is usually too low.
Can I increase power without more stores?
Yes, by running longer, by using a more precise outcome measure, or by matching stores more closely to reduce baseline variation. All three are worth considering before concluding a test is impossible.
Does a bigger effect need fewer stores?
Considerably fewer, which is why the effect size input matters so much. Sizing for the smallest interesting effect rather than the smallest actionable one inflates the requirement.
What confidence level should be used?
Conventional levels are a starting point and the right answer depends on the consequences of each kind of error. A test informing a large irreversible commitment deserves more stringency than one informing a small reversible one.
Should the calculation be documented?
Yes, before the test, with the inputs. It is the evidence that the design was deliberate and it is what allows a null result to be interpreted correctly.
What if power is achievable only with the whole estate?
Then a staged or matched-market design may be the alternative, or the question may need a different measurement approach. Running an underpowered version is the option that produces no information.
Does clustering assignment reduce power?
Yes, because stores within a cluster are correlated, so the effective sample is smaller than the store count. Geographic clustering to control spillover has this cost and it should be in the calculation.
Can I pool several tests?
Where they are comparable, pooling increases power and it requires consistent design and measurement. Planning for it in advance is what makes it possible later.
What is the single most useful output of the calculation?
The detectable effect size. It tells everybody what question the test can answer before anybody has spent anything. General guidance on statistical significance and its limits is worth reading alongside it.
### What if the calculation says the test is not feasible?
Then say so before spending. A test that cannot detect a commercially relevant effect produces an ambiguous result that will be read as a negative one, which is worse than not testing.
The alternatives are a longer flight, a more precise measure, tighter matching, or accepting a larger detectable effect as the question being asked.
Should the calculation be shared with the network?
Usually yes, because it determines the store count being bought and a network that understands the requirement can help meet it. It also sets a shared expectation about what the test will and will not settle.
### How should the calculation be documented?
With the variability estimate, its source, the assumed effect, the confidence level and the resulting store count. Five lines, attached to the test plan.
That record is what allows a null result to be interpreted a year later, when nobody remembers what the test was capable of detecting.
Can the variability estimate be borrowed?
From a comparable category in the same estate, at a stretch, and it should be labelled as borrowed. Using a figure from a different channel or format will usually be wrong in an unknown direction.
Does a matched design reduce the requirement?
Yes, because matching reduces baseline variation between groups. It is one of the cheapest ways to improve power without adding stores.
Considering NRS Digital Media? Start at nrsdigitalmedia.com, read the media deck, or contact the team.