Holdout stores in an in-store media test
A holdout is a set of stores deliberately excluded from a campaign so that their sales provide a comparison. In a channel where activation is store by store, holdouts are unusually easy to arrange and unusually rarely used.
They are the difference between a result that is evidence and one that is consistent with several explanations.
How should holdout stores be chosen?
Randomly, within groups of similar stores. Stratify by the characteristics that matter — traffic, category performance, region, format — and randomise within each stratum.
Choosing holdouts because they are less important, or because they were awkward to activate, builds a difference into the comparison before the campaign starts.
How many stores should be held out?
Enough that the comparison has power to detect the effect that would change a decision. That is a calculation from the variability of store sales and the expected effect size, not a proportion picked by convention.
Holding out too few produces a comparison that cannot distinguish a real effect from noise, which is the most common failure in these tests.
What invalidates a holdout?
Spillover, where shoppers visit both test and holdout stores. In dense urban areas with overlapping catchments this is a genuine risk, and it biases the measured effect toward zero.
The mitigation is to hold out by geographic cluster rather than by individual store, at the cost of needing more stores for the same statistical power.
What else has to be held constant?
Everything other than the in-store media: price, promotion, distribution and any other marketing activity in those stores. A holdout that received a different price is not a holdout.
That requires coordination with the trade and promotions teams, which is the practical reason holdouts do not happen more often.
What should be checked before analysis?
Pre-period parallelism between the groups, screen uptime in the test stores, distribution in both, and whether anything unusual occurred. Four checks, and each can invalidate the comparison.
Pre-period parallelism is the most important. Groups that diverged before the campaign were never comparable.
What is the cost of a holdout?
The foregone media in those stores, which is a small share of a campaign. Against that is the difference between knowing whether the campaign worked and believing it did.
That trade is usually favourable and it is routinely declined because the holdout looks like wasted budget rather than like a measurement cost.
How should results be presented?
Effect size with a confidence interval, the pre-period chart, and the compliance data. Those three together are what allow somebody else to judge the result.
Frequently asked questions
How large should the holdout be?
Enough for power, which is a calculation rather than a convention. A common failure is holding out a token number of stores and then finding the comparison cannot detect anything.
Should holdouts be geographically separate?
Where catchments overlap, yes, because spillover biases the result. Clustering the assignment geographically is the standard mitigation and it requires more stores overall.
Can I hold out by daypart instead?
It is possible and it creates a different confound, since the dayparts have different audiences. Store-level assignment is cleaner where it is available.
What if the sales team objects?
The objection is usually that holdout stores are disadvantaged, which is true for the flight and small in scale. Framing it as a measurement cost with a stated duration usually resolves it.
Do the holdouts get the campaign afterwards?
Commonly yes, once the test period ends, which makes the arrangement easier to agree. That should be planned rather than improvised.
How long should the test run?
The full flight plus a post period, with the analysis window fixed in advance. Stopping when the result looks good substantially inflates the chance of a false positive.
What about stores that were meant to be held out and got the campaign?
Record it and analyse as assigned where possible. Analysing only the compliant stores reintroduces selection.
Does this work for a small brand?
Yes, if the store count supports the power calculation. A brand in very few stores may not be able to run a powered test, and knowing that in advance is better than running one anyway.
What is the single most valuable element?
The pre-period parallelism chart. It is the evidence that the comparison means anything, and it takes minutes to produce. Measurement practice in out-of-home is discussed by bodies such as Geopath.
### How should holdouts be explained internally?
As a measurement cost with a stated duration and a stated benefit. Framed as lost sales it will be resisted; framed as the price of knowing whether the programme works it usually is not.
It helps to commit in advance to giving the holdout stores the campaign afterwards, which removes most of the commercial objection.
Can a holdout be reused across flights?
Rotating which stores are held out is generally better, so that no store is persistently disadvantaged and so that the comparison is not always built on the same subset. Rotation should be planned rather than improvised.
### What should be checked in the holdout stores?
That they received nothing: no screens live, no other campaign activity, no different price or promotion. A holdout contaminated by other activity is not a holdout.
Verifying that is a compliance check in the holdout as much as in the test group, and it is usually skipped because the holdout is assumed to be passive.
How should the holdout be chosen for a small estate?
By stratified random assignment, which works at any size, with the power calculation deciding whether the estate supports a test at all.
Can the network be trusted to withhold?
It is a delivery instruction like any other and it should be verified from playout logs. A holdout store that received the campaign is a serious contamination.
Considering NRS Digital Media? Start at nrsdigitalmedia.com, read the media deck, or contact the team.