Measuring against a pre-period baseline
The most common measurement of an in-store campaign compares sales during the flight against sales before it. It is easy, it is available, and it is the weakest design in use, because everything else also changed between the two periods.
It can be strengthened considerably without a full experimental design, and the strengthening is worth doing.
What does a before-and-after comparison contain?
The campaign's effect plus seasonality, trend, competitor activity, distribution changes, weather and price movements. All of those move sales between periods and none of them are separated by the comparison.
That is why a before-and-after difference is consistent with an effect rather than evidence of one.
What is the cheapest improvement?
A comparison group. Stores that did not receive the campaign, over the same period, absorb the common factors and leave the difference attributable to the campaign.
That single addition converts the weakest design into a usable one, and in a store channel the comparison stores are frequently available at no extra cost.
What if no comparison group is possible?
Then the baseline should be modelled rather than taken as the prior period: adjusted for seasonality, trend and any known events. It remains weaker than a comparison and it is considerably better than a raw difference.
The model's assumptions should be stated, because they carry the whole conclusion.
How long should the pre-period be?
Long enough to establish a stable baseline and to capture the relevant seasonality, which usually means considerably longer than the flight. A one-week baseline before a four-week flight is measuring one week's noise.
It should also exclude any unusual events, which means looking at it rather than taking a fixed window mechanically.
What about the post-period?
It belongs in the measurement. Effects can persist and they can also reverse, and a window ending with the flight captures neither.
For anything with a promotional element, the post-period is where pantry effects appear, and omitting it overstates the net result systematically.
What should be checked before drawing a conclusion?
Distribution, price, screen uptime and competitor activity across both periods. Any of those moving is an alternative explanation, and checking them takes less time than defending a conclusion that ignored them.
How should the result be reported?
As a difference with its alternative explanations addressed and its uncertainty stated. A before-and-after figure presented as a campaign effect, with no comparison group and no checks, is a number rather than a finding.
Frequently asked questions
Can a before-and-after comparison ever be sufficient?
Where the effect is large, the period short, and the alternatives checked and excluded, it can be persuasive. It remains a weaker claim than a controlled comparison and should be described as such.
How do I pick a comparison group?
Stores that did not receive the campaign and that tracked the test stores closely before it. The pre-period parallel is the evidence that the comparison is valid, and it should be shown.
What if all stores got the campaign?
Then a geographic or temporal comparison is the fallback, both weaker. Planning a holdout before the flight is far easier than constructing a comparison afterwards.
Does a longer flight make this easier?
It gives more data and more opportunity for other things to change. Longer flights need more careful control of the alternatives rather than less.
Should I compare against the same period last year?
It controls seasonality and introduces a year of other changes. It is a useful supplementary comparison and a poor primary one.
How do I handle screen downtime?
Exclude or flag the affected stores. A store whose screen was dark for much of the flight is not a test store, and including it dilutes the measured effect.
What about stores that gained distribution during the flight?
They confound the measurement and should be identified. Distribution change is one of the most common alternative explanations in this channel.
Is a modelled baseline acceptable?
With its assumptions stated, yes, and it should be presented as a model output rather than as an observation. The method carries the conclusion.
What is the single strongest improvement?
Hold out some stores. It costs nothing but the foregone media in those stores and it changes the claim from suggestive to evidential.
### What should be written down before the flight?
The baseline period, the comparison group if any, the primary measure, the window and the checks to be run. Five items, and agreeing them in advance is what prevents the analysis being shaped by the result.
It also makes the analysis faster, because the decisions that usually take a week of discussion have already been made.
How should the baseline be shown?
As a chart covering the pre-period, the flight and the post-period, for both groups where a comparison exists. A reader can see the parallelism, the effect and the aftermath in one view, which no table conveys as well.
### What should be done when the result is surprising?
Check the alternatives before celebrating or dismissing it. A surprisingly large effect is as likely to indicate a confound as a success, and the checks are the same either way.
Distribution, price, competitor activity and delivery are the four to examine, and an unexplained surprise in any of them is the finding rather than the headline number.
Should the baseline be recomputed after the flight?
Recomputing it with the flight period excluded is the standard approach, and changing the baseline definition after seeing the result is not. Fix the method in advance.
Is a year-on-year comparison a valid baseline?
It controls seasonality and introduces a year of other changes. Useful as a supplementary view and weak as the only one.
How long should the post-period be?
Comparable to the flight where carryover or pantry effects are plausible. A window ending with the flight systematically flatters any campaign with a promotional element. General measurement practice is discussed by bodies such as Geopath.
Should the baseline exclude unusual weeks?
Yes, with the exclusions stated in advance. A baseline containing an unusual week is a baseline that will make the flight look better or worse for no campaign-related reason.