Reading a lift result that came back flat
A test that shows no measurable effect is the most common result and the most commonly misread. Flat can mean the campaign had no effect; it can also mean the test could not detect one, the campaign did not run as planned, or the measurement was confounded.
Distinguishing between those is what makes the result useful.
What is the first thing to check?
Whether the campaign ran. Screen uptime, delivered plays against plan, and whether the product was actually available in the test stores. Any of those failing produces a flat result for reasons unrelated to the creative or the channel.
This check is quick and it resolves a meaningful share of flat results.
What is the second?
Whether the test could have detected an effect. The detectable effect size at the achieved sample answers it, and if that figure is larger than any plausible campaign effect, the test was never capable of a positive result.
A null from an underpowered test is uninformative rather than negative, and reporting it as a failure is a mistake with consequences for the next decision.
What confounds could flatten a real effect?
Spillover between test and control stores, a competitor campaign running simultaneously, a distribution change, or a price move in one group. Each pushes the measured difference toward zero.
Spillover is the most common in dense areas and the easiest to design around in advance.
Could the creative be the cause?
Possibly, and it is the hardest to establish from a single flat test. A creative problem and a channel problem look identical in the result, which is why creative variation within a test is worth building in where the sample allows.
What about the outcome measure?
A campaign aimed at awareness measured on immediate sales will frequently show nothing, because the measure does not correspond to the objective. That mismatch is decided at the design stage and discovered at the result.
Agreeing the measure against the objective before the flight prevents it.
What should be concluded from a well-run flat test?
That the campaign did not produce an effect large enough to detect with this design, in this category, over this window. That is a real and useful finding, and it is narrower than "in-store media does not work."
Stating it precisely is what keeps it usable.
What should happen next?
Either a redesigned test addressing the identified weakness, or a decision that the channel is not a priority for this category. Both are legitimate; what is not useful is repeating the same design and expecting a different answer.
How should a flat result be communicated?
With the checks that were run, the detectable effect size, and the specific conclusion. A flat result presented without those invites an interpretation far broader than the evidence supports.
Frequently asked questions
Does a flat result mean the channel does not work?
No. It means this campaign, in this category, at this weight, over this window, did not produce a detectable effect. Generalising beyond that is unsupported by a single test.
Should I run the test again?
Only with a change: more power, better compliance, a different measure or different creative. Repeating an identical design is likely to produce an identical result.
How often are flat results caused by execution?
Frequently enough that compliance should always be checked first. A campaign delivered at a fraction of plan in a meaningful share of stores is a delivery finding rather than an effectiveness one.
Could the effect be real but delayed?
For awareness objectives, plausibly, which is why a post-period matters and why sales may be the wrong measure entirely. A measure matched to the objective is the fix.
What if the control group outperformed?
Check the groups' comparability before the flight. A control that was always going to grow faster was never a control, and the pre-period chart shows it.
Is a small positive result the same as flat?
If the interval includes zero, statistically yes, and the point estimate still carries information for a decision about risk. Reporting both is more useful than declaring it null.
Should a flat result be shared internally?
Yes. A programme that reports only its successes teaches nothing and its credibility collapses the first time somebody checks.
Does a flat result argue for more weight?
Not by itself. More weight against an untested creative in a possibly mismatched measure repeats the uncertainty at higher cost.
What is the most valuable thing to take from it?
The list of what to change. A flat result with a diagnosed cause is worth more than a positive one with none.
### How should a flat result change the next plan?
By addressing the specific weakness identified rather than by increasing weight. More budget against the same design usually produces a more expensive version of the same ambiguity.
If the diagnosis is compliance, fix delivery. If it is power, add stores or length. If it is the measure, change it. Each points at a different next step.
Should a flat result be shared with the network?
Yes, with the diagnosis. A network that learns its delivery was short has something to fix, and one that is told only that the campaign did not work has nothing to act on.
### What should be recorded about a flat test?
The design, the compliance data, the detectable effect size and the diagnosis. Four elements, and together they make the result useful to whoever plans the next one.
Without them, a flat result becomes a remembered impression that the channel did not work, which is both less precise and more damaging than the actual finding.
How should it be communicated upward?
As a specific finding with a next step, not as a verdict on the channel. A result reported without its scope will be generalised by whoever hears it.
Is a flat result ever good news?
It is when it prevents a larger commitment to something that does not work, which is the whole purpose of testing before scaling.
Should the network be asked for its view?
Yes, because it sees delivery detail the buyer may not. Its interpretation is an input rather than a verdict, and it frequently identifies a delivery cause quickly. General measurement practice is discussed by bodies such as Geopath.
How should the next test differ?
In the specific dimension the diagnosis identified. Repeating the same design at higher weight is the most common and least informative response.