The short answer
Incrementality testing is one of the most direct ways to validate cause and effect in marketing but a large share of tests fail not because of the math, but because of weak design, executional drift, or results that can't be acted on. Before launching a test, ask your provider how they control for contamination, what quality checks govern execution, whether the hypothesis and success metric are precisely defined, how results feed into ongoing decisions, and whether lag effects and cross-channel halo are accounted for. The answers tell you whether you'll get causal insight or just an expensive snapshot.
Incrementality testing has become a standard tool for scaled retail eCommerce brands that want causal validation - the strongest available evidence that a channel or campaign is actually driving growth, not just correlating with it. The methodology is sound. The problem is that a significant share of test failures happen before the first impression is served: in the design, the execution, or the way results are interpreted and used.
These five questions are the ones worth asking any provider before you commit time and budget.
1. How do you prevent contamination and external factors from biasing results?
Geo-based holdout tests are powerful, but they sit in the real world. People commute between regions. Platforms don't always respect geographic boundaries. Local events - a competitor promotion, a stockout, a regional news cycle - can shift performance in ways that have nothing to do with your test.
A provider worth working with will document how they handle this, not just assert that their methodology is robust.
What good looks like: Test and control geographies are selected with spillover in mind - ideally using mobility-aware groupings that account for commuting patterns and cross-region exposure. There's a pre-period fit analysis showing that test and control regions behaved similarly before the test launched. Local shocks are monitored throughout the run, with pre-defined rules for exclusion or adjustment if something material happens. Contamination diagnostics are shared as part of the results, not buried.
Red flags: Generic geo lists with no evidence of spillover checks. No record of concurrent campaigns or local disruptions. No documentation of how the control group was validated. If these controls aren't in place, treat reported lift as directional rather than definitive.
2. What are your quality-control processes for execution?
Executional drift is one of the most common reasons test results can't be trusted. Incorrect dates, missing audience suppressions, overlapping tests in the same region, parameters changed mid-run without logging - any of these can compromise validity before the analysis even begins.
What good looks like: A pre-flight checklist reviewed and signed off before launch, covering audience definitions, spend caps, exclusions, and start dates. A timestamped runbook that logs every campaign change, creative swap, outage, and promotion during the test window. Account- or campaign-level exclusions to prevent platform bleed, verified against delivery data. Agreed thresholds for health checks and interim reviews, including criteria for extending or re-running the test if core assumptions break.
Red flags: No written setup documentation. Overlapping tests within the same region or audience. Mid-test parameter changes with no change log. Without operational discipline, even strong analytical frameworks can't recover validity after the fact.
3. Is the hypothesis single and specific, and does the success metric match it?
Every robust test starts with one causal hypothesis and one primary success metric. When those aren't locked in before launch, results can look statistically significant while offering nothing actionable.
What good looks like: A single primary hypothesis stated precisely - for example, "increasing prospecting spend in the US will lift new-customer acquisition" - with one primary metric (incremental new-customer lift) and secondary metrics for context (branded search, assisted conversions). For awareness tests, the primary metric should reflect the funnel stage being tested: brand-lift or awareness delta, not iROAS. Power analysis, duration planning, and stop rules are defined upfront, not adjusted after the data comes in.
Red flags: Undefined or shifting hypotheses. iROAS used as the sole KPI for upper-funnel campaigns. No evidence of power or duration planning. When the hypothesis and metric don't match the funnel stage being tested, the resulting lift number describes something real, but not what the test was designed to measure.
4. How do results feed into ongoing decisions, not just a one-time report?
Incrementality tests are snapshots. They show how marketing performed under specific conditions during a specific window. That's genuinely useful - but only if the learning carries forward. A lift report delivered at the end of a test and never integrated into planning is an expensive way to answer a question once.
What good looks like: Test results calibrate an always-on measurement model, so learnings remain relevant beyond the experiment window rather than aging immediately. The provider can show how lift results translate into concrete budget and channel decisions, connecting the test directly to day-to-day optimization. Results are layered with other sources - MMM trends, platform attribution - so marketing and finance share a single view rather than arguing from separate data sets.
Red flags: The engagement ends at "here's your lift report." No plan for integrating learnings into always-on systems. Tests running in isolation with no connection to the broader measurement mix. A good test explains what worked; a good system makes sure that explanation actually changes what happens next.
5. How do you account for lag effects and cross-channel halo?
Campaign impact rarely stops at the edge of a test window or a single channel. A Meta campaign may influence branded search or Amazon sales weeks after the test concludes. A YouTube burst can drive consideration that converts elsewhere, through a different channel, at a different time. If those effects aren't captured, the test can materially under- or over-estimate true incremental value.
What good looks like: Sufficient read windows to capture delayed conversions, or decay models that adjust for adstock - how the effect of spend carries over and fades across time. Correlated movement across search, direct, and marketplace channels is measured where feasible, with controls to avoid double-counting. For awareness campaigns, brand-lift or sentiment data is connected to downstream sales signals to show the full-funnel arc.
Red flags: Declaring results after very short windows. No mention of adstock or lag assumptions. Ignoring branded search or marketplace spillover entirely. When lag and halo go unaccounted for, test results describe a fragment of the impact rather than the whole.
How Fospha thinks about incrementality
Fospha is not an incrementality test provider - it is the always-on measurement layer that makes test results more useful by integrating them into a continuously updating model. Incrementality testing and always-on measurement answer different questions. Tests provide causal validation - a rigorous, point-in-time read of whether a channel drove incremental outcomes. Always-on Daily MMM provides continuous, daily signal across the full channel mix. Both matter, and they are most powerful when they work together.
Fospha's approach is to treat incrementality tests as calibration inputs, not isolated reports. When test results are available, they feed directly into the model - sharpening estimates of lag and halo effects and improving the accuracy of daily forecasts. The result is that causal learnings from a single test don't sit in a slide deck: they compound into a continuously improving view of what is driving growth.
That's the difference between a test that validates and a system that learns.
Common questions
Q: How long should an incrementality test run?
Duration depends on the channel, the funnel stage being tested, and the conversion volume in your control group. Tests that end too early frequently lack statistical power and produce unreliable lift estimates. For lower-funnel direct-response tests, two to four weeks is a common minimum. For upper-funnel awareness tests where lag effects are significant, longer windows or post-period read extensions are often necessary. Your provider should produce a power analysis before launch that specifies the minimum detectable effect and the duration required to achieve it.
Q: Can you run multiple incrementality tests at the same time?
Yes, but with care. Overlapping tests in the same geography or against the same audience introduce contamination risk - if both tests are running in the same regions, results from each can be biased by the other. Providers should have clear protocols for isolating concurrent tests, including geographic separation and audience exclusions. If your provider can't explain how overlapping tests are managed, run them sequentially rather than simultaneously.
Q: What's the difference between a geo holdout test and a conversion lift study?
A geo holdout test withholds advertising from a defined geographic region and measures the difference in outcomes between the held-out region and regions where advertising ran normally. A conversion lift study, run natively within platforms like Meta, withholds ads from a randomly selected audience segment rather than a geography. Geo holdouts tend to be more conservative and harder to game, but require sufficient geographic variation. Platform-native lift studies are easier to run but rely on the platform's own infrastructure, which introduces some dependency on the platform's methodology and reporting.
Q: What makes an incrementality test result "trustworthy" enough to act on?
Statistical significance is necessary but not sufficient. A result worth acting on also has: a pre-registered hypothesis and metric that match the funnel stage being tested; documented contamination controls; a clean execution log with no undisclosed mid-test changes; and a read window long enough to capture lag effects. If any of these are missing, treat the result as a directional signal rather than a definitive finding, and build the gap into how confidently you apply it to budget decisions.
Related reading
- What is incrementality testing and when should you use it?
- How does click measurement differ from impression measurement?
- What is a Daily MMM and why does measurement cadence matter?
See how Fospha works in practice. 30-minute walkthrough on your data, your channels, this quarter.
