Incrementality Testing: What it is and how it fits into your measurement system

When to run tests, when to rely on models, and how to combine both.

We have previously covered the three foundational approaches to marketing measurement and how to triangulate between them. This lesson goes deeper on one of those approaches: incrementality testing.

What does "incrementality" mean?

Incrementality is the extra revenue, conversions, or outcomes that a specific marketing activity actually causes, not what it happens to correlate with.

Neither MTA nor MMM can fully isolate the causal impact of a single factor from everything else happening at the same time. That's what makes incrementality tests the gold standard for proving cause and effect: they use controlled experiments (test group vs. control group) to measure it directly.

There are good reasons you can't rely on tests alone.

What tests can and can't tell you

Incrementality tests are the most conclusive form of measurement, but they can't be your only tool:

- They need large audiences, long run times, and significant technical expertise to set up correctly.

- Most tests require a holdout group that doesn't see your marketing, sacrificing potential revenue while the test runs. Others, like spend-lift studies, increase spend in specific markets instead, a budget cost rather than a revenue cost.

- Each test only answers one specific question, for that channel, that time period, that audience. You can't extrapolate to your whole media mix.

- A test measures the lift at one spend level. Scale the budget significantly and the return might not hold.

- Tests don't run cleanly in parallel. Two tests covering overlapping audiences or channels might contaminate each other's results, so teams typically run them one at a time, slowing how fast they learn.

Most importantly, a test delivers a snapshot rather than a continuous read. Many brands are making today's decisions based on test results from last year. That's the core limitation: high confidence for one question, no picture of how everything fits together over time.

Tests are an investment in insight, not an always-on measurement system. Before we look at how to make the most of them, here's what's available.

What are the different test types?

All tests follow the same principle: compare a test group (exposed to the activity) to a control group (not exposed). The method used to divide the audience determines the test type.

User-based experiments (Randomised Control Trials)

Individual users are randomly sorted into test and control groups before the campaign begins. This works best inside platforms that can track users at the individual level. Common types include:

Market-based experiments

Used when you can't track individual users, or when the activity affects entire regions (like TV or radio). Entire geographic markets are treated as test or control groups. Common types include:

Calibration: from snapshot to system

Incrementality tests are most useful when the result doesn't just live in a slide deck. The real lift in value comes from feeding those results back into your always-on measurement.

This is calibration: using the measured incremental lift to correct assumptions in your Daily MMM. For example:

- Your Daily MMM says "invest more in TV" but can't tell you which network. A geo-lift test on Network A vs. Network B gives you the causal proof to shift budget confidently.

- Your MTA says Paid Social has a £6 ROAS. Your Daily MMM says £3. A conversion lift study reveals the true incremental ROAS is £4, and you use that to recalibrate both models.

Without calibration, every test only answers one question and then sits in a slide deck. With calibration, every test permanently improves the accuracy of your entire measurement system. The results compound rather than decay.

The continuous learning loop

Traditional MMMs require a complete re-run to incorporate new test results, which means you might wait months before the model reflects what you just proved. Modern platforms solve this by accepting test results as inputs that guide the model's learning without requiring a full rebuild. When an incrementality test delivers a proven result, the model absorbs it and improves its accuracy across all channels, not just the one you tested.

That's the difference between episodic testing and a continuous system. Design the test, run it, ingest the results, and use them to calibrate the model. Each test makes the next recommendation sharper.

In action

CarParts put in $223k of additional Meta spend based on Fospha's recommendations. They got $3.1M back in incremental revenue, validated through a live conversion lift study. That's what it looks like when always-on measurement and incrementality testing work together.

Making your tests go further

Incrementality tests give you the most conclusive evidence of marketing impact. But their cost, speed, and narrow scope mean they work best as a strategic tool rather than an always-on system. Run tests where it matters most, then use calibration to make those results improve your entire measurement system over time.

In the next lesson, we'll look at how measurement can be extended beyond your own website to capture the full impact of your marketing, including marketplace and retail channels.

‍