What Is Incrementality Testing? How to Measure the Real Lift From Your Ads

Your dashboard shows how many users signed up after interacting with your ads, but incrementality testing shows how many signed up because of them.

When I spoke with Emmanuel Ekokotu, CMO at Blockchain-Ads, about campaign impact, he explained:

“Last click gives all the credit to the ad a user clicked right before converting. That’s usually retargeting or a branded search, sitting at the end of a journey other ads started.”

An incrementality test helps establish how many additional conversions your advertising generated. But for the results to mean anything, you need to track a conversion event that matters to your business and count it once per user.

This guide covers how to calculate lift, run holdout tests, measure incremental acquisition costs, and use the results to make budget decisions.

Is Your Dashboard Overcounting Conversions From Your Ads?

Your dashboard may credit ads for conversions that would have happened anyway. For example, someone researching a crypto exchange may already intend to register. If they see your ad and later sign up, the dashboard may count that conversion even if the ad didn't cause it.

Incrementality in marketing is the number of conversions your advertising caused, beyond those that would have happened anyway. Incrementality testing measures the additional conversions your advertising caused by comparing two groups:

• Treatment group: Users eligible to receive the ads being tested.
• Control (holdout) group: Similar users excluded from the ads being tested.

Both groups should have a similar likelihood of converting before the test begins. By showing ads to one group and withholding them from the other, advertisers can estimate how many conversions happened because of the ads.

If the treatment group converts at a higher rate, the difference helps estimate incremental lift. Advertisers need reliable conversion tracking and enough data to confirm that the difference isn't due to chance.

Incrementality vs Attribution

Incrementality vs. attribution is about whether your ads caused a conversion or received credit for it. Attribution shows which marketing touchpoints get credit, while marketing incrementality measures how many additional conversions your ads generated.

Marko Rendevski, SVP Advertising Services & Partnerships at Blockchain-Ads, explains:

“Attribution tells advertisers which channel received credit for a conversion, but it does not prove that the advertising caused that conversion.”

For example, last-click attribution may credit a retargeting ad when an existing user clicks it and converts, even if that user would have converted without the ad. Similarly, platform-reported conversions show which conversions received credit under the platform's attribution rules, not how many the advertising caused.

Advertisers can use attribution to understand credited touchpoints and incrementality to determine which campaigns generate additional business.

3 Ways Incrementality Is Measured in Programmatic Advertising

The right incrementality measurement method depends on which users, markets, or campaign activity an advertiser can withhold, the available conversion data, and the decision the test needs to inform. Three approaches are:

Hold Back People With a Holdout Test

As explained earlier, a holdout test randomly splits eligible users into a treatment group that can receive ads and a control group excluded from the campaign. Advertisers compare conversion rates between the groups to estimate incremental lift.

For example, a crypto exchange could randomly split its customer list and retarget only one group. A higher conversion rate in that group suggests incremental lift, provided the difference is statistically credible.

This method suits retargeting campaigns where advertisers can identify users, control campaign eligibility, and track conversions across both groups. A platform-managed conversion lift study, such as a Meta conversion lift study, can use the same randomized design, but its available controls and reporting depend on the platform.

To keep the results reliable:

• Randomize before launch: Assign users to groups before advertising begins.
• Prevent contamination: Keep control users out of the tested campaign and check for overlapping ads.
• Compare assigned groups: Include users according to their original assignment, even if they never received an impression, to avoid exposure-selection bias.
• Check statistical power: Ensure the test has enough users and conversions to detect a meaningful effect.

Hold Back Regions With a Geo Lift Test

A geo lift test runs ads in selected markets while withholding them in comparable markets.

For instance, a crypto exchange could advertise in one country and withhold ads in another with similar historical conversion trends. Historical and control-market data help estimate what the advertised country would have achieved without the campaign.

Three free, open-source tools can help plan and analyze these tests:

1. Meta GeoLift

• What it is: An R package for designing and measuring geographic lift experiments.
• How it works: Uses historical market data to select test markets, assess statistical power, and construct a synthetic control from weighted combinations of unaffected markets.
• Best used for: Checking whether a proposed geo test can detect lift and measuring the results afterward.

2. CausalImpact

• What it is: An R package for estimating the impact of a campaign launch, pause, or other intervention.
• How it works: Uses historical conversion data and unaffected control series to predict what would have happened without the change, then reports the estimated impact and uncertainty.
• Best used for: Analyzing campaign changes when suitable control data exists, including cases without randomized market assignments.

3. Meridian GeoX

• What it is: Google's open-source framework for geographic advertising experiments.
• How it works: Helps design and analyze holdback tests (withholding ads), go-dark tests (stopping existing spend), and heavy-up tests (increasing spend).
• Best used for: Comparing spending strategies and generating experimental results that can help calibrate a Marketing Mix Model (MMM).

These tools cannot compensate for poorly matched markets or advertising spillover into control markets. Check historical trends, conversion volume, and overlapping campaigns before relying on the lift estimate.

Hold Back Time by Pausing Spend

A spend-pause test stops advertising for a set period to measure changes in conversions. Unlike audience or geographic holdouts, it does not require splitting users or markets.

A simple before-and-after comparison is the least reliable approach because crypto price swings, sporting events, promotions, and seasonal demand can affect conversions independently of advertising. Users exposed before the pause may also convert during it.

To improve reliability:

• Plan the pause: Set dates in advance and avoid changing other campaigns during the test.
• Track external factors: Record market events, promotions, and seasonal changes that could influence conversions.
• Estimate the baseline: Use historical trends and unaffected control data to estimate conversions without the campaign.

How to Turn a Lift Result Into a Real Cost per Conversion

To understand how to calculate incrementality, start by estimating the additional conversions your ads generated. Divide campaign spend by that number to get incremental cost per acquisition (iCPA), the cost of each additional conversion rather than each attributed conversion.

For a randomized user-level holdout test, use this incrementality formula:

Worked Example: Cost per Incremental First Trade

Suppose a crypto exchange spends $5,000 on a retargeting campaign and tracks each user's first trade.

At the control group's 2% conversion rate, the campaign generated an estimated 100 additional first trades.

When I asked Emmanuel how advertisers should use these results, he explained:

“Turn the result into a cost per incremental conversion, and compare it with what a customer is worth to you. Take your spend and divide it by the extra conversions the group that saw your ads produced over the holdout. If that cost sits well under your customer lifetime value, scale. If it sits above, cut or rebuild.

Then test again before you double the budget. A result at today's spend only describes today's spend.”

Applying Emmanuel's approach, here's how the exchange's incremental CPA compares with its reported CPA:

The dashboard reports $20 per attributed first trade, but the experiment estimates $50 per additional first trade. Before scaling, the exchange should confirm that the lift is statistically credible and the acquisition cost makes economic sense. A result at $5,000 in spend doesn't guarantee the same efficiency at $10,000.

💡Blockchain-Ads' customer acquisition cost guide explains how to calculate acquisition costs and assess them against customer value.

How to Measure Incrementality on Blockchain-Ads

Advertisers can run two incrementality tests on Blockchain-Ads with setup support from their account manager. A country-level holdout measures prospecting campaigns, while a customer-list holdout measures retargeting campaigns. Here's how to get started:

Choose Your Conversion Event and Fire It Once per User

In my discussion with Emmanuel, he emphasized choosing the right conversion event:

“Test on the conversion event your campaign was built for, as long as it happens often enough to read inside your test window. Then check whether the people who saw your ads went on to pay.”

For example, a crypto exchange optimizing for sign-ups should also track first trades in both groups to measure whether those sign-ups lead to additional traders. Blockchain-Ads’ conversion tracking guide explains how to set up these events in HUB.

To configure server-to-server (S2S) conversion tracking with a third-party tracker in Blockchain-Ads HUB:

1. Go to “Advertising > Tracking > Conversions” in HUB.
2. Click “Create Conversions” and select "Server-to-server."
3. Name your conversion and click "Save."
4. Copy the postback URL using the code icon beside your conversion.
5. Add Blockchain-Ads as a traffic source in your tracker, using clickid as the Click ID parameter.
6. Paste the postback URL into your tracker.
7. Map {clickid} to your tracker's click ID macro.
8. Set your campaign's destination URL to the tracker link containing {clickid}.
9. Configure one postback per user for the chosen milestone, then test the conversion in HUB.

💡The S2S installation guide covers the complete setup, including optional conversion values and deduplication.

Record the same conversion event in your backend for both treatment and control groups. You'll need these records to count conversions from users who never received the tested ads.

Hold Out Matched Countries for a Prospecting Campaign

A prospecting holdout starts with countries that have similar historical conversion trends. Use a few months of data to identify comparable countries with sufficient conversion volume, then randomly assign them to treatment and control groups.

Begin measuring after the campaign's initial learning period, when delivery has stabilized. Keep learning-period data separate from the test results.

To target the treatment countries in Blockchain-Ads HUB:

1. Open “Audience & Targeting” and enable “Manual Targeting”.

2. Expand “Geo” and select “Enter another location”.

3. Click “Select Another Country” to open the country picker.
4. Search for your treatment countries or use “Quick Selection."
5. Select the treatment countries, leaving control countries unselected.

6. Review “Total Audience Size” and click “Continue”.

💡The GEO targeting guide covers country selection and exclusions.

Keep bids, budgets, and creatives consistent throughout the test. Prevent overlapping campaigns or promotions from affecting the control countries, as they can distort the lift estimate.

Split Your Own Customer List for a Retargeting Campaign

A retargeting holdout measures whether advertising to existing customers generates additional conversions. Randomly split your customer list into two equal groups, then advertise to one while withholding ads from the other.

To create the treatment audience in Blockchain-Ads HUB:

1. Prepare the treatment group's first names, last names, and email addresses.
2. Open “Audience & Targeting” and enable “Manual Targeting."
3. Expand “Create new segment” under "Segments."
4. Click “+ Create new segment” and build the audience with your data.

5. Upload the treatment list and save it as a custom audience.
6. Add the custom audience to your campaign, excluding the control group.

Blockchain-Ads matches uploaded contacts against profiles in its audience data. Aim for at least 1,000 matched users, not merely uploaded contacts. This doesn't guarantee enough conversions to detect statistically meaningful lift.

Compare conversions across both original groups using your backend records, including treatment users who never received an impression. Keep control users out of overlapping retargeting campaigns and promotions to prevent contamination.

Compare Conversions Against the Baseline

After either holdout test, pull campaign spend from the “GEO” tab on Blockchain-Ads' “Reports” page or the API's daily country breakdown. Collect conversion data for both groups from your backend, since the held-back group never appears in Blockchain-Ads reports.

Your incrementality analysis depends on the test:

• Prospecting: Compare conversions in advertised countries against an estimated baseline using historical trends and control-country data.
• Retargeting: Compare conversion rates across the original treatment and control groups, including users who never received an impression.

Calculate incremental CPA using campaign spend and estimated incremental conversions. Check the statistical reliability of the lift estimate and compare the cost against the economic value of each conversion before adjusting your budget.

💡Read our paid media budget allocation guide for a detailed breakdown.

Mistakes That Make Incrementality Measurements Unreliable

Even a well-planned incrementality test can produce misleading results. Watch for these four mistakes:

Counting Repeat Events as New Conversions

Duplicate postbacks can inflate conversions and skew lift. Deduplicate events using transaction IDs, and count first deposits or first trades only once per user in both groups.

Changing Settings or Stopping Early

Changing bids, budgets, or targeting mid-test can bias the results. Set the test duration and required sample size in advance, keep settings stable, and avoid stopping when early results look favorable.

Testing Through a Big Match Week or a Price Swing

Major sporting events can spike betting activity, while crypto price swings can affect trading conversions independently of your ads. Avoid these periods when possible, or account for their impact on treatment and control groups.

Letting the Held-Back Group See the Ads

Overlapping campaigns can expose control users to the ads being tested, contaminating the holdout. Check campaign exclusions and keep the original treatment and control assignments intact throughout the test.

Your Incrementality Testing Readiness Checklist

Before committing budget to an incrementality test, make sure your team can produce reliable results and act on them.

• Identify the campaign or budget decision the test will inform.
• Set the minimum incremental lift that would justify changing your budget.
• Confirm your audience size and conversion volume can support the test.
• Assign responsibility for tracking, exclusions, and result analysis.
• Decide what results would justify scaling, reducing spend, or retesting.
• Reserve enough budget to repeat an inconclusive test.

If you want to take a step further, talk to the Blockchain-Ads team about your campaign goals and measurement setup.

FAQs

What is the difference between A/B testing and incrementality testing?

Incrementality testing vs. A/B testing comes down to what they measure. Incrementality testing measures additional conversions caused by advertising using a no-ad control group. 

In contrast, A/B testing compares variations, such as creatives or landing pages, to identify which performs better. A/B tests can also measure incrementality when they include a no-ad control.

How long should an incrementality test run?

Run the test long enough to collect sufficient conversions and account for conversion lag. Set the duration based on baseline conversion rates, expected lift, and sample size. Avoid stopping early because results look favorable.

When is your account ready for an incrementality test?

Your account is ready when you have reliable conversion tracking, sufficient conversion volume, and enough campaign history to establish a baseline. Check how many channels are running, how long campaigns have been live, and whether you can isolate a holdout group without interference from other campaigns.

Do you need MMM to run incrementality tests?

No. MMM vs. incrementality comes down to measurement scope. Marketing mix modeling (MMM) estimates channel-level impact using aggregated historical data, while incrementality tests measure the causal effect of a specific advertising intervention. You can run incrementality tests without MMM and use their results to calibrate MMM estimates.