Incrementality On Amazon: Proving Your Ads Caused The Sale

Why ROAS Cannot Answer This

The distinction is worth stating precisely because it is frequently blurred by people who understand it perfectly well.

Attribution assigns credit for a conversion that occurred, according to rules about which touchpoints count and over what window. It is an accounting exercise performed on things that happened. Incrementality asks a different question: how many of those conversions would have occurred without the advertising?

The reason this matters commercially is that budget decisions made on attributed ROAS are systematically distorted in a predictable direction. Campaigns capturing existing demand look excellent. Campaigns creating new demand look mediocre. Money flows toward the former, which is exactly backwards if you are trying to grow.

None of this makes ROAS useless. It is a fine operational metric for comparing two similar campaigns or spotting a break. It is a poor instrument for deciding whether a channel deserves to exist, and it is used for that constantly.

Where Attribution Overstates Most

Overstatement is not evenly distributed. It concentrates in four places, and knowing which they are tells you where testing is worth the effort.

The last row is the one to sit with. If branded search is overstated and category acquisition is understated, then a brand optimizing to attributed ROAS is steadily moving budget away from growth and toward credit-taking, while its dashboard improves the whole time.

That pattern is worth checking before you test anything, because campaign overlap in particular has a cheaper fix. If multiple campaigns are serving the same shoppers without exclusion logic, tightening the structure resolves double-counting without an experiment. Our Amazon PPC strategy guide covers that structural work.

What Incrementality Actually Means

One source puts it well: most brands that claim to test incrementality are actually measuring attribution shift. The distinction is worth guarding.

Turning off a campaign and watching attributed sales move between campaigns is not an incrementality test. That is credit relocating. A real test requires two comparable groups, one exposed and one deliberately withheld, and measures the difference in total outcomes between them, not the difference in what got credited.

The comparison that matters is total units or total revenue across the whole business for those groups. If pausing a campaign leaves total sales unchanged while attributed sales collapse, you have learned something valuable: the campaign was relabelling demand rather than creating it.

The Four Test Designs

Each has different requirements and different weaknesses. Pick on what you can actually run rather than what is theoretically cleanest.

A note on the fourth. Platform lift studies have a known measurement asymmetry in the wider industry, where exposed groups are tracked more completely than control groups, producing apparent lift that partly reflects better observation rather than better outcomes. That criticism is documented mainly against other platforms, and the general principle stands: a measurement run by the party being measured deserves scrutiny.

For most Amazon sellers without DSP, design three is the realistic option, and it is also the one where the Amazon-specific complication in section six bites hardest.

Sizing The Test

Thin spend produces thin signal. If your test cannot move enough volume to register a measurable gap between exposed and control, you cannot separate effect from noise, and you will have paid for an inconclusive result.

Decide the following before launch rather than after, because deciding afterward is how brands talk themselves into a conclusion the data does not support.

1. The confidence threshold. Commonly 95 percent. Set it in writing before you start.
2. The minimum effect worth detecting. If a 3 percent lift would not change your decision, do not design a test capable only of finding 3 percent.
3. The duration. Two to four weeks is the typical window. Shorter rarely accumulates enough volume; longer accumulates too much contamination.
4. The primary metric. Total units or total revenue across the business, not attributed sales. Write this down or someone will report the wrong one.
5. What you will do with each outcome. Both directions. A test with only one actionable result is not a test.

Geographic tests carry additional data requirements. One source describing geo methodology calls for first-party performance data at least at state level, reported weekly or daily, with at least six months of clean historical data before the test to establish baselines. That is a meaningful prerequisite and many brands do not have it.

The Ranking Trap

Here is what makes Amazon different from every other platform you might run this on, and it is largely absent from generic incrementality guidance.

Amazon's organic ranking responds to sales velocity. Products that sell well rank better, and ranking better produces more sales. Advertising feeds that loop, because ad-driven sales are still sales.

So when you pause advertising on a product to measure incrementality, two things happen simultaneously. Ad-driven sales stop, which is what you intended to measure. And the product's sales velocity drops, which degrades its organic position, which reduces organic sales, which degrades velocity further.

The measurement consequence is specific and it runs in one direction. Your test will show a larger sales drop than the advertising alone caused, because part of the decline came from ranking decay triggered by the pause. You will conclude the ads were more incremental than they were.

The commercial consequence is worse and outlasts the test. Ranking lost during a pause does not return the moment you resume. You have to rebuild velocity to recover position, which means the true cost of the experiment includes a recovery period nobody budgets for. One source refers to needing patience to weather the short-term ranking dip, which is a mild way of describing a real expense.

For how the ranking mechanism works in detail, our breakdown of Cosmo against the older A9 and A10 models covers the velocity relationship this exploits.

Designing Around It

The ranking effect cannot be eliminated, and it can be reduced enough that a test is worth running.

Test on a subset, never the whole catalog. Pause on a controlled group of products and compare against a matched set that keeps running. This contains the ranking damage to products you chose to risk, and the matched control also absorbs seasonality and category movement.

Choose products with established rank. A product with a long stable ranking history has more resilience than a recent launch still building position. Never run this on a product in its ramp period.

Keep the window short. Two weeks limits decay compared with four. There is a genuine trade against statistical power, and on Amazon the shorter end is usually the better compromise.

Prefer audience holdouts where you can. A DSP audience holdout withholds ads from a slice of people rather than removing ads from a product entirely, so aggregate velocity holds up and ranking is largely protected. This is the strongest argument for running incrementality work through DSP if you have it.

Reduce rather than eliminate. Cutting spend substantially instead of going dark preserves some velocity while still creating a measurable gap. Less clean statistically, considerably cheaper in ranking terms, and often the right trade.

Measure the recovery too. Track how long ranking takes to return after the test. That number is part of the cost of testing and you will want it before deciding whether to test again.

Reading The Result Honestly

The failure mode after a test is not usually bad arithmetic. It is over-claiming.

Incremental ROAS is the measured revenue lift divided by the spend that produced it. If pausing spend cost you materially less revenue than the platform claimed that spend generated, your incremental figure is lower than the reported one, and the correct comparison for a budget decision is incremental ROAS against your contribution-margin breakeven rather than against the reported number.

Three honesty checks before you present anything.

• State the confidence interval, not the point estimate. A result of 2.1x with a range spanning 0.8x to 3.4x is a different finding from 2.1x with a tight band, and only one of them justifies a decision.
• Say what was confounded. On Amazon that means naming the ranking effect explicitly and acknowledging your measured lift is inflated by it.
• Do not generalise beyond the test. A result on ten products in one category during three weeks in September is a result on ten products in one category during three weeks in September.

The single most useful discipline is to write down what you expect before you look. A prediction recorded in advance turns a result into evidence. A prediction formed after seeing the number is a story.

The Benchmark Numbers

Several figures circulate in this space. They are directionally useful and individually unciteable, and the distinction matters.

The last one is worth keeping for one reason. It demonstrates that a properly run test can return no measurable lift at all, and that this is a finding rather than a failure. The chain reallocated the budget. That is what a test is for.

What none of these figures support is skipping the test. Adopting a published benchmark as your own incrementality rate is the same error as trusting attributed ROAS, differing only in which number you are believing without evidence.

Turning A Result Into A Decision

Most incrementality work dies here, producing a slide instead of a budget change.

If incremental ROAS clears your margin breakeven, the campaign is profitable at its true rate even if the honest number is far below the reported one. Keep it. This is a common and underappreciated outcome, and cutting on the basis that the number came down is an overreaction.

If it sits below breakeven, reduce rather than eliminate first. Branded search in particular has defensive value against competitors bidding on your terms, which is real and does not appear in an incrementality figure.

If lift is indistinguishable from zero, that is a strong signal and you should act on it. Reallocate to something with demonstrated incrementality, which usually means the category and competitor targeting that attributed reporting has been undervaluing all along.

If the result is ambiguous, resist re-running immediately. Each test costs ranking. Improve the design and wait for a period where the read will be cleaner, avoiding seasonal distortion.

Whichever way it goes, the comparison is against your contribution margin rather than against a ROAS target inherited from somewhere. Our contribution margin playbook covers building that per SKU, and revenue against actual profit covers why the distinction changes conclusions.

Common Ways Tests Fail

Recognisable patterns, each of which invalidates the result rather than merely weakening it.

1. Improvising the holdout inside live trafficking. Deciding the control group after campaigns are running invites contamination. Build the control logic in advance.
2. Measuring attributed sales instead of total sales. The most common error. You will observe credit moving and mistake it for demand moving.
3. Running during a distorted period. Prime Day, holiday peaks, and your own promotions all break the comparison between test and control.
4. Changing something else mid-test. A price change, a listing update, a stockout. Any of these ends the experiment whether or not anyone notices.
5. Underpowering it. Too little spend, too few products, too short a window. Produces an inconclusive result at full cost.
6. Ignoring the ranking effect. Amazon-specific, and it inflates measured incrementality in a direction that happens to flatter the campaign you were testing.
7. Deciding the threshold afterwards. Choosing what counts as significant once you have seen the number guarantees you find what you wanted.

Number four deserves particular attention on Amazon because stockouts are common and devastating to a test. A product that runs out mid-experiment has left the study, and if it was in your test group the result is not salvageable.

When Not To Test

Incrementality testing is fashionable and it is not always the right next step. Several situations call for something else.

When your campaign structure has overlap problems. If several campaigns target the same shoppers without exclusions, fix that first. You will resolve most double-counting without spending ranking on an experiment.

When spend is too small to read. Below meaningful volume, no design will separate signal from noise. You will pay the full ranking cost for an inconclusive answer, which is the worst available outcome.

When you would not act on the result. If branded search is going to keep running for defensive reasons whatever the number says, the test is expensive curiosity.

When products are ramping. Never test on products still establishing rank. The ranking cost is highest and the baseline is least stable exactly there.

During Q4. Distorted demand, maximum ranking value, and the highest cost of a mistake. Test in a quiet period.

The Cheaper Thing To Do First

Before designing an experiment, look at what proportion of your ad-attributed revenue comes from exact-match branded terms. If it is large, you already know most of that revenue was probably coming anyway, and you can act on that suspicion by trimming branded bids modestly and watching total revenue rather than by running a formal test. That is not rigorous and it is nearly free, and for many brands the directional answer is enough to move budget sensibly.

Save the formal test for when the stakes justify the ranking cost, meaning a large budget line, a genuine internal disagreement, or a decision that will not be made without evidence. Our guide to attribution tracking covers the measurement groundwork that makes any of this readable, and Amazon DSP for growing brands covers whether the audience-holdout route is available to you at all.