CRO means conversion rate optimization: changing your store so more visitors buy. The first question is not which agency. It is whether your store has enough orders to test at all. Start with the math, then use these 15 questions and the scorecard to compare agencies.
Short answer
Before hiring a Shopify CRO agency, check your traffic. Spotting a 10% lift at a 2.5% conversion rate takes about 62,400 visitors and 1,560 orders per version. Then ask how long a test takes on your store, how winners are picked, which tool runs the tests, and who owns the data. craftberry says testing needs about 1,000 orders a month and publishes retainers from about $3,500. Evolve Media Agency sells a $495 CRO Audit, a $1,495 Sprint and $995 a month testing, the last only to stores with a few thousand orders a month.
~62,400
visitors per version for a 10% lift
example at 2.5% conversion
~1,560
orders per version for the same test
about 3,120 orders in all
15
questions to ask on the first call
with good answers below
$495
our CRO Audit, the usual first step
sprint $1,495, testing $995 a month
An A/B test splits visitors between two versions of a page and counts who buys. To tell a real lift from luck, each version needs enough orders. Too few, and the test either runs for months or ends with a "winner" that was chance.
The rule of thumb most testers use: visitors per version is about 16 × p × (1 − p) ÷ (lift × p)², where p is your conversion rate. That gives 95% confidence and 80% power, which means an 80% chance of catching a lift that is really there.
The surprise is that the number of orders hardly moves with conversion rate. A 10% lift needs roughly 1,550 orders per version whether you convert at 1.5% or 3.5%. So count orders, not sessions.
Worked example. A 10% test at 2.5% conversion needs about 3,120 orders across both versions. At 1,000 orders a month, with all traffic in the test, that takes about 95 days. At 2,000 orders it takes about 47 days. At 3,000 it takes about a month, and at 500 it takes about six months.
That is why craftberry draws its line near 1,000 orders a month and why our own testing plan is sold only to stores with a few thousand. Below that, a good agency sells research and fixes, not a testing program. Our Shopify conversion rate optimization guide covers the fixes you can ship without a test.
Analytics, recordings, a phone checkout walk-through and customer words come first. Each test should start from a problem seen in your data, not a list of tips.
The agency writes down how many orders each version needs and the end date. It does not stop the test when the chart looks good.
Run at least two full weeks so weekday and weekend buyers both count. Avoid sales events, which bring different shoppers.
Revenue per visitor or conversion rate, picked before launch. Add-to-cart rate and bounce rate are helpers, not deciders.
Idea, dates, visitors and orders per version, result with confidence, and the decision. Losing tests teach as much as winners.
A winning test is only revenue once it is in the live theme. Ask for the ship date and a check of the numbers in the month after.
A year of tests is a record of what your customers respond to. It is worth more than any single winner. If it lives in the agency's tool account, it leaves with the agency.
You pay the tool directly or the agency bills it at cost, and you can log in any day. The agency is added as a user.
A shared drive or doc you control, with the hypothesis, screenshots of both versions, the numbers and the decision for each test.
Ask for the visitor and order counts per version as a file. A slide with a lift percentage cannot be checked later.
Winning code should be in your store's own theme, with a note of what changed, so the next developer can find it.
Good answer: a number, with the math behind it. craftberry states (checked Oct 22, 2026) that reliable testing generally needs about 1,000 orders a month. We sell monthly testing only to stores with a few thousand. Weak answer: "We can test anything."
Good answer: they run the sample-size math with your orders and conversion rate, on the call or in writing. If they will not, that is the answer.
Good answer: an audit and direct fixes, measured month before against month after. That is what craftberry says it does below its line, and what our audit and sprint are for.
Good answer: research before ideas. Analytics, session recordings, a walk through checkout on a phone, customer surveys or reviews, then a ranked list by likely impact and effort. A list of "best practices" with no data from your store is a template.
Good answer: 95% confidence, a sample size set before the test starts, and at least two full weeks so weekdays and weekends both count. craftberry states 95% and at least two full business cycles.
Good answer: "Not on a win." Checking a test every day and stopping when it looks good is called peeking. It turns luck into fake winners. Stopping a clear loser early to protect sales is reasonable.
Good answer: one primary metric, chosen before the test. Revenue per visitor is common because it counts both conversion and order size. Conversion rate alone can rise while revenue falls if a test pushes cheaper orders.
Good answer: the tool by name, installed on your store under your account and billing. craftberry names Intelligems as its testing partner. If the agency's own account runs the tests, you lose the history when it leaves.
Good answer: "You do." Every test plan, screenshot, result and the raw export should be yours, in a shared folder you control. Get this in writing; it is the record of what works on your store.
Good answer: the idea and why, the dates, visitors and orders per version, the primary metric with the confidence level, and a plain decision: ship, drop or retest. A report of only winners hides how many tests failed.
Good answer: a number and a definition. Ask whether it counts only finished tests. Many tests do not win, and that is normal; a claimed win rate near 100% means tests are too timid or the counting is generous.
Good answer: a named developer, inside the fee or billed at a stated rate. craftberry says development is billed separately. Ask how a winner gets into your live theme, and how fast.
Good answer: a number in writing tied to work you can count. craftberry publishes retainers from around $3,500 a month for research and 3 to 5 tests. Ours: CRO Audit $495, CRO Sprint $1,495, monthly testing $995.
Good answer: the term and notice period in writing, and what you keep on the way out. Our testing is month to month, paid 100% upfront. Long first contracts on a store that has never tested are a risk.
Good answer: a name and permission to call, at similar orders a month and category. Case studies on a site are chosen by the agency; a phone call is not.
Pay most attention to questions 1, 2, 8 and 9. The first two tell you whether testing fits your store. The last two decide whether you keep what you paid for.
An agency that sells monthly testing to a store with a few hundred orders a month is selling tests that cannot finish. Ask for the sample-size math.
"We stop when it hits 95%" without a planned sample size is peeking. Early winners found this way often vanish after launch.
When you leave, the test history goes with it. The tool should be on your store under your account.
If you never see a losing test, you cannot judge the win rate or the quality of the ideas.
Results depend on your product, price and traffic. A guarantee usually means a target set low, a long contract, or a metric chosen after the fact.
No named developer. Development billed by the hour with no cap. A first contract longer than three months. A sales lead who will not be on the account. Ideas pitched before anyone looked at your data.
Score each line 0, 1 or 2 after the call. Zero means no answer or a red flag; one means vague; two means specific and in writing. Any red flag is an automatic stop. Use the same sheet on us.
From Shopify Analytics. Divide orders by sessions for your conversion rate. Bring these three numbers to every call; they decide whether testing is even possible.
At your conversion rate, work out how many orders a 10% test needs per version and how many weeks your store takes to get there. That tells you whether to shop for testing or for fixes.
Store revenue, orders a month, conversion rate, average order value, theme, apps and the one number you want to move. The same input makes the answers comparable.
Take notes in the agency's words. Score each line 0, 1 or 2 on the scorecard. Any red flag ends it, whatever the total.
A fixed-price audit shows how the agency thinks before you commit to monthly testing. Read it for evidence: data, recordings and a reason for each fix.
Primary metric, test count, test length and who owns the testing tool account. Put them in the scope before the first invoice.
A testing retainer will pay for tests that end inconclusive. Buy an audit (compare Shopify CRO agency pricing), ship the fixes, and compare the month before to the month after.
If few people visit, CRO works on a small base. Fix how you get visitors first, or the lift will be real and still small in dollars.
If orders in Shopify and your analytics tool disagree badly, no test result can be trusted. Fix tracking before anything else.
Speed and mobile bugs are fixes, not tests. Any developer can ship them, and you should not pay a testing fee to find them.
If nobody can change the theme for weeks at a time, winners pile up unused. Line up a developer first.


Founder, Evolve Media Agency
Ian has run Evolve Media Agency since 2017 and has spent more than a decade in ecommerce, building and selling his own brands and producing listing content, video and ads for Amazon, Shopify and TikTok Shop brands. He writes every page on this site himself and the prices quoted for Evolve are the prices you pay.
Service
CRO Audit $495, CRO Sprint $1,495, monthly testing $995 for stores with enough orders.
Roundup
12 CRO agencies ranked with published prices, traffic minimums and how each one tests.
Comparison
App, freelancer and agency prices, and the traffic math that decides if A/B testing works.
Free · 30 minutes · No pitch deck
Book a free 30-minute call with 90 days of sessions and orders. We will run the sample-size math with you, tell you whether you should buy testing or fixes, and answer all 15 questions on this page.
Tell us what you sell and where you are stuck. Ian replies within one business day, and you get an email right away with a link to book a free 30-minute call.
Rather talk now? Call or text 719-257-3557 or pick a time on Calendly.