Home/Shopify CRO Services/How to Hire a Shopify CRO Agency
[01] How to Hire a Shopify CRO Agency

How to Hire a Shopify CRO Agency: 15 Questions to Ask First

CRO means conversion rate optimization: changing your store so more visitors buy. The first question is not which agency. It is whether your store has enough orders to test at all. Start with the math, then use these 15 questions and the scorecard to compare agencies.

Last reviewed October 23, 2026Prices checked October 2026By Ian Smith

Short answer

Before hiring a Shopify CRO agency, check your traffic. Spotting a 10% lift at a 2.5% conversion rate takes about 62,400 visitors and 1,560 orders per version. Then ask how long a test takes on your store, how winners are picked, which tool runs the tests, and who owns the data. craftberry says testing needs about 1,000 orders a month and publishes retainers from about $3,500. Evolve Media Agency sells a $495 CRO Audit, a $1,495 Sprint and $995 a month testing, the last only to stores with a few thousand orders a month.

~62,400

visitors per version for a 10% lift

example at 2.5% conversion

~1,560

orders per version for the same test

about 3,120 orders in all

15

questions to ask on the first call

with good answers below

$495

our CRO Audit, the usual first step

sprint $1,495, testing $995 a month

[02] Traffic First

Can Your Store Run a Test at All?

Disclosure: We are Evolve Media Agency and we wrote this guide. We sell Shopify CRO audits, sprints and monthly testing, so you may be asking these questions of us. We wrote the good answers so that our own answers can be checked against them, and the scorecard works the same on us as on anyone else. Where we name another agency as an example, the fact comes from that agency's own page on Oct 22, 2026, linked. No agency pays us to appear here.

An A/B test splits visitors between two versions of a page and counts who buys. To tell a real lift from luck, each version needs enough orders. Too few, and the test either runs for months or ends with a "winner" that was chance.

The rule of thumb most testers use: visitors per version is about 16 × p × (1 − p) ÷ (lift × p)², where p is your conversion rate. That gives 95% confidence and 80% power, which means an 80% chance of catching a lift that is really there.

The surprise is that the number of orders hardly moves with conversion rate. A 10% lift needs roughly 1,550 orders per version whether you convert at 1.5% or 3.5%. So count orders, not sessions.

Example storeSample needed
1.5% conversion, 10% liftAbout 105,100 visitors and 1,580 orders per version
2.5% conversion, 10% liftAbout 62,400 visitors and 1,560 orders per version
3.5% conversion, 10% liftAbout 44,100 visitors and 1,540 orders per version
2.5% conversion, 20% liftAbout 15,600 visitors and 390 orders per version

Worked example. A 10% test at 2.5% conversion needs about 3,120 orders across both versions. At 1,000 orders a month, with all traffic in the test, that takes about 95 days. At 2,000 orders it takes about 47 days. At 3,000 it takes about a month, and at 500 it takes about six months.

That is why craftberry draws its line near 1,000 orders a month and why our own testing plan is sold only to stores with a few thousand. Below that, a good agency sells research and fixes, not a testing program. Our Shopify conversion rate optimization guide covers the fixes you can ship without a test.

[03] Test Method and Reporting

What a Sound Test and a Useful Report Look Like

Research before ideas

Analytics, recordings, a phone checkout walk-through and customer words come first. Each test should start from a problem seen in your data, not a list of tips.

Sample size set before launch

The agency writes down how many orders each version needs and the end date. It does not stop the test when the chart looks good.

Full weeks only

Run at least two full weeks so weekday and weekend buyers both count. Avoid sales events, which bring different shoppers.

One primary metric

Revenue per visitor or conversion rate, picked before launch. Add-to-cart rate and bounce rate are helpers, not deciders.

A report for every test, not only winners

Idea, dates, visitors and orders per version, result with confidence, and the decision. Losing tests teach as much as winners.

Winners shipped and checked

A winning test is only revenue once it is in the live theme. Ask for the ship date and a check of the numbers in the month after.

[04] Who Owns the Data

Your Store, Your Tool, Your Test History

A year of tests is a record of what your customers respond to. It is worth more than any single winner. If it lives in the agency's tool account, it leaves with the agency.

The testing app is installed under your account

You pay the tool directly or the agency bills it at cost, and you can log in any day. The agency is added as a user.

Every test plan and result is in a folder you own

A shared drive or doc you control, with the hypothesis, screenshots of both versions, the numbers and the decision for each test.

Raw exports, not just slides

Ask for the visitor and order counts per version as a file. A slide with a lift percentage cannot be checked later.

Theme changes in your theme library

Winning code should be in your store's own theme, with a note of what changed, so the next developer can find it.

[05] The 15 Questions

15 Questions and What a Good Answer Sounds Like

1

1. How many orders a month do you need before you will run A/B tests?

Good answer: a number, with the math behind it. craftberry states (checked Oct 22, 2026) that reliable testing generally needs about 1,000 orders a month. We sell monthly testing only to stores with a few thousand. Weak answer: "We can test anything."

2

2. How long would a test take on my traffic?

Good answer: they run the sample-size math with your orders and conversion rate, on the call or in writing. If they will not, that is the answer.

3

3. What do you do if my store is too small to test?

Good answer: an audit and direct fixes, measured month before against month after. That is what craftberry says it does below its line, and what our audit and sprint are for.

4

4. How do you decide what to test first?

Good answer: research before ideas. Analytics, session recordings, a walk through checkout on a phone, customer surveys or reviews, then a ranked list by likely impact and effort. A list of "best practices" with no data from your store is a template.

5

5. What confidence level and test length do you use?

Good answer: 95% confidence, a sample size set before the test starts, and at least two full weeks so weekdays and weekends both count. craftberry states 95% and at least two full business cycles.

6

6. Do you ever stop a test early?

Good answer: "Not on a win." Checking a test every day and stopping when it looks good is called peeking. It turns luck into fake winners. Stopping a clear loser early to protect sales is reasonable.

7

7. Which metric decides a winner?

Good answer: one primary metric, chosen before the test. Revenue per visitor is common because it counts both conversion and order size. Conversion rate alone can rise while revenue falls if a test pushes cheaper orders.

8

8. Which testing tool do you use, and whose account is it?

Good answer: the tool by name, installed on your store under your account and billing. craftberry names Intelligems as its testing partner. If the agency's own account runs the tests, you lose the history when it leaves.

9

9. Who owns the test data and the results?

Good answer: "You do." Every test plan, screenshot, result and the raw export should be yours, in a shared folder you control. Get this in writing; it is the record of what works on your store.

10

10. What does a test report show?

Good answer: the idea and why, the dates, visitors and orders per version, the primary metric with the confidence level, and a plain decision: ship, drop or retest. A report of only winners hides how many tests failed.

11

11. What is your win rate, and how do you count it?

Good answer: a number and a definition. Ask whether it counts only finished tests. Many tests do not win, and that is normal; a claimed win rate near 100% means tests are too timid or the counting is generous.

12

12. Who builds the tests and ships the winners?

Good answer: a named developer, inside the fee or billed at a stated rate. craftberry says development is billed separately. Ask how a winner gets into your live theme, and how fast.

13

13. What is the fee, and what is it for?

Good answer: a number in writing tied to work you can count. craftberry publishes retainers from around $3,500 a month for research and 3 to 5 tests. Ours: CRO Audit $495, CRO Sprint $1,495, monthly testing $995.

14

14. What is the contract term, and how do I leave?

Good answer: the term and notice period in writing, and what you keep on the way out. Our testing is month to month, paid 100% upfront. Long first contracts on a store that has never tested are a risk.

15

15. Can I talk to a client at my size?

Good answer: a name and permission to call, at similar orders a month and category. Case studies on a site are chosen by the agency; a phone call is not.

Pay most attention to questions 1, 2, 8 and 9. The first two tell you whether testing fits your store. The last two decide whether you keep what you paid for.

[06] Red Flags

Five Answers That End the Call

They will test on any traffic

An agency that sells monthly testing to a store with a few hundred orders a month is selling tests that cannot finish. Ask for the sample-size math.

Tests are called at a percentage, not a sample size

"We stop when it hits 95%" without a planned sample size is peeking. Early winners found this way often vanish after launch.

The testing tool is in the agency's account

When you leave, the test history goes with it. The tool should be on your store under your account.

Only winners are reported

If you never see a losing test, you cannot judge the win rate or the quality of the ideas.

A guaranteed lift

Results depend on your product, price and traffic. A guarantee usually means a target set low, a long contract, or a metric chosen after the fact.

Smaller warnings worth a follow-up

No named developer. Development billed by the hour with no cap. A first contract longer than three months. A sales lead who will not be on the account. Ideas pitched before anyone looked at your data.

Trust problems often beat test ideas. Shipping, returns, reviews and payment badges near the buy button are fixes many stores can make before testing; see trust signals that convert in 2026.
[07] The Scorecard

Score Each Agency Out of 30

Score each line 0, 1 or 2 after the call. Zero means no answer or a red flag; one means vague; two means specific and in writing. Any red flag is an automatic stop. Use the same sheet on us.

What you are scoringPoints
Order minimum stated with math (Q1)0 to 2
Test length on your traffic (Q2)0 to 2
Plan for a store too small to test (Q3)0 to 2
Research before ideas (Q4)0 to 2
Confidence level and minimum length (Q5)0 to 2
No early stopping on wins (Q6)0 to 2
One primary metric set in advance (Q7)0 to 2
Tool named, in your account (Q8)0 to 2
You own all test data (Q9)0 to 2
Sample report shows every test (Q10)0 to 2
Win rate with a definition (Q11)0 to 2
Named developer, shipping plan (Q12)0 to 2
Fee in writing, tied to work (Q13)0 to 2
Term and exit in writing (Q14)0 to 2
Reference at your size (Q15)0 to 2
Reading the total: 24 to 30 is a strong candidate. 16 to 23 is worth a second call with the weak lines as the agenda. Under 16, keep looking.
[08] The Hiring Process

From Your Numbers to a Signed Scope

1

Pull 90 days of sessions, orders and conversion rate

From Shopify Analytics. Divide orders by sessions for your conversion rate. Bring these three numbers to every call; they decide whether testing is even possible.

2

Run the sample-size math before the first call

At your conversion rate, work out how many orders a 10% test needs per version and how many weeks your store takes to get there. That tells you whether to shop for testing or for fixes.

3

Send three agencies the same brief

Store revenue, orders a month, conversion rate, average order value, theme, apps and the one number you want to move. The same input makes the answers comparable.

4

Ask the 15 questions and score the answers

Take notes in the agency's words. Score each line 0, 1 or 2 on the scorecard. Any red flag ends it, whatever the total.

5

Start with an audit or a short sprint

A fixed-price audit shows how the agency thinks before you commit to monthly testing. Read it for evidence: data, recordings and a reason for each fix.

6

Agree the 90-day numbers in writing

Primary metric, test count, test length and who owns the testing tool account. Put them in the scope before the first invoice.

[09] When Not to Hire

When You Should Not Hire a CRO Agency Yet

You have fewer than about 1,000 orders a month

A testing retainer will pay for tests that end inconclusive. Buy an audit (compare Shopify CRO agency pricing), ship the fixes, and compare the month before to the month after.

Your traffic is the problem, not the page

If few people visit, CRO works on a small base. Fix how you get visitors first, or the lift will be real and still small in dollars.

Tracking is broken

If orders in Shopify and your analytics tool disagree badly, no test result can be trusted. Fix tracking before anything else.

The site is slow or broken on phones

Speed and mobile bugs are fixes, not tests. Any developer can ship them, and you should not pay a testing fee to find them.

You cannot ship winners

If nobody can change the theme for weeks at a time, winners pile up unused. Line up a developer first.

[10] Who Wrote This

About the Author

Ian Smith, founder of Evolve Media Agency

Ian Smith

Founder, Evolve Media Agency

Ian has run Evolve Media Agency since 2017 and has spent more than a decade in ecommerce, building and selling his own brands and producing listing content, video and ads for Amazon, Shopify and TikTok Shop brands. He writes every page on this site himself and the prices quoted for Evolve are the prices you pay.

[11] FAQ

Hiring a CRO Agency Questions

What should I ask a Shopify CRO agency?
Start with traffic: how many orders a month it needs to test, and how long a test would take on your store. Then ask how it picks ideas, what confidence level and test length it uses, and which metric decides a winner. Finish with ownership and terms: which tool runs tests and in whose account, who owns the data, the fee, the term, and a reference at your size.
How much traffic do I need for A/B testing on Shopify?
Example: at a 2.5% conversion rate, spotting a 10% lift takes about 62,400 visitors and 1,560 orders per version, or about 3,120 orders in all. At 1,000 orders a month that takes about 95 days. craftberry puts its line near 1,000 orders a month; we sell testing only to stores with a few thousand.
Who should own CRO test data?
You should. The testing app should be installed on your store under your account, and every test plan, screenshot, result and raw export should sit in a folder you control. If the agency's account runs the tests, the history leaves when the agency does.
What are red flags when hiring a CRO agency?
Selling tests on traffic too small to finish them, calling tests at a percentage with no planned sample size, running tests in the agency's own tool account, reporting only winners, and guaranteeing a lift.
What should a CRO test report include?
The idea and the reason for it, the start and end dates, visitors and orders per version, the primary metric with its confidence level, and a decision: ship, drop or retest. It should cover every test, including the losers.
What if my store is too small for A/B testing?
Buy an audit and ship the fixes directly, then compare the month before to the month after. Fix tracking, speed, mobile bugs and trust signals first. Our CRO Audit is $495 and the CRO Sprint is $1,495.
How much does a Shopify CRO agency cost?
Published examples: craftberry retainers from around $3,500 a month for research and 3 to 5 A/B tests, with development billed separately. Evolve Media Agency: CRO Audit $495, CRO Sprint $1,495, monthly testing $995, paid 100% upfront. Many agencies quote only.
Is a CRO agency worth it for a $1M store?
Usually not for monthly testing. At an $80 average order value, $1M a year is about 1,040 orders a month, so a 10% test at 2.5% conversion takes about three months. An audit and a sprint of fixes usually pays back faster at that size.

Free · 30 minutes · No pitch deck

Bring Your Orders to the Call

Book a free 30-minute call with 90 days of sessions and orders. We will run the sample-size math with you, tell you whether you should buy testing or fixes, and answer all 15 questions on this page.

Free 30-minute call

Want Ian to Look at Your Brand?

Tell us what you sell and where you are stuck. Ian replies within one business day, and you get an email right away with a link to book a free 30-minute call.

  • Brand management, PPC, listings, launches and content
  • Amazon, Shopify and TikTok Shop brands
  • A real reply from Ian, not a sales bot

Rather talk now? Call or text 719-257-3557 or pick a time on Calendly.

Your information stays with us. We never share it.