Do You Need a CRO Agency, or an App and a Freelancer?
The answer depends less on your budget than on your order count. Below a certain number of orders a month, an A/B test cannot finish in a useful time, and no agency, app or freelancer changes that. Here is the math, the real October 2026 prices for each route, and what each one costs a $500,000, $2M and $5M store.
Short answer
Under about 1,000 orders a month, skip A/B testing: use free Microsoft Clarity and a one-time audit or fix project ($495 to $1,495 at Evolve Media Agency). From about 1,000 to a few thousand orders (big changes only near 1,000), use a testing app (Shoplift from $99 a month, Intelligems from $69, Convert from $299 on annual). Add a freelancer at roughly $25 to $150 an hour, and it works if someone in-house owns it. Above that, or with nobody to own it, an agency fits: $995 a month at Evolve, about $3,500 at craftberry, from $10,000 at Invesp. For a $2M store, $995 a month plus a $99 app pays back with a 1.64% lift at a 40% margin.
Your Order Count Decides, Then Who Owns the Work
CRO (conversion rate optimization) is the work of getting more of your current visitors to buy. It has two halves. The first is finding and fixing what is broken: a slow mobile page, a confusing variant picker, missing trust signals at checkout. The second is A/B testing, where a tool splits visitors between the current page and a changed one and measures which sells more.
Apps run the second half. They do not tell you what to test, design the change, or stop you from calling a winner too early. A freelancer can do some of that. An agency is paid to do all of it. The first half, fixing what is broken, needs no testing traffic at all, which is why small stores should start there.
Under about 1,000 orders a month
Do not pay for testing. Install Microsoft Clarity (free) to watch real sessions, then buy one audit or fix project, or work through our free CRO playbook.
About 1,000 to a few thousand orders a month
An app plus a freelancer, if you or a team member can own the roadmap. An agency sprint if nobody can.
A few thousand orders a month and up
A testing program, run by an agency or a strong in-house lead with an app. The math now finishes tests in two to four weeks.
Apps, Freelancer or Agency: Price, Scope, Fit and Terms
CRO apps, do it yourself
App plus a freelancer
CRO agency
What the price columns leave out. The real cost of the app route is your time. Someone has to read session recordings, pick the one change most likely to matter, build it, check it on a phone, start the test, and leave it alone until it finishes. That is a few hours a week, every week. If that person is the founder, the app is cheap and the hours are not.
The freelancer route moves the building off your plate but not the deciding. A good Shopify developer will build exactly what you ask for; fewer will tell you it is the wrong test. The agency route moves both, at a higher monthly price, and adds someone whose job is to say when a result is noise. Each step up the ladder buys judgment, not software. The apps are the same tools in every column.
One more line item: the agency fee rarely includes the app. Spiralyze says its flat fee includes testing tools, and Shoplift's managed plan is the app itself. Everywhere else, budget for the app on top.
Below This Line, A/B Tests Cannot Finish
A test is "significant" when the difference is big enough that luck is an unlikely explanation. The common standard is 95% confidence with 80% power (an 80% chance of catching a real lift). A widely used shortcut for the visitors each version needs is:
visitors per version = 16 × p × (1 − p) ÷ (p × lift)², where p is your conversion rate and lift is the relative change you want to detect.
Worked example at a 2.5% conversion rate: for a 10% lift, 16 × 0.025 × 0.975 ÷ 0.0025² = 62,400 visitors and about 1,560 orders per version, so about 124,800 visitors for a two-version test. For a 20% lift, 15,600 visitors and 390 orders per version. The order count barely moves with conversion rate: at 2% or 3% it is still about 1,550 orders per version for a 10% lift.
$500,000 store (example)
About 521 orders and 20,800 sessions a month at an $80 average order value (AOV). A 10% test takes about six months; a 20% test about 46 days. Testing is not the tool here.
$1M store (example)
About 1,042 orders and 41,700 sessions a month. A 10% test takes about three months; a 20% test about 23 days. Only bold changes can be tested.
$2M store (example)
About 2,083 orders and 83,300 sessions a month. A 10% test takes about 46 days on all traffic; a 20% test about 11 days, though you should still run two full weeks.
$5M store (example)
About 5,208 orders and 208,000 sessions a month. A 10% test finishes in about 18 days. This is where a monthly program earns its keep.
Pick the Route That Matches Your Store and Your Team
Apps win
You have the orders, a person who likes the work, and a few hours a week. Shoplift at $99 or Intelligems from $69 gets you a real testing engine for about a tenth of our $995 monthly testing fee. Add Clarity for free and you have the same kind of tools agencies use.
A freelancer wins
You have a roadmap and need hands: a Shopify developer to build variations, or a designer for one product page. At $25 to $150 an hour you pay only for the work, with no monthly floor.
An agency wins
Nobody in-house owns conversion, you want fixes shipped and measured without managing a person, or you sell on Amazon too and want one team for both. At $5M and up, a program like craftberry's or Invesp's has the traffic to run several tests a month.
Another agency beats us
If you want 3 to 5 tests a month, craftberry's stated retainer (about $3,500, for stores doing 1,000+ orders and $2M+) fits better than our 1 to 2 tests at a time. At $6M+ with a senior team in mind, Invesp Growth.
Mobile is the real problem
If your phone conversion rate trails desktop by a wide margin, start with our mobile conversion guide. Most of those fixes need no test at all.
Yearly Cost and Break-Even, at $500K, $2M and $5M
Example math. Break-even assumes a 40% gross margin (what is left of a sale after product, shipping and fees) and counts first-year revenue only. Break-even lift = yearly cost ÷ 0.40 ÷ yearly revenue.
$500,000 store
Clarity: $0. A $495 audit or $1,495 sprint: break-even lift 0.25% or 0.75%. Shoplift Core plus a freelancer at 10 hours a month ($849 a month, $10,188 a year): 5.1%. An agency at $995 a month plus Shoplift ($13,128 a year): 6.6%. Buy the one-time fix.
$2M store
Shoplift Core alone ($1,188 a year): 0.15%. Shoplift plus a freelancer ($10,188): 1.27%. Evolve testing plus Shoplift ($13,128): 1.64%. craftberry at about $3,500 plus Shoplift ($43,188): 5.4%. Is an agency worth it here? Yes if nobody in-house owns testing and your page traffic can finish a test; otherwise app plus freelancer, or a $1,495 sprint first.
$5M store
Convert Growth at $399 as the example app. App plus a freelancer at 20 hours ($1,899 a month, $22,788 a year): 1.14%. Evolve testing plus app ($16,728): 0.84%. craftberry plus app ($46,788): 2.34%. Invesp Growth from $10,000 a month ($120,000 a year, before any app): 6%. A program is the normal buy; pick by how many tests a month you want.
When You Should Buy No CRO at All Yet
Your store does not get traffic yet
CRO works on visitors you already have. If you get a few hundred sessions a day, the money belongs in product content and traffic first.
Your photos and reviews are weak
No app or agency can test its way past a product page with poor images and no reviews. Fix those, then measure.
Your site is slow on phones
A page that takes four seconds to load on a phone loses before any test starts. Our theme performance guide covers the fixes.
You want a guaranteed number
No app, freelancer or agency controls your customers. We do not guarantee a lift; we guarantee the audit, the fixes and the measurement.
Five Questions to Answer Before You Pick a Route
How many orders did you take last month?
Run the formula above with your own conversion rate. If a test at the lift you expect takes more than about eight weeks, do not pay for testing. At 1,000 orders a month, only 20%+ changes clear that bar.
Who will own the roadmap?
If the answer is "nobody," an app will sit unused. Hire a freelancer only if someone can direct them.
Do you have a developer?
Most test variations need theme code. Self-serve apps do not build them. Intelligems says its quote-only managed service builds tests; ask any managed plan who writes the code.
Which metric will you judge by?
Revenue per visitor, usually. Conversion rate alone can rise from discounts that cost you margin.
What is the exit?
Self-serve apps are usually monthly; Intelligems Unlimited is three months; Shoplift's managed plan is annual. Evolve is month to month.
About the Author
Ian Smith
Founder, Evolve Media Agency
Ian has run Evolve Media Agency since 2017 and has spent more than a decade in ecommerce, building and selling his own brands and producing listing content, video and ads for Amazon, Shopify and TikTok Shop brands. He writes every page on this site himself and the prices quoted for Evolve are the prices you pay.
Agency vs App Questions
Do I need a CRO agency or just an app?
Is a CRO agency worth it for a $2M store?
How much traffic do I need for A/B testing?
What do Shopify A/B testing apps cost?
Is Microsoft Clarity enough instead of Hotjar?
What does a CRO freelancer cost?
Can a small Shopify store do CRO without testing?
Does Evolve Media Agency sell CRO testing to small stores?
Not Sure Which Route Fits? Bring Your Order Count
Book a free 30-minute call. Tell us last month's sessions and orders and we will run the test math with you and tell you whether an app, a freelancer or an agency fits, including when the answer is not us.