Before You Commission a CRO Engagement: A 12-Point Readiness Check

It is budget season, and somewhere in next year’s plan sits a line for conversion work. A retainer, an audit, an agency to finally fix the gap between the traffic you pay for and the orders you keep. The instinct at this point is to start comparing agencies. The more useful question comes one step earlier: can your store turn a CRO engagement into signal yet, or will it turn your money into confident noise?

Most guides help you pick the partner. Almost none ask whether the store is ready to be worked on. This one is the readiness check to run before you sign anything, so the engagement starts on a foundation that can actually measure whether the work is helping.

The regret that CRO guides never mention

The regret most teams carry out of a first CRO engagement is not that they hired the wrong agency. It is that they hired any agency a quarter too early. The pattern is consistent enough to describe in advance: six months in, three or four tests have run, the results are mixed or flat, and the contract quietly lapses. The agency points to low traffic and messy tracking. The team points to the agency. Both are partly right, and neither names the real cause.

The real cause is upstream of the agency entirely. The engagement was commissioned into a store that could not yet convert testing into a reliable answer. No baseline to measure lift against, tracking that was quietly miscounting, or too little traffic for any result to clear significance inside a reasonable window. Drop a capable team into that and they will still produce reports, still run tests, still send confident-looking numbers. The numbers just will not mean what everyone assumes they mean. That is the specific outcome this check exists to prevent, and it is preventable in an afternoon of honest self-assessment before the budget is committed.

Why readiness decides the outcome, not the agency

A CRO engagement turns effort into signal only when three things are already true: you have a baseline to measure against, tracking you can trust, and enough traffic to reach statistical significance. Miss any one of the three and even a strong agency produces results that look precise and tell you nothing.

Each of the three fails in its own quiet way. Without a documented funnel baseline, the word “lift” has nothing to be measured against, so every result becomes a story rather than a number. With broken tracking, tests are built on data that is already wrong, and a confident percentage calculated from bad events is worse than no percentage at all because it invites action. And without traffic, significance never arrives: Shopify’s own guide to A/B testing is blunt that a real winner has to clear a confidence threshold, not a few good days, and reaching that threshold takes volume most smaller stores do not have. Below a certain size, formal testing is simply premature, and the higher-return move is to watch real sessions and ship the obvious fixes rather than to run experiments that cannot conclude.

This is why disciplined CRO begins with diagnosis, not a test list. Locate where the store actually leaks, quantify what it costs, and only then decide whether testing or plain fixing is the right instrument. The same order of operations runs through our own Shopify conversion leak diagnostic, and it is the discipline a readiness check is really testing for: are you set up to diagnose before you spend a quarter proving small things in the wrong place?

The 12-point readiness check

Score your store out of twelve. Treat anything you cannot check not as a reason to abandon CRO, but as the work to finish before the engagement, so it starts on signal instead of noise. The twelve fall into four groups, and the groups matter as much as the items: a store strong in one and empty in another is not ready, because the weak group is where the confident noise comes from.

Data and measurement

1. GA4 and Shopify Analytics are both configured, and you trust the numbers. Shopify Analytics gives you quick commerce data; GA4 gives you per-page sessions, device splits, and segmented funnels. You need both, and you need to believe them.
2. Conversion events and attribution are validated, not assumed. Someone has confirmed that purchases, add-to-carts, and checkout steps fire correctly and are not double-counted or lost inside a checkout sandbox. Tests built on flawed events produce flawed winners.
3. You have a documented baseline of your funnel ratios over the last 90 days. Sessions to product view, to reached checkout, to completed order, pulled over a full quarter so seasonality washes out. Without it, no future result has a reference point.

Traffic and significance

4. Your key pages carry enough traffic to reach significance in a reasonable window. If a test on your highest-traffic page would take months to conclude, formal testing is not your next move yet. Enterprise experimentation programs often assume six-figure monthly visitor volumes for a reason.
5. You know your hero SKUs and your highest-converting channels. In most stores a small number of products and one or two channels carry the majority of revenue. If you cannot name yours, the engagement will spend its first weeks discovering what you could have told it.
6. You have chosen a primary metric that ties to money. Revenue per visitor or contribution margin, not conversion rate in isolation, because a discount can lift conversion rate while quietly destroying margin.

Velocity and implementation

7. You have development capacity to ship changes into your theme or Plus checkout. A backlog of winning ideas that nobody can implement is a backlog, not a program. Confirm whether the work lands on your developers or the agency’s.
8. You have agreed a realistic testing cadence. A believable number of experiments per month at your traffic level, not an aspirational one. Velocity is where learning compounds, but only at a pace your traffic can validate.
9. You have a place to log every test, hypothesis, result, and learning. Including the tests that lose, since most do, and the losses are where the understanding accumulates.

Organization and scope

10. There is a named owner and a decision-maker. One person accountable for the program and one who can approve shipping a winner without a committee. Engagements stall in the gap between them.
11. Your budget is sized to your scale. A fee that ignores your traffic and revenue is a mismatch in either direction. The program has to be affordable relative to the revenue it can plausibly move.
12. You have clear goals and a time horizon of at least one quarter. CRO is a compounding practice, not a one-month fix. A goal without a horizon becomes an expectation nobody can meet.

Count your checks. Ten to twelve, and you are ready to commission with confidence. Six to nine, and you are close: fix the gaps first, and the same engagement will produce far better answers. Below six, and an engagement started now will mostly pay an agency to discover what this list just showed you for free.

If you have already committed

If you have already signed, or you are mid-engagement and it is producing noise, the move is not to cancel reflexively. Pause net-new testing for a beat. Fix the tracking and establish the baseline first, because every test running on top of broken measurement is spending your budget to generate numbers you will not be able to trust. Renegotiate the cadence toward diagnosis rather than volume, and re-scope the current phase to research: where is the store actually leaking, and is this store even ready for the test-and-iterate loop, or is it still in watch-and-fix territory? A good agency will welcome the reset, because a diagnosed store is one they can actually show results on.

Run it against your store

You can work through these twelve points yourself before the next budget cycle, and it is worth doing even if you never hire anyone. If you would rather have them run against your live store, that is precisely what a diagnostic is for. Flatline is a Shopify Premier Partner, and our conversion leak diagnostic is the natural entry point ahead of any engagement: it establishes the 90-day baseline, validates that your tracking is telling the truth, and tells you which of the twelve you already have and which to close before a single test goes live. That diagnostic sits inside the same ecommerce agency practice that runs the engagement itself, so nothing gets re-explained once testing starts. If you want that readiness picture for your own store before you commit a budget, get in touch and we will walk through it with you.

Frequently asked questions

How much traffic do I need before running CRO A/B tests? 

Enough to reach statistical significance on your key pages within a reasonable window, which depends on your baseline conversion rate and the size of lift you want to detect. As a rough guide, enterprise experimentation programs often assume six-figure monthly visitors, and many practitioners consider formal testing premature for smaller stores. Below that, watching real sessions and shipping obvious fixes usually returns more than formal experiments that cannot conclude.

Do I need CRO if I’m under $1M GMV? 

You need conversion work; you may not yet need a formal testing program. At smaller scale, the higher-return version of CRO is diagnostic and observational: read your funnel, watch session replays, and fix the obvious friction you find. Formal A/B testing tends to pay off later, once traffic can support significance and the obvious problems are already solved.

What’s the difference between a CRO readiness check and a CRO audit? 

A readiness check asks whether your store can turn an engagement into signal: data, traffic, velocity, and organization. An audit examines the store itself for specific conversion problems to fix. The readiness check comes first. Running an audit or a test program before you are ready produces findings you cannot reliably measure or act on.

Should I fix tracking before or during a CRO engagement? 

Before, wherever possible. Tracking is the instrument every result is read from, so testing on top of broken measurement spends budget generating numbers you cannot trust. Validating events and attribution first is not a delay to the program; it is the precondition that makes the program’s output meaningful.

Key takeaways

• The common regret after a first CRO engagement is not the wrong agency, but commissioning any agency a quarter too early, into a store that could not yet turn testing into signal.
• Readiness, not the agency, decides the outcome: without a baseline, trustworthy tracking, and enough traffic for significance, even a strong team produces confident noise.
• Score your store against the twelve points across data, traffic, velocity, and organization. Ten or more means ready; below six means an engagement would mostly pay to discover these gaps.
• Below the traffic threshold for significance, watch-and-fix returns more than formal testing. Diagnose before you build a test backlog.
• If you have already committed, pause net-new testing, fix tracking and baseline first, and re-scope the current phase to diagnosis.
• The readiness check works as your own pre-commit tool, and a conversion leak diagnostic is the natural way to have it run against your store before a budget is signed.