Ask a room of Shopify merchants how to run an A/B test and most will repeat the same advice. Change one button, run it for two weeks, call the winner at 95% confidence, repeat. That advice is why plenty of stores finish a year of testing with a folder of screenshots and no revenue to show for it.

The mechanics aren't the problem. The order of the decisions is. Two weeks and a thousand visitors per variant can only detect a large effect, and a button colour is rarely a large one, so the test ends flat, gets filed as inconclusive, and the next quarter repeats the exercise on a headline. Meanwhile the two changes that would have moved money, a pricing rule and a checkout offer, never got tested because the tool couldn't reach them.

Here's the sequence I'd use instead. Decide the number you'll judge on before you build anything. Size the test before you launch it. Choose the biggest change your traffic can detect. And when the hypothesis lives on the payment or shipping step, stop pretending a theme experiment can reach it and use a route built for that page. Everything below is that sequence.

Table of Contents

• Shopify A/B Testing Methods at a Glance

• Metrics That Decide the Winner

• Design and Run the Test Without Ruining the Result

  • Step 1 Write one falsifiable hypothesis

  • Step 2 Measure a clean baseline

  • Step 3 Size the test on power, not on a traffic rule

  • Step 4 Change one variable and split traffic evenly

  • Step 5 Audit the instrumentation before you trust a single number

  • Step 6 Freeze the shop and stop only at the planned sample

  • Step 7 Read the interval, then keep a control group

• Testing Checkout When Themes Cannot Reach It

• Read the Result and Pick Your Next Test

Shopify A/B Testing Methods at a Glance

An A/B test on Shopify is a controlled experiment. Visitors are divided at random between the experience you run today and a modified version of it, whether that's a theme, a product page, a checkout flow or an offer, and both groups are measured over the same period so the comparison is fair (Convertica's Shopify A/B testing guide). Randomisation settles who sees what. Simultaneity settles when they see it. Break either one and you're measuring your ad mix or the calendar, not your change.

The method follows from the change, not from the app you already pay for. A hero banner, a rewritten product description or a sticky add-to-cart bar lives in the theme, so a theme experiment reaches it. Price rules, bundles and personalised collections need logic that runs before the page renders, which means the split happens server side. Anything on the shipping or payment step needs a route the theme never touches. Choosing the method that reaches the page is the first decision, and it's the one that causes the most projects to fail.

Before you open any app, write down the page where the change lives and pick the row that reaches that page. If the answer is checkout and you're not on Plus, the honest move is to move the hypothesis upstream, which is a design choice worth making deliberately rather than by accident.

One distinction trips up developers constantly. Shopify Functions run custom discount, delivery and payment logic inside checkout but they don't split traffic. A Function gives you the rule, not the experiment. You still need one of the splitting routes above to prove the rule earns more, and on Plus, Checkout Extensibility is the checkout-side option when a change has to sit on the payment or shipping steps.

Apparel brands get one more route worth naming. The image set is usually the highest-impact variable on a product page, and comparing flat-lay shots with on-model photography on the same PDP is a clean test of presentation, because the variant serves from the theme and the order value arrives intact.

Metrics That Decide the Winner

The rule I'd write above the analytics dashboard: a variant wins when revenue per visitor goes up and no guardrail breaks. Everything else either supports that sentence or gets ignored. Revenue per visitor divides total revenue by sessions, so it absorbs both halves of the commercial question at once. Did more visitors buy, and did each buyer spend more? A conversion lift on a smaller basket is a pay cut dressed as a win, and it's the most common false positive in Shopify testing.

Two rows belong in the primary slot. Revenue per visitor is the default and orders per visitor is its close relative, because both end at a completed order. Add-to-cart rate ends at a click, which is why it works as an early signal and fails as a judge. Average order value, checkout initiation and refunds are things you watch. Contribution margin is the final gate, and it's the one that saves you from a lift that sells more of the product you make least on.

Deciding well starts with knowing what normal looks like. Online Brand Growth's breakdown of ecommerce conversion rates is a useful second opinion when your own baseline looks too good or too bad to be real, and our guide to conversion rates in ecommerce covers the category benchmarks that turn a raw percentage into a target worth chasing.

The rule: if the primary metric is significant and a guardrail is broken, the test is a loss. Don't relitigate it because the conversion chart looked good on day nine.

Write the scoreboard before launch. One primary metric, named guardrails, and a damage threshold for each one. Do that and the results meeting takes fifteen minutes. Skip it and the meeting turns into an argument about which number mattered.

Design and Run the Test Without Ruining the Result

The failure mode in Shopify testing is almost never the statistical method. It is a test designed after launch. A merchant picks a variant, ships it, watches the dashboard, and discovers on day twelve that the hypothesis was vague, the baseline was polluted or the sample was never reachable. Here is the order I would use, with the reason each step exists and the slip that ruins it.

Step 1 Write one falsifiable hypothesis

A hypothesis you cannot lose teaches you nothing. Name the change, the audience and the expected effect in one sentence: a sticky add-to-cart bar on mobile product pages will raise revenue per visitor among phone sessions over four weeks. That sentence tells you which page to modify, which segment to analyse and which metric decides. The slip is a brief like "improve the product page", which becomes a five-element redesign and answers no question at all.

Step 2 Measure a clean baseline

Take the baseline conversion rate from a stable four-week window with no sitewide sale, no new ad channel and no theme work in progress. The baseline sets the sample size, converts your minimum detectable effect into an absolute number and gives you the variance you need later. Pulling a baseline from Black Friday week, or from a month when checkout was throwing errors, is how teams end up with a test that can never reach significance.

Step 3 Size the test on power, not on a traffic rule

Set the minimum detectable effect to the smallest lift that would change a decision, then plan at roughly 95% confidence and 80% power, calculating traffic per variant rather than for the test as a whole. At a 3% baseline, detecting a 10% relative lift at that confidence and power takes tens of thousands of sessions per variant. The ACCS glossary of A/B testing terms lays out the arithmetic, and the number is why small lifts rarely reach significance on modest traffic.

Small stores feel that number immediately. StorePilot's Shopify A/B testing guide gives the other end of the scale, where a store converting at 2% needs roughly 80,700 visitors per variant to detect a 10% lift. Divide the required sample by daily eligible traffic. If the answer runs past about eight weeks, the change is too small for the traffic you have, so test something bigger or switch to a higher-volume proxy metric such as checkout initiation.

When even a large change cannot be powered, the honest alternative is qualitative work. Ten session recordings and five customer interviews on a low-traffic store routinely buy more than another underpowered experiment, which is the argument for user experience research before another variant.

Step 4 Change one variable and split traffic evenly

Duplicate the live theme and change only the variable your hypothesis names. Split eligible traffic 50/50 and keep eligibility rules, attribution and checkout event firing identical on both sides. Shopify's guide to A/B testing treats 1,000 visitors per variation as a practical floor and is explicit that it is not a power threshold, because the requirement depends on baseline, traffic, minimum detectable effect and variance. The classic slip is a variant that differs in three ways, which is a redesign wearing a test's clothes.

The video below walks through the setup before you duplicate anything.

Step 5 Audit the instrumentation before you trust a single number

Check the split ratio first. If 54% of sessions landed in the variant, your assignment logic is broken and the result is already void. Then confirm that add to cart, checkout initiation and purchase events fire on both versions, that order value is captured rather than inferred, and that returning visitors are counted once instead of being re-randomised on every visit. Most "surprising" Shopify results die here, not in the statistics.

Step 6 Freeze the shop and stop only at the planned sample

Run at least two full weeks so weekday and weekend behaviour both appear, and freeze overlapping promotions, new app installs and theme edits for the duration. Stop at the predeclared sample or date, not when the variant looks good. Checking the dashboard every morning and calling it the moment p slips below 0.05 inflates your false-positive rate, and the maths does not care that the chart looked decisive on day nine. Two experiments that touch the same funnel step run one after the other, never together.

Practical rule: the only acceptable early stop is a guardrail breach, such as a checkout error spike or a refund surge. Business risk justifies stopping. A pretty chart does not.

Step 7 Read the interval, then keep a control group

Analyse the primary metric at the planned sample, read its confidence interval rather than the point estimate, and walk every guardrail. A lift whose interval spans zero is a flat result with better marketing. After rollout, keep a small control cohort for a few weeks wherever the mechanics allow it, because novelty, seasonality and audience mix can pull an apparent winner back toward the old baseline once the launch promotion ends.

Testing Checkout When Themes Cannot Reach It

Most Shopify testing guides are wrong about checkout, and not in a small way. They frame A/B testing as a theme exercise, then add a line saying checkout tests are impossible. The architecture moved under them. Shopify deprecated checkout.liquid for information, shipping and payment pages on August 13, 2024, and for thank-you and order-status pages on August 28, 2025, so JavaScript-based testing tools can no longer inject scripts or conditionally modify most checkout elements (Varify's breakdown of checkout testing on Shopify).

What survives is the merchant's real question. Can the offer that decides whether someone pays be tested at all? Yes, with a different route.

1. Move the hypothesis upstream. A shipping-threshold message, a payment-method reassurance line or a discount nudge can be served on the cart or a landing page and measured before the shopper reaches the payment step. Most checkout hypotheses survive the move.

2. Use Checkout Extensibility on Plus and keep checkout itself stable. Presidio has built advanced Shopify Plus checkout extensions that support complex promotional logic and experimentation without destabilising checkout.

3. Carry the identity through. Pass the session identifier from the cart step into the order so revenue can be attributed to the variant that produced it. Without that thread, you're comparing two populations and guessing which was which.

4. Judge on revenue per visitor, not completion. Completions can rise while average order value, upsell attachment or payment-method mix falls, and only a revenue denominator catches it.

The trap is aggregate reporting. Say a shipping-threshold banner lifts checkout completion while shrinking basket size. The completion chart reads as a win and the business loses margin. A country breakdown settles the argument quickly, because currency, tax rules and payment mix differ by market, and an aggregate number hides both the gain and the loss. The discipline of optimising the Shopify checkout is mostly about choosing the denominator before you fall in love with the numerator.

Read the Result and Pick Your Next Test

Expect losses. A programme that only reports wins is either testing trivia or measuring badly. Blend Commerce's review of Shopify A/B testing revenue impact shows how far published tallies drift apart. One historical compilation counted 159 tests, with 58.86% producing a winner, while a newer 60-test tracker reported 48 wins, 11 losses and 1 inconclusive result, an 81.4% win rate across the 59 decided tests.

The same review documents what a decisive result looks like. A 19-day experiment sent shoppers straight to the cart after they selected Add to Cart. The mobile variant reached 98% statistical significance, with conversion rate up 47.7% and revenue per visitor up 71.4%, and the winner was rolled out to mobile visitors only. It's also one UK homewares store, one device segment, one implementation and one 19-day window, so read it as evidence of what a clean, well-scoped win looks like, not as a target for your own store.

The gap between 58.86% and 81.4% is the case for reporting sample definition, eligibility rules, significance thresholds and inconclusive outcomes alongside any result. An agency quoting an average uplift without those four things is selling a number rather than a result.

Then apply the same four moves after every test. Write the meaning in one sentence, because if you can't, you haven't learned anything transferable. Log the hypothesis, primary metric, sample size and outcome in a test register so the knowledge outlives whoever ran the test. Choose the next test from the largest drop in the funnel, not from the most interesting idea in the room. And read the register before you brief anything new, so a change that lost six months ago doesn't get proposed again with fresh enthusiasm.

A store that runs twelve properly sized tests a year learns more than one that runs forty small ones, and it ends the year with a decision register instead of a folder of screenshots. Fewer tests, bigger changes, one number that decides.

Presidio runs conversion optimisation and structured experimentation for Shopify and Shopify Plus brands, from sample-size planning and metric design through to checkout extensibility builds for offers a theme can never reach. If your test history is a stack of inconclusive results, we can audit the funnel and tell you which experiments your traffic can support. Start by booking a call with Presidio to take control of your Shopify storefront.