Shopify A/B Testing: Run a Homepage Experiment Without an App

You can A/B test a Shopify homepage with the native Experiment option in Rollouts if your store is on Grow or higher. Create a control and a treatment, make a focused change to the published theme within the rollout, then compare the available metrics before choosing to roll back or apply the change. Rollouts are available on Basic or higher, but Experiments require Grow or higher. The workflow here is for a Shopify Online Store theme, not a headless storefront or POS.

A/B testing compares a control, the existing experience, with a treatment, the changed experience. Shopify’s Rollouts requirements explain plan and permission eligibility. Its rollout types distinguish an Experiment from a Launch, which applies a change rather than comparing it with a control.

What you need before starting

This procedure assumes an existing Shopify Online Store on Grow or higher, a published theme that is not vintage or using the Canvas editor, and two homepage template sections that the theme allows you to reorder. It also assumes you can edit rollouts and the theme. A store owner or organization owner can manage rollouts; a staff role needs Markets > Rollouts and the relevant Online store > Themes and Online store > Edit code permissions.

Rollouts do not support vintage themes or Liquid-template changes. A theme with a Canvas badge opens in a different editor, so the theme-editor instructions below do not apply to it. Theme and checkout rollout changes apply to the Online Store; this procedure does not test headless storefronts or POS visitors.

Steps to run a Shopify homepage A/B test

1. 1. Write the hypothesis and measurement plan. Define the customer problem, eligible audience, one primary outcome, guardrails, planned sample, representative business window, and decision rule. This makes the test answer a specific business question rather than simply produce a dashboard result.

For example, you might hypothesize that moving an existing product-discovery section above an editorial section helps homepage visitors find products and increases add-to-cart rate without worsening purchase conversion. That is a hypothesis to test, not a predicted or guaranteed result. Keep the treatment focused on one conceptual change so the result is easier to interpret.

Plan the sample before launch. The required sample depends on the baseline rate, the smallest effect you want to detect, statistical significance, statistical power, and the number of variants. Estimate the run window using realistic eligible experiment traffic, and include normal business cycles. SDG Engineering’s Shopify testing guidance explains these planning considerations. There is no single traffic threshold or fixed duration that suits every store. If the available traffic cannot support a useful sample in a practical window, research the problem, sharpen the hypothesis, or fix a known defect directly instead of treating a weak result as conclusive.

1. 2. Confirm that the store and theme are eligible. Check that the plan is Grow or higher, that your account has the required access, and that the published theme fits the editor workflow. Basic includes Rollouts, not Experiments. This check prevents building a procedure your store cannot run.
2. 3. Duplicate the published theme as a backup. In Online Store, open the published theme’s actions menu and choose Duplicate. Confirm that a copy named “Copy Of” followed by the original theme name appears on the Themes page. Keep this as a safety copy; the treatment in this method is edited inside the rollout, not by publishing the duplicate to everyone.

Shopify allows up to 20 themes in the admin before another duplication requires space. Deleting a theme is permanent, so review what is stored before removing anything. See Shopify’s theme duplication instructions.

1. 4. Create and save an Experiment rollout. Go to Markets > Rollouts > Create rollout, enter a descriptive name, and select Experiment rather than the default Launch. Save the rollout before editing the main theme. An Experiment compares a treatment with a control; a Launch does not provide that A/B comparison.
2. 5. Add the homepage change to the rollout. In Changes, choose Add changes > Online store. Under Prepare changes to your current theme, select Edit theme. In the editor, confirm the rollout context in the theme-name menu, then use the template menu to open the homepage. Under Sections, find the two existing sections in the Template area and drag the product-discovery section above the other section using its drag handle. This puts the proposed treatment in the rollout instead of publishing a new homepage to all visitors.

Use homepage template sections for a homepage-only layout test. Header and footer groups are shared across store pages. If a section returns to its original position after you drag it, that location is not allowed by the theme. Click Save in the theme editor, then Apply in the Add changes dialog. Shopify’s rollout setup instructions cover the main-theme edit path, and its section editing guidance explains section groups and drag-and-drop limits.

1. 6. Set the audience, traffic allocation, split, and schedule. In Audience, review Eligibility to see which markets the rollout applies to. Set Traffic allocation to control how much eligible traffic enters the rollout, then set Traffic split to divide that experiment audience between control and treatment. Keeping these settings distinct gives you a realistic estimate of traffic per arm.

For example, 50% Traffic allocation combined with a 50% split sends 25% of the original eligible audience to each arm: 0.50 × 0.50 = 0.25. The other 50% stays outside this experiment. This arithmetic assumes overlapping rollouts do not change effective allocation. Counting each arm as half of all eligible visitors would overstate its traffic in this setup.

In Scheduling, set a future start date and time, then set the end date and time. Shopify uses the store timezone in Settings > General. A new Experiment defaults to an even split and an end date 90 days after it starts; treat that as a setup default, not evidence that 90 days is enough for your sample or business cycle. Check the displayed end behavior before saving. Shopify describes these controls in its rollout creation guide.

1. 7. Preview both layouts and review rollout conflicts. Use the theme editor’s mobile preview control and switch back to desktop to inspect each layout. Check the intended section order, links, and unchanged content. Resolve any “Resolve rollout conflicts” message before saving; read a “Review rollout conflicts” message for possible visibility or traffic effects. When checking the storefront in a private window, remember that Incognito does not prevent browser fingerprinting.
2. 8. Save the schedule and check that the rollout becomes active. Save the completed settings. A saved rollout with a future start date is Scheduled; an ongoing rollout is Active. Open it from Markets > Rollouts to review analytics and traffic allocation. If it remains Draft, check whether it has a start date and changes.
3. 9. Monitor the experiment without changing its design mid-run. Watch traffic allocation, data quality, storefront errors, performance, campaigns, and inventory incidents. Avoid changing the treatment, control, audience, or primary decision rule while the experiment is active, because changing the experience can affect the result.

Uncustomized parts of the rollout theme stay synchronized with the published theme, so an unrelated live-theme change can still matter during the experiment. Overlapping rollouts can also change effective traffic; Launch rollouts receive traffic before Experiments. If traffic looks lower than expected, inspect the rollout’s traffic details and conflicts rather than assuming the configured allocation is the effective allocation.

1. 10. Evaluate the planned outcome and guardrails. Review results after the planned sample and representative business window. Consider the effect size, uncertainty, traffic balance, and any operational incidents against the decision rule you set before launch. A favorable directional change alone does not establish a winner, and a neutral result does not prove the versions perform identically.

The native theme Experiment dashboard has a fixed metric set: conversion rate over time, bounce rate, reached-checkout rate, and add-to-cart rate. Shopify defines conversion rate as the percentage of Online Store sessions that lead to an order; reached-checkout rate compares sessions reaching checkout with total sessions; add-to-cart rate concerns visits with items added to cart; and bounce rate covers visits ending after one pageview. The rollout analytics documentation says metrics cannot be customized. Revenue per session, margin, and arbitrary custom events are not among the listed theme Experiment metrics. Launch reporting is different, so do not treat its sales or average-order-value metrics as Experiment dashboard options.

1. 11. End the rollout and choose what happens to the homepage change. While it is Active, go to Markets > Rollouts, open the experiment, and click End. Choose Roll back changes to retain the prior experience, or Apply changes to keep the treatment permanently. Click End again to confirm, then check the public homepage against your decision.

Shopify warns that applying the changes cannot be reverted through that rollout action. If an Experiment reaches its scheduled end and rolls back, the treatment is not automatically kept as a winner. Archiving is also not a pause: an archived rollout cannot be restarted, and its theme changes are permanently deleted. Review Shopify’s rollout management instructions before ending or cleaning up the experiment.

Common mistakes that can weaken the result

• Confusing plan access: Basic supports ordinary Rollouts, but running Experiments requires Grow or higher.
• Treating the 90-day default as a sample-size recommendation: plan the required sample and run window from your own baseline and eligible experiment traffic.
• Counting the split but ignoring allocation: the experiment split applies to traffic allocated to the rollout, not necessarily to all eligible visitors.
• Changing the treatment or decision rule during the run, or stopping at the first favorable fluctuation, can make the result harder to interpret.
• Expecting a custom metric or automatic winner deployment: theme Experiment metrics are fixed, and the merchant chooses whether to roll back or apply the change when ending an active rollout.

How to verify the experiment is complete

Before calling the test complete, confirm that the rollout reached its planned sample and business window, that the intended control and treatment received traffic, and that material campaigns or inventory incidents are accounted for. Then confirm the rollout is ended and the public homepage matches the choice you made: the original experience after rollback, or the treatment after permanent application. Record the outcome and its limitations so the next homepage decision starts with what this experiment actually established.

Quick reference

• Eligibility: Grow or higher, appropriate rollout and theme permissions, and a supported Online Store theme.
• Setup: Markets > Rollouts > Create rollout > Experiment; save before adding the Online store theme change.
• Traffic: set Traffic allocation and Traffic split separately, then account for overlapping rollouts.
• Finish: assess the fixed Experiment metrics against your plan, then explicitly roll back or apply the treatment.