So what is A/B testing?
Your conversion rate is the share of visitors who buy. Conversion rate optimization (CRO, if you want the fancy name) is the ongoing work of lifting that number: more sales from the traffic you ALREADY have, with no extra ad spend to get it.
An A/B test is the safest way to do that work. Your current version is the control, the new version is the treatment, your traffic is split between them at the same time, and you compare the results. Because both versions run side by side, a good week or a dead week hits them equally.
Good things to A/B test
The tests I've seen work really well are the BIG messaging and imagery ones.
- A totally different headline: what you say first, at the top of your homepage or product page
- A new hero image: the first picture people see
- Different product photos: polished against real-life, product-only against on a person
- A new way of explaining your offer: what's included, why it's different, who it's for
- A button color
- One word in a subheading
- A few changes at once, because when a version wins you won't know what did it
Tiny tweaks need heaps of traffic before there's a clear winner. A big change in what you say or what you show is something people actually notice, so a difference can show up sooner.
Real tests from other brands
Now for the fun part: what happened when other brands tested this stuff.
Use them for ideas on what to test, and expect your own numbers to look different.
Imagery tests
Polished homepage photos
Raw, customer-style photos
Renders and heavily designed graphics
Realistic on-body photos
Smaller product images
Bigger product images, with the description showing on hover
Fulton's homepage had always been the polished one, and the raw content was saved for ads. They flipped it, and the less "on brand" version made MORE money. 👀
Messaging tests
The homepage as it was
A new "what makes us different" section: a review badge and a table comparing the brand with others
A plain list of benefits with emoji
A grid of benefits with icons
Value propositions and testimonials further down the page
The same content moved up to sit right under the hero image
The word "Autoship"
The word "Subscribe"
And look at how small some of these are to build! Nobody redesigned a whole website here. One brand changed a single word. Another moved content it ALREADY had further up the page. So a change your customer really notices doesn't have to be a big job for you.
Check profit before you pick a winner
A version can win on sales and still leave you with less. This one shows it:
A science-led landing page
A page built around a buy one, get one free offer
More people bought, and each order made less. Whether that's a win depends on your margins, which is why the winner of any test gets picked on PROFIT. Check orders, average order value and your margin for each version before you call it. 💸
How to set up a test in Shopify
Shopify has testing built in. The feature is called Rollouts, and the A/B test inside it is called an Experiment.
Which plan you need: Rollouts is available on the Basic plan or higher, and Experiments are available on the Grow plan or higher. Vintage themes can't use it. The quickest check is your own admin: follow the path above and see whether Experiment is in the rollout type menu. Source: Shopify Help Center
If your plan doesn't include Experiments, I like using Visually for this.
Build the version you want to test
- From your Shopify admin, go to Online Store > Themes.
- Duplicate your live theme and rename the copy something obvious, like "Test: new hero image".
- Open the copy in the theme editor, make your ONE change and save it. Leave it unpublished.
Create the Experiment
- Go to Markets > Rollouts and click Create rollout.
- In the Name section, give it a name you'll still recognize in a month.
- In the rollout type menu, select Experiment.
- In the Changes section, click Add changes.
- Choose Online store, and under Replace the current theme select the copy you made.
- Click Apply.
- In the Audience section, check the Traffic split. It starts at 50% control and 50% treatment, and that's a good place to leave it.
- In the Scheduling section, click Set end date and pick one. A new Experiment is set to end 90 days after it starts, so choose your own date.
- If you see Resolve rollout conflicts, fix what it flags. Then save.
Once it's running, leave both themes alone! Editing either version while the Experiment is active can affect the results. If something breaks, open the rollout and click Pause, then Resume when it's sorted.
How to read your results
Go to Markets > Rollouts and click your Experiment. For a theme test, Shopify shows:
- Conversion rate over time: the headline number
- Add to cart rate: are more people starting to buy?
- Reached checkout rate: are they getting further?
- Bounce rate: are more people leaving straight away?
Those are all rates, so open your regular analytics too and compare orders, sales and average order value for the test period.
What makes a result easier to trust:
- Let it run through full weeks: weekday shoppers and weekend shoppers behave differently, so you want both in there. The tests above ran anywhere from 7 to 36 days.
- Don't call it early: an early lead can flip. Set your end date and check in weekly.
- Look at how many orders each version has: if each side only has a handful, the gap between them could easily be luck.
If the two versions finish neck and neck, that tells you something too: that change doesn't matter much to your customers. Keep the one you prefer and test something bigger next.
To finish the test, open your Experiment and click End. Choose Apply changes to make the new version permanent, or Roll back changes to keep what you had, then click End to confirm. After you apply a rollout you can't revert it, so keep your original theme in your theme library as a backup.