A/B Testing

Your discount settings are a guess until you test them

Every incentive amount, minimum order value, and stacking rule you picked was a decision made without data. A/B testing runs two setups against each other on your own traffic and tells you, honestly, which one actually converts better.

See Plans

Guessing costs you either way

Every prevention setting is a trade-off. A bigger incentive convinces more shoppers to stay — and it eats into margin on every order it touches, not just the ones it changed. A minimum purchase amount can filter out buyers who would have converted anyway. Letting the incentive stack with other discounts can compound cost you never planned for.

Pick any of these settings by feel and you’re either giving away more than you need to, or holding back an incentive that would have added revenue. There’s no way to know which without comparing two setups against each other, on the same store, at the same time.

That’s what A/B testing is for: instead of deciding by instinct, two configurations run side by side on real shoppers, and you read off which one actually wins.

How a test is set up

Six things are true of every test, by design — not by configuration.

Pick the goal the test is judged on

Cart-to-order rate or total abandonment rate. Whichever you choose, that one number decides the winner — nothing else.

Set up Group A and Group B

Group A starts as your current live settings. Group B is the setup you want to test against it. The fields you can change between the two are listed below.

See how long it will actually take, before you start

The setup form estimates a timeline from your store’s own recent cart volume and conversion rate. If your traffic is too thin for the difference you’re trying to detect, it says so up front, instead of leaving you to find out weeks into a test that was never going to reach an answer.

Choose how it ends

Manual — you stop it whenever you decide. Or volume-based — it stops automatically once a set number of carts have been split across the two groups.

Traffic splits 50/50, automatically

Every shopper is assigned to Group A or Group B at random, in equal proportion, for the life of the test.

One test per store, at a time

Starting a second test while one is already running is blocked. Two overlapping tests would draw from the same shoppers, and neither result could be trusted.

What you can test

Group A and Group B can differ on exactly these fields — nothing else.

Incentive type. A percentage discount, a fixed-amount discount, or free shipping.
Incentive amount. How much of a discount each group offers.
Minimum purchase amount. Whether the incentive requires a minimum order value to unlock, and what that minimum is.
Stacking rules. Three separate toggles — whether the incentive combines with other order discounts, product discounts, and shipping discounts.
Product & collection exclusions. Which products or collections sit outside the incentive entirely, set independently for each group.

Results, read honestly

A gap between two numbers is not automatically a winner. The results page is built around that fact.

A confidence interval, not a raw number

Every reported gap between Group A and Group B comes with a confidence interval, so you see the range the true difference is likely to fall in — not a single point estimate that could just be noise.

It says when a gap isn’t real yet

Until a result clears statistical significance, the page labels the gap as within normal random variation, rather than declaring a leader off a number that could easily have gone the other way.

What your current traffic can even detect

Alongside the verdict, you see the smallest true difference this test’s traffic could reliably tell apart from noise. A real effect smaller than that will look flat — the page says so, instead of letting you read "no difference" as "nothing changed."

How much more data a smaller effect needs

If the result isn’t significant yet, you get an estimate of how many more carts — and roughly how many more days at your traffic — it would take to reliably detect a smaller effect, instead of an open-ended "keep waiting."

When a test does have a clear winner, promoting it is one action. The winning setup replaces your current live settings and becomes the new default — the losing setup is retired, not left running alongside it.

Frequently Asked Questions

How long does a test take to finish?

It depends entirely on your store’s traffic and the size of the difference you’re trying to detect. The setup form estimates this from your own recent cart volume before you start the test, rather than leaving you to guess.

Can I run more than one test at a time?

No — one active test per store. Running two at once would mean both draw from the same shoppers, which contaminates the results of each.

What if the two groups end up close?

The results page tells you directly when a gap is within normal random variation instead of calling a winner off a raw number, and shows the smallest real difference your current traffic could reliably detect.

What happens once a test ends?

If there’s a clear winner, promote it in one click and it becomes your new default. If there isn’t, you keep your original settings and can start a new test.

What can I actually change between Group A and Group B?

Incentive type, incentive amount, minimum purchase amount, whether it stacks with other order, product, or shipping discounts, and which products or collections are excluded.

Stop guessing. Start testing.

A/B testing is included on every NavonaAI plan.

See Plans