Skip to content

A/B testing rules

Choose how careful Crovio is before it calls a test, with presets or your own rules.

On this page

Settings → A/B testing decides when Crovio calls a test. Crovio waits for the noise to settle, keeps collecting to confirm the result holds, then calls it. The rules apply to every test's results, including tests already running.

Pick a preset

  • Balanced: recommended for most stores.
  • Cautious: fewer false winners, slower calls. For big stores and big decisions.
  • Fast: quicker calls with more risk of a fluke. For low-traffic stores.
  • Custom: your own rules, set below.

Click Save rules. Reset to Balanced undoes your changes.

Fine-tune the rules

StageRuleWhat it does
1. Collect enough dataShortest runNever call a test sooner. Keep it to whole weeks so every weekday is counted
Require a minimum per groupOptional. A fixed number of orders (or goal completions) every group must reach, on top of the target Crovio sets from the size of the lift
2. Noise settlesLeader holds steady forDays in a row the same group must lead with a steady lift
Counts as leading atChance to win a group needs before it counts as the leader
3. Confirm it holdsKeep running at leastExtra days after the result settles
Until every group hasMore data than it had when the result settled
But no longer thanFor slow tests that would take weeks to gather the extra data
4. DecisionCall a winner atChance to win needed once confirmation is over
No meaningful difference withinCalled flat once the range rules out lifts bigger than this either way
Time limitNo leader by then means any difference is too small to measure at your traffic

The values for each preset are in How Crovio decides a winner.

Was this page helpful?