Skip to content

How Crovio decides a winner

The three stages every test goes through before Crovio calls it, and the rules you can adjust.

On this page

Crovio never calls a test on a fixed number of orders. A big store can need thousands of orders per group to tell a 3% lift from noise; a small store needs far fewer to spot a 30% lift. So every test goes through three stages, checked on the cumulative result at the end of each day.

1. Collect enough data

  • At least a shortest run (7 days by default) so every weekday is counted.
  • Enough data in every group to measure a lift the size of the one showing, at 80% power and 95% confidence. The smaller the lift, the longer this takes. Crovio shows the target, for example "A lift this size needs about 1,200 orders per group".
  • Optionally, a fixed minimum per group that you set (off by default).

2. Noise settles

The same group leads, with a strong chance to win (90% by default), and the lift barely moves for several days in a row (5 by default). That's the point where early swings have died down. The test shows Settled, confirming.

3. Confirm it holds

The test keeps running for a few more days (3 by default) until every group has more data than when it settled (20% more by default), or a maximum number of days for slow tests (7 by default). If the result moves, the test goes back to collecting and shows Results moved, still settling.

The call

Only a result that held through confirmation is called:

  • [Variant] is likely better: chance to win at or above the call bar (95% by default).
  • Control is likely better: control beats every variant at the same bar.
  • No meaningful difference: the 95% range rules out lifts bigger than 5% either way (checked after twice the shortest run), or the test reached its time limit (42 days by default) with no leader. Safe to end and keep whichever version you prefer.

Presets

Change the rules in Settings → A/B testing. They apply to every test's results.

RuleBalanced (default)CautiousFast
Shortest run7 days14 days7 days
Leader holds steady for5 days7 days3 days
Counts as leading at90%90%85%
Keep running at least3 days7 days2 days
Until every group has20% more data30% more data10% more data
But no longer than7 days14 days5 days
Call a winner at95%97%90%
No meaningful difference within5%3%8%
Time limit42 days56 days28 days
  • Balanced: recommended for most stores.
  • Cautious: fewer false winners, slower calls. For big stores and big decisions.
  • Fast: quicker calls with more risk of a fluke. For low-traffic stores.

Pick Custom and use Fine-tune the rules to set each one yourself, including Require a minimum per group. Reset to Balanced undoes your changes. A/B testing settings.

The maths, briefly

Win chance is a Bayesian probability (flat prior) that a group beats control. Revenue per visitor and average order value use a normal approximation; the interval is the 95% range for the lift. Crovio also checks that visitors split the way you set them (sample ratio mismatch) and warns you if not.

Was this page helpful?