Holdout Group
A holdout group is a set of shoppers deliberately excluded from an intervention so their behavior can serve as the baseline it's measured against.
Without a holdout group, a business can only see what happened to shoppers who received an intervention — there's nothing to compare it to. The holdout group shows what would have happened to a comparable set of shoppers who didn't see it, which is what makes incremental lift measurable rather than assumed.
Assignment to a holdout group needs to be random, or at least uncorrelated with anything that would make those shoppers behave differently from the group that received the intervention. Otherwise the comparison is confounded, and any difference in outcome could be explained by the two groups being different to begin with, not by the intervention.
A holdout is typically kept as a small, ongoing slice of traffic rather than run once — so lift can be measured on a rolling basis, not just during a single test window.
See it in the product
How NavonaAI implements this, in detail.
Related terms
Looking for another term?
Browse the full glossary, or see what NavonaAI adds back to your store.