Statistical Significance
Statistical significance is a measure of how likely an observed difference between two variants is to be real rather than the product of random chance.
A result is typically called statistically significant when the probability of seeing a difference this large, if there were truly no difference at all, falls below a threshold agreed on before the test started. It's a statement about confidence in the result, not about how large or commercially meaningful the difference is.
Statistical significance and practical significance are different questions. A large enough sample can make a tiny, commercially irrelevant difference statistically significant, while a genuinely important difference can fail to reach significance if the sample is too small — which is exactly what minimum detectable effect describes.
Checking for significance too early or too often during a running test inflates the chance of a false positive — a difference that looks real in the moment but disappears with more data. This is one of the reasons a test's required sample size and duration are set in advance rather than decided by watching results as they come in.
See it in the product
How NavonaAI implements this, in detail.
Related terms
Looking for another term?
Browse the full glossary, or see what NavonaAI adds back to your store.