Statistical method & guardrails — found from sitemap — Statsig

Cited by

Facts read from this source

• Stats engine overview Report an error

  “The Statsig Stats Engine is the set of statistical calculations behind the experiment results you read in Pulse. It turns exposure and metric data into a metric delta, confidence interval, and p-value for each metric, applies variance reduction, and supports frequentist, SPRT, and Bayesian analysis methods.”

• Metric deltas Report an error

  “Metric deltas: The difference in mean metric value between the test and control groups, in absolute and relative terms.”

• Confidence intervals Report an error

  “Confidence intervals: The gray, red, and green bars in Pulse, which represent the uncertainty in each metric delta.”

• P values Report an error

  “p-values: The probability of observing a metric delta at least as extreme as the one measured, if the treatment has no effect on that metric.”

• Topline projected impact Report an error

  “Topline and projected impact: The daily effect of the experiment on a company metric, and the estimated effect if you launch the test group to all users.”

• Variance reduction Report an error

  “Variance reduction: The techniques Statsig applies so the same effect reaches significance with fewer users.”

• Cuped Report an error

  “CUPED: Uses each user's pre-exposure data to reduce variance. Statsig applies it automatically.”

• Pre experiment bias Report an error

  “Pre-experiment bias: How Statsig detects groups that already differed before exposure and flags the affected metrics.”

• Frequentist sequential testing Report an error

  “Frequentist sequential testing: Adjusts p-values and confidence intervals so you can read results early without inflating false positives.”

• Sprt Report an error

  “Sequential probability ratio tests: Compares the likelihood of your data under the alternative and null hypotheses, and can accept either one.”

• Bayesian experiments Report an error

  “Bayesian experiments: Reports chance to beat control and expected loss, with optional informative priors.”

• Standard error variance Report an error

  “Standard error and mean variance: How much each group's mean would vary across repeated samples. This is the input to every confidence interval and p-value.”