Statistical method & guardrails — found from sitemap — Statsig

Cited by

Facts read from this source

• Bh purpose Report an error

  “The Benjamini-Hochberg (BH) procedure adjusts the significance level when a scorecard tests many metrics, so that only a controlled share of the results you call significant are false positives. That share is the false discovery rate.”

• Bh vs bonferroni Report an error

  “BH therefore rejects more null hypotheses than Bonferroni for the same p-values. Use BH when a scorecard has many metrics and a small, controlled share of false positives is acceptable. Use Bonferroni when any single false positive is costly.”

• Bh configuration Report an error

  “You can enable the BH procedure for individual experiments, or configure global Experiment Settings to use it by default.”

• Bh scope options Report an error

  “Test groups (multiple treatment hypotheses): For each metric, Statsig aggregates the p-values from each variant and runs the BH procedure on that list. Metrics in the scorecard: For each variant, Statsig aggregates the p-values from each metric and runs the BH procedure on that list. Both test groups and metrics: Statsig aggregates all p-values and runs the BH procedure once.”

• Bh breakdown exclusion Report an error

  “Statsig doesn't apply the BH procedure to the p-values of event-dimension or user-property breakdowns of an experiment metric. Statsig compares only the top-line metric results to the new significance level.”

2 facts read from this page are not shown because they could not be confirmed on the page as captured.