Statistical method & guardrails — found from sitemap — GrowthBook

Cited by

Facts read from this source

• Stat engines Report an error

  “GrowthBook provides both Bayesian and frequentist approaches to experiment analysis.”

• Default engine Report an error

  “We default to Bayesian statistics because they provide a more intuitive framework for decision making for most customers”

• Engine selection level Report an error

  “You can choose between the two statistics engines at the Organization or Project level.”

• Bayesian prior default Report an error

  “At GrowthBook, we default to an improper, uninformative prior.”

• Bayesian proper prior Report an error

  “By default, we use a Normal distribution with mean 0 and standard deviation 0.3.”

• Proper prior toggle Report an error

  “You can easily turn on proper priors by visiting the organization settings, going to the Bayesian engine settings, and turning on the “Proper Prior”.”

• Inferential metrics Report an error

  “GrowthBook uses fast estimation techniques to quickly generate inferential statistics at scale for every metric in an experiment - Chance to Win and Relative Uplift (along with Absolute Change and Scaled Impact).”

• Chance to win threshold Report an error

  “You typically want to wait until this reaches 95% (or 5% if it’s worse).”

• Frequentist t test Report an error

  “The current frequentist engine computes two-sample t-tests for relative percent change”

• Cuped both engines Report an error

  “In fact, tools like CUPED are available for both engines.”

• Srm check Report an error

  “Sample Ratio Mismatch (SRM) detects when the traffic split doesn’t match what you are expecting (e.g. a 48/52 split when you expect it to be 50/50)”

• Multiple exposures check Report an error

  “Multiple Exposures which alerts you if too many users were exposed to multiple variations of a single experiment (e.g. someone saw both A and B)”

• Guardrail metrics Report an error

  “Guardrail Metrics help ensure an experiment isn’t inadvertently hurting core metrics like error rate or page load time”

• Min data threshold Report an error

  “Minimum Data Thresholds so you aren’t drawing conclusions too early (e.g. when it’s 5 vs 2 conversions)”

• Variation id mismatch Report an error

  “Variation Id Mismatch which can detect missing or improperly-tagged rows in your data warehouse”

• Suspicious uplift detection Report an error

  “Suspicious Uplift Detection which alerts you when a metric changes by too much in a single experiment, indicating a likely bug”

• Quality checks customizable Report an error

  “Many of these checks are customizable at a per-metric level.”

• Multiple testing corrections scope Report an error

  “GrowthBook only provides multiple testing corrections for the frequentist engine”

• Dimension grouping Report an error

  “In the country example, only the top 20 countries will be shown individually. The rest will be lumped together into an (other) category.”

• Open source Report an error

  “The implementation is fully open source under an MIT license and available on GitHub.”

Scores citing this record

• The CRO Consultant Statistical method & guardrails
• The Data Protection Officer Statistical method & guardrails
• The E-Commerce Manager Statistical method & guardrails
• The Growth Lead Statistical method & guardrails
• The Product Engineer Statistical method & guardrails
• The Skeptic Statistical method & guardrails