Experiment types & delivery — found from sitemap — Statsig

Cited by

Facts read from this source

• Experiment requirements Report an error

  “Before Statsig can create an experiment, it needs a name, a hypothesis, at least one target application, and a scorecard with at least one primary metric.”

• Experiment types Report an error

  “Experiment Type: The default is Standard A/B/n. Statsig also supports other types, such as Switchback Tests and A/A tests.”

• Layers Report an error

  “By default, your experiment runs in its own layer. A layer manages multiple experiments and feature gates together.”

• Templates Report an error

  “Template: Pre-fill the experiment's configuration from a saved template.”

• Scorecard metrics schedule Report an error

  “Statsig computes scorecard metrics daily, and they're eligible for advanced treatments such as CUPED and Sequential Testing.”

• Scorecard metrics types Report an error

  “Primary Metrics: The metrics you expect the experiment to directly affect. Secondary Metrics: Metrics you monitor for unintended side effects.”

• Hypothesis template Report an error

  “We believe that [change or feature] for [user segment] will [desired outcome] because [reason or insight]. We will measure success using [primary metric], and monitor [guardrail metrics] to ensure no negative impact.”

• Allocation Report an error

  “Allocation is the percentage of eligible users that Statsig assigns to your experiment, up to 100%.”

• Allocation decrease warning Report an error

  “Decreasing allocation biases group allocation and pollutes your metric results, so don't decrease allocation without resetting the experiment.”

• Targeting Report an error

  “For more advanced targeting (for example, progressive rollouts), or to keep targeting criteria when you launch the experiment, reference an existing feature gate.”

• Groups parameters Report an error

  “Groups are the variant labels in the console, such as Control or Test. Parameters are the values your code reads to change the product or feature's behavior.”

• Group rename safe Report an error

  “Your code reads parameter values through the SDK and never reads group names, so you can rename or re-describe a group at any time without a code change or any effect on what users experience.”

• Id type randomization Report an error

  “By default, experiments randomize users by userID. To use a different ID type, such as stableID for device-level experiments, follow the user-level experiment steps with one change before you click Create:”

• Id mapping warehouse native Report an error

  “Warehouse Native: Supports ID mapping between different identifier types (for example, stableID to userID) through Entity Property Source configuration.”

• Id mapping cloud Report an error

  “Cloud: Doesn't support mapping between ID types. Experiments started with stableID analyze only events with stableID, and experiments started with userID analyze only events with userID.”

• Isolated experiments Report an error

  “To create an experiment that excludes users exposed to other experiments, follow the user-level experiment steps with one change before you click Create: Select the Layer option, then select an existing Layer or create a new one.”

• Bucketing salts Report an error

  “Only Project Administrators can copy bucketing salts. To copy a bucketing salt, click the three-dot menu button on the Experiments page.”

• Experiment actions Report an error

  “Add Decision Framework: Attach a decision framework that defines success criteria and ship-versus-iterate logic for the experiment.”

• Default confidence interval Report an error

  “By default, the Experiment Results section displays 95% confidence intervals without Bonferroni correction.”

• Bonferroni correction Report an error

  “Bonferroni Correction: Apply this correction to reduce the risk of false positives in experiments with multiple test groups. Statsig divides the significance level (α) by the number of test groups.”

• Target duration Report an error

  “A target duration is optional, but it helps you wait long enough for the experiment to reach full power. You can set the target as a number of days or a number of exposures.”

• Results computation window Report an error

  “Target durations longer than 90 days: By default, Statsig computes Experiment Results for the first 90 days, though the experiment itself can run longer.”

• Target notifications Report an error

  “Statsig also notifies you through email and Slack (if integrated) when you reach the target.”

• Hypothesis advisor Report an error

  “Hypothesis Advisor gives instant feedback on experiment hypotheses and flags what's missing. Admins can set custom requirements that Statsig uses to guide experimenters toward more complete hypotheses.”

• Hypothesis advisor enablement Report an error

  “This Statsig AI feature is off by default. Enable it from Settings > Experiment > Project > Statsig AI.”

• Feature gate alternative Report an error

  “If you only need to roll out a single change safely, without measuring variant lift, create a feature gate instead.”

• Power analysis calculator Report an error

  “Use the Power Analysis Calculator to determine which target works best for your metrics.”

1 fact read from this page is not shown because it could not be confirmed on the page as captured.