Experiment statistics
How Gamebeast calculates experiment results: per-unit metrics, the z-test and Welch t-test, 95% confidence intervals, significance, minimum sample sizes, and the sample ratio mismatch check.
Gamebeast calculates experiment results with standard frequentist tests: a two-proportion z-test for conversion metrics and a Welch t-test for counts and values, each with a 95% confidence interval and a two-sided p-value. It also checks every experiment for sample ratio mismatch, a sign that assignment went wrong. This page is the exact method, so you can check a result by hand.
Results are available from the AI assistant and over MCP (ask how an experiment is doing), and you can compare groups yourself in the Query Builder by breaking any metric down by experiment group.
What is counted
Every metric is computed per enrolled unit, a player or a server depending on the experiment.
- The denominator is every unit enrolled in the group, including units that never fired the event, which count as zero. This keeps groups comparable: a change that stops players reaching an event shows up as worse, not as a smaller, better-looking group.
- The numerator counts only markers sent after the unit first entered the experiment, and only inside the experiment's run.
- The window ends at the earliest of the configured end date, the time the experiment was terminated, and now. After termination everyone is back on the base configuration, so later markers say nothing about the groups.
Metric kinds
| Kind | Measures | Test |
|---|---|---|
| Conversion | Share of units that fired the event at least once | Two-proportion z-test |
| Count per unit | Average number of times each unit fired the event | Welch t-test |
| Sum per unit | Average total of a numeric value per unit, such as spend | Welch t-test |
Each metric can be narrowed with marker property and argument filters. Up to 5 metrics are measured at once.
How each group is compared with the control
Every group is compared with the control group, which is the first group unless you pick another.
- Absolute lift is the treatment mean minus the control mean, with a 95% confidence interval.
- Relative lift is the absolute lift divided by the control mean, with a 95% confidence interval from the delta method. It isn't shown when the control mean is zero.
- The p-value is two-sided. For conversion, the z-test uses the pooled standard error for the test statistic and the unpooled standard error for the interval. For counts and values, Welch's t-test doesn't assume equal variances; its degrees of freedom come from the Welch–Satterthwaite equation.
The verdict
| Verdict | When |
|---|---|
| Significant, positive | p < 0.05 and the treatment mean is higher |
| Significant, negative | p < 0.05 and the treatment mean is lower |
| Inconclusive | p ≥ 0.05 |
| Insufficient data | Fewer than 100 units in either group, or the test is undefined (no variance) |
The verdict compares each group with the control at a 5% significance level. It doesn't correct for testing several metrics or several groups at once, so with many comparisons expect some to come out significant by chance, and name your primary metric before the experiment starts.
The test is fixed-horizon: decide how long the experiment runs before you start, and read the verdict at the end. Checking repeatedly and stopping as soon as a result turns significant inflates the false-positive rate.
Sample ratio mismatch
Gamebeast compares the number of units enrolled in each group with the weights you configured, using a chi-square goodness-of-fit test. A p-value below 0.001 raises a warning that the groups are unevenly sized. That usually means something went wrong with assignment or with how markers are sent from one group, and the lift shouldn't be trusted until it's explained. The strict threshold is the conventional one, because this is a data-quality alarm rather than a result.
See also
- Experiments: create, run, and end an experiment.
- Offline and delivery reliability: how markers sent offline are counted.