← Maria TushakovaBack to portfolio

The A/B Testing Lab

See what actually happens inside an experiment.

H₀: no difference · H₁: a difference in either direction. Choose the test before seeing results.

Conversion: one binary outcome per user. Planning uses the two-proportion normal approximation.

01 / DESIGN YOUR EXPERIMENT

How much evidence do you need?

Choose a metric, set its assumptions and see the trade-offs.

Your experiment

INPUTS

One user. One assignment. One outcome.

Sliders move in 0.5 steps. Type a value for finer precision.

%
0.5%90%
% relative

Relative change ·

Negative = decrease; positive = increase. MDE is the magnitude of the change; the sign chooses its direction. Lower can be better for undesirable events, such as bad reviews.

Converted: 20%Did not: 80%

An exact complement: p + (1 − p) = 100%.

CONTROL →TREATMENT TARGET
Traffic & duration
%
What does MDE mean?

Simple: An effect you want a good chance of detecting.

Technical: The specified alternative effect used to choose n for a target power. It is not a threshold below which significance is impossible.

Two complements. Different questions.

ALWAYS 100%

Drag a boundary to explore the cost of being more certain.

NOMINAL CONFIDENCE 1 − α
INPUT · α 0.1–99.9%A false alarm ↔ nominal coverage
DESIRED POWER 1 − β
INPUT · β 0.1–99.9%Miss a real effect ↔ detect it
The definitions, without the ambiguity

α: A false alarm. Technically P(reject H₀ | H₀ true).

Nominal confidence: The intended long-run coverage of an interval procedure under its assumptions. It is not the probability that this observed interval contains a fixed true effect.

β: Missing the specified real effect. Technically P(fail to reject H₀ | specified H₁).

Power: Detecting the specified real effect. Technically P(reject H₀ | specified H₁). There is no single power value for every possible true effect.

Your probabilities, made visible

Move a boundary. Watch the distributions respond.

SELECTED TEST
Same x-axis · each curve has area 100%

Power and β refer to the chosen true effect. SE describes the spread of estimated conversion differences across repeated experiments; pp means percentage points, not relative uplift.

Explain the areas & show calculations

H₀ tails: α/2 + α/2 H₁ inside both thresholds: β Alternative tails: powerDashed lines: both rejection thresholds

Read the graph & understand the approximation

The curves show possible estimates of treatment minus control under each assumed truth, not individual users. The centre of H₁ is an assumption, not an observed result. Areas belong to their own curves, not their overlap.

We use a normal approximation with allocation-weighted pooled variance at the null boundary and separate arm variances under the specified alternative. Real tests estimate the pooled rate from each sample; theoretical power is an approximation. Sparse expected counts make it less reliable. One-sided testing is not equivalence or non-inferiority testing.

See the formulas behind this chart

D = treatment − control. Φ is the standard normal cumulative distribution function; z is its quantile.

p_pool = (n₀p₀ + n₁p₁) / (n₀ + n₁)
SE₀ = √[p_pool(1−p_pool)(1/n₀ + 1/n₁)]
SE₁ = √[p₀(1−p₀)/n₀ + p₁(1−p₁)/n₁]
c = z(1−α/2) × SE₀
Reject H₀ when D̂ < −c or D̂ > c
Power = Φ((−c−D)/SE₁) + 1−Φ((c−D)/SE₁)
β = 1−Power

Planning finds the smallest integer control n meeting desired power, with treatment n rounded up to the chosen allocation. At extreme settings the minimum of two users may already exceed the target. The normal approximation is unreliable at very small counts.

Download statistical source ↙

All browser formulas and interactions are included in this single HTML file. Download the readable statistical source above to inspect it separately.

Same errors. A different story.

Keep the null hypothesis explicit. The error follows from the truth and the decision.

Decision

α and β are conditioned on different possible truths. They do not sum to 100%. Here 1−α is correct non-rejection under H₀; nominal CI coverage is a property of a separate interval procedure.

Follow the connections

α significance→Rejection threshold(s)→Power also depends on n & true effect↔β 1 − power

Exact complements
Power + β = 100%
α + nominal confidence = 100%

Fixed-sample relationship
n ↑ → SE ↓ → power ↑
Hold effect, allocation and α fixed.

Planning relationship
|MDE| ↓ → required n ↑
Hold target power and α fixed.

Assumptions & responsible interpretation

Independent users, fixed horizon, a pre-specified two-sided or directional one-sided test, no repeated-peeking or multiple-testing correction. No clustering, attrition or delayed-outcome model. Traffic duration is a minimum: allow full business cycles and outcome maturation. An insignificant result does not prove equivalence. Two-sided tests can reject in either direction; one-sided tests only reject in the selected direction. Choose before seeing results; significance alone does not imply business value.

Browser calculations are independently checked against SciPy and statsmodels; a separate older Python/Streamlit edition may not include the new directional modes. Normal CDF approximations can differ at the last decimal place. Conversion simulation uses independent binomial draws; sparse conversion tables use Fisher’s exact test. Mean simulation uses exact summaries from explicitly normal populations and Welch analysis; it is not a realistic revenue generator. Mean planning treats supplied SDs as fixed and uses normal power, so achieved Welch rejection rates can differ. Heavy tails, small samples and uncertain historical SDs require additional assessment.

Power method reference ↗ · Confidence interval reference ↗

THE A/B TESTING LAB

Understand the experiment,
not just the answer.

A hands-on introduction to power, error rates, uncertainty and metric distributions.

This browser edition runs entirely on your device. This single-file browser edition includes all application source and the open-source Plotly library loaded from its official CDN. A separate Python/Streamlit implementation and reference tests are available in the full project package.

No accounts, data collection or paid APIs are used by the application itself.