Your experiment
INPUTSOne user. One assignment. One outcome.
Sliders move in 0.5 steps. Type a value for finer precision.
Relative changes require a positive baseline. Use absolute units for zero or negative means.
SD is the spread of individual user outcomes, in the same units as the mean — not SE. Historical sample SDs are plug-in assumptions here. Equal defaults can be changed independently.
Revenue, zeros and heavy tails
For revenue per user, include all eligible users, including those with zero revenue. Estimate the mean and SD at that same user level over a comparable outcome window. Revenue per purchaser is a different, conditional metric.
Planning uses a normal approximation to the mean difference and treats the specified SDs as fixed. Small n, skewness, rare outliers or uncertain historical SDs can make power inaccurate. There is no universal sample-size threshold guaranteeing the CLT is adequate. Validate the design against representative data before business use.
Relative change ·
Negative = decrease; positive = increase. MDE is the magnitude of the change; the sign chooses its direction. Lower can be better for undesirable events, such as bad reviews.
An exact complement: p + (1 − p) = 100%.
Traffic & duration
What does MDE mean?
Simple: An effect you want a good chance of detecting.
Technical: The specified alternative effect used to choose n for a target power. It is not a threshold below which significance is impossible.