A/B Test Design and Spec
Test spec doc: hypothesis, variants, sample size, duration, metrics, guardrails. Test live in the platform with QA verified. Results document with decision and next steps.
Before you start
- Hypothesis backlog with prioritization
- Testing platform (Optimizely, VWO, GrowthBook, Statsig)
- Sample size calculator for power
- Hypothesis to test (with rationale)
- Baseline metric and conversion rate
- Estimated minimum detectable effect (MDE)
The steps
- Validate the hypothesis is test-worthy — Confirm: clearly stated hypothesis ('changing X will improve Y because Z'), expected impact justifies testing time, and metric is reliable to measure. Hypothesis without a 'because' is a guess, not a hypothesis.
- Calculate required sample size and duration — Use the calculator: baseline conversion, minimum detectable effect, statistical power (typically 80%), significance level (typically 95%). Output: visitors per variant. Estimate duration based on traffic — minimum 1 week, maximum 4 weeks.
- Design control and variant — Control = current state. Variant = the change you're testing. Change ONE thing per test if possible — multivariate is harder to interpret. Specify: visual change, copy change, behavior change. Make sure you can recreate it later if it wins.
- Define success metrics and guardrails — Primary metric: the one you're testing. Secondary metrics: things that should also improve (or at least not regress). Guardrail metrics: things that must not break (e.g., page load time, error rate). Document all three.
- Configure the test in the platform — Set up: variants, traffic allocation (typically 50/50), targeting (which page, which user segment), duration, and metrics. Pre-build the analytics report so results are reviewable in real-time.
- QA before launch — Test the variant on staging or via QA mode. Verify: variant displays correctly, tracking fires correctly for both control and variant, no console errors. Don't launch broken tests — they waste sample.
- Launch and monitor early signals — Launch. Within 24-48h, check guardrails (no regression in error rate, load time). Don't peek at primary metric — early peeking inflates false-positive rate. Wait for the calculated sample size.
- Read results and document outcome — When sample size hits target, read results: did the variant beat control with statistical significance? Did secondary metrics agree? Document outcome (winner, no winner, or inconclusive) and decide: ship variant, ship control, or iterate.
If it goes wrong
Test concludes at 90% significance — should we ship?
Don't ship. Either the test needs more sample or the effect is too small. Pre-define significance threshold and stick to it.
Variant wins but the win doesn't replicate post-launch
Often Simpson's paradox or sampling bias. Investigate by segment. If unable to replicate, the original test was likely flawed.
Stakeholders pressure to call the test early
Educate on statistical significance. Early calls inflate false positives. Hold the line on the calculated sample size.
All OpenLabor playbooks