Build a credible offer, verify the full conversion path and plan a defined binary experiment using explicit denominators, sample assumptions and stopping rules.
State the traffic source, audience intent, offer and completed action that matters. Define the eligible unit and observation window: for example, one assigned visitor with a confirmed form submission within the chosen period. Record exclusions and distinguish a button click from a completed lead or purchase. Establish a baseline using those same definitions.
Write a clear promise supported by the actual product, price and evidence. Address the questions needed to make the decision and use an action label that describes the next step. Build readable text, accessible controls and a form that asks for necessary information. Check the chosen text/background colors and mobile layout.
Complete the journey using test submissions: successful delivery, invalid input, duplicate click, network failure and the final confirmation. Confirm that analytics count the completed outcome once under the defined unit, with necessary consent and data handling. Reconcile a small set of test records between the page, backend and reporting system before interpreting rates.
Choose one clear hypothesis and a primary binary outcome. For an independent, equal-allocation, fixed-horizon A/B design, enter the baseline, meaningful absolute difference, significance level and power in the sample planner. Review feasibility and approximation limits. Predeclare allocation, exclusions, observation window, stopping rule and guardrails; repeated users, sequential monitoring or multiple tests need a matching design.
Randomize eligible units according to the design and keep assignment stable where required. Check instrumentation, allocation balance and operational guardrails without repeatedly stopping at a favorable significance result. Report each variant’s eligible denominator and completed-outcome count. Keep traffic-source or campaign changes visible because they can affect interpretation.
At the planned analysis point, use the statistical method specified in the design and report effect size, uncertainty and guardrails. The sample planner does not analyze observed results. Compare any gain with costs, lead quality or refunds before deciding to ship, revise or stop. Download the hypothesis, results and decision so the next test builds on evidence.
Write and download the measurement specification, experiment and decision.
Express a compatible completed-outcome count as a percentage of its eligible denominator.
Check selected landing-page text and backing colors.
Plan approximate per-variant samples for one equal-allocation independent binary comparison.
Loading your checklist…
statsmodels documents a two-proportion normal planning approximation with a pooled null variance and separate alternative variance. The two-sided approximation ignores the far rejection tail.
Check actual foreground/background combinations and rendered text sizes alongside the functional form journey.
Keep these task-specific records with the tested version and review date.
| Record | Include | Verify |
|---|---|---|
| Outcome | Independent unit, denominator and observation window | One completed outcome counted under the same rule |
| Design | Baseline, absolute difference, allocation, alpha and power | Sample and stopping method match the design |
| Decision | Effect, uncertainty, guardrails and costs | No winner claim from the planner or raw rate alone |
A click can help diagnose a funnel, but it is not the same outcome as an accepted lead or paid order.
Choose a change that would matter to the decision before looking for a convenient sample size.