Plan an independent binary-outcome experiment, check assignment counts and analyze the actual 2×2 table with effect intervals and explicit decision criteria.
Define the user or other independent randomization unit, eligibility, binary outcome, observation window, allocation and practical improvement worth detecting. Set primary and guardrail metrics, alpha, the stopping rule and any multiplicity or sequential method before seeing outcomes. Repeated events from one assigned user are not automatically independent observations.
For a fixed-design equal-allocation binary test, use the baseline rate and an absolute percentage-point alternative in the sample planner. Check the resulting plan in the independent-binary-rate power mode using both sample counts and assumed rates. Plan exposure duration from eligible traffic and outcome maturity; continuous or dependent outcomes need a different design.
Compare actual assigned counts with the intended allocation using goodness of fit when its assumptions apply. Investigate sample-ratio mismatch, duplicate units, missing outcomes and logging changes before interpreting treatment effects. Build outcome denominators from the prespecified observation window and record every exclusion and unresolved data-quality issue.
Enter successes and failures for groups A and B in the separate 2×2 mode. Use the prespecified Pearson test without continuity correction when its approximation is suitable, or the specified Fisher conditional test for an appropriate sparse-table analysis. Report both rates, B-minus-A percentage-point difference and the explicitly labeled Newcombe interval.
Compare the test p-value with prespecified alpha while retaining its actual precision. Evaluate effect size, interval, guardrails, implementation cost and data limitations together. The displayed Newcombe interval is not the inversion of the Fisher test, and threshold crossing alone is not evidence that an experiment should ship.
Choose release, further study or no change using the original decision criteria and observed evidence. Save allocation counts, outcome table, test/interval methods, exclusions, stopping rule and any deviations. If the study used repeated peeking, clustering or multiple tests, use the corresponding analysis rather than relabeling this fixed-design calculator result.
Write and download the task brief, measured results and handoff decisions as Markdown.
Plan a fixed-design, equal-allocation binary-rate comparison from a baseline and absolute percentage-point alternative.
Use the power mode corresponding to the prespecified independent binary-rate or mean-comparison design.
Use goodness of fit for category allocation or the separate 2×2 mode for independent binary outcomes.
Compare a reported p-value with the prespecified threshold without inferring effect size or hypothesis probability.
Loading your checklist…
SciPy documents goodness of fit for category counts and the need for compatible totals and justified degrees of freedom. An allocation check is a different question from the treatment-outcome table.
SciPy documents the two-sided conditional table-probability definition used by the Fisher mode. The difference interval is calculated separately.
The ASA statement emphasizes transparent reporting and distinguishes statistical thresholds from effect size and the importance of a result.
Keep the following evidence with the actual version used for this task.
| Record | Include | Check |
|---|---|---|
| Design | Unit, outcome, alternative, allocation and stop rule | Prespecified before outcomes |
| Evidence | Assignment counts, 2×2 outcomes and exclusions | Investigate data-quality problems |
| Decision | Rates, difference, interval, test and guardrails | Keep practical costs and limitations visible |
A sample-ratio check does not test whether treatment changed conversion.
A nonsignificant test is not proof that effects are absent or variants are equivalent.