Plan a statistical comparison, match methods to the sampling design, report effects and uncertainty, and preserve assumptions and analysis decisions.
State the population, outcome and quantity to estimate, such as a mean difference or difference in binary rates. Identify the independent unit and whether observations are paired, clustered, weighted or ordered in time. Record the comparison direction, practically meaningful effect and sampling or assignment process before choosing a test.
For new independent binary groups, the Binary A/B Sample Size Calculator can estimate an approximate per-group count. Statistical Power provides the explicitly labeled known-variance, independent-means or binary normal approximations. Enter a justified alternative and design assumptions; use a method matched to other designs elsewhere. Prespecify the analysis family, multiplicity approach, missing-data policy and stopping rule.
Reconcile included and excluded counts with the source, inspect outcome distributions and check pairing or clustering before analysis. Document missingness and influential observations. Choose a method appropriate to the design and its assumptions; a normality test alone is not a universal test-selection rule. Record software versions, options and any changes from the original plan.
Use the analysis environment that implements your selected method. For two independent binary groups, this site’s 2×2 mode offers Pearson chi-square without continuity correction or a conditional two-sided Fisher test, plus a separately calculated Newcombe rate-difference interval. Its goodness-of-fit mode instead compares category counts with expected counts. Neither mode handles paired outcomes, general multiway tables or continuous-outcome t-tests.
Report the effect in its original units, the interval method and confidence level, the test and the unrounded p-value. P-value Interpreter compares a supplied p-value with your specified alpha; it does not compute the test or correct multiple testing. Explain practical importance and model limitations. A non-rejection does not establish equivalence or prove no effect.
Keep the planned and exploratory analyses, effect estimates, intervals, raw and adjusted p-values where applicable, denominators, exclusions and sensitivity results. Explain the decision and what evidence could change it. Link approved data snapshots, code, software versions and the analysis plan. Review and download the report in Markdown Preview, checking that no result or limitation was lost.
Plan approximate per-group sample counts only for the supported independent binary comparison.
Explore prospective power using the named approximation and a justified alternative effect.
Use the distinct goodness-of-fit or independent 2×2 mode only when it matches the analysis design.
Compare a supplied unrounded p-value with the prespecified threshold while retaining context.
Review and download the complete authored analysis record, including effects and limitations.
Loading your checklist…
The ASA emphasizes context, transparent reporting and effect magnitude. Crossing an alpha threshold alone is insufficient for a scientific or practical conclusion; a p-value is not the probability that a hypothesis is true.
NIST explains confidence as a property of a repeated interval-construction procedure under its assumptions. It is not the percentage of raw observations lying between the bounds.
SciPy documents that ttest_ind assumes equal variances by default and uses Welch’s method when equal_var=False. This independent-sample procedure does not model paired or clustered observations.
statsmodels documents Bonferroni, Holm and false-discovery-rate procedures. Select the error criterion and a method whose dependence assumptions suit the design, then retain both original and adjusted results.
SciPy describes the fixed-margin null distribution for a 2×2 Fisher test. Report the table orientation and two-sided definition; do not substitute an unrelated goodness-of-fit calculation.
The site’s binary table interval and test are separate calculations; the Newcombe interval is not obtained by inverting Fisher’s test.
| Question | Report |
|---|---|
| Independent binary groups | All four cells, test choice, B−A rates and interval method |
| Category goodness of fit | Observed/expected counts, fitted-parameter adjustment and degrees of freedom |
| Other designs | Appropriate external model, assumptions, estimate and uncertainty |
| Multiple analyses | Prespecified family, error criterion, raw and adjusted results |
Justify the planning alternative using the decision and prior evidence; do not label 80% power or p<0.05 a universal quality grade.
Precise software output does not repair selection bias, dependence, measurement error or an incompatible model.
Save the source version and code with explicit options so another authorized analyst can reproduce the comparison.