Your team changes a procedure, collects another month of data, and sees a better average. Which statistical test should you use to check the improvement?
Start with two questions: What did you measure, and are the before-and-after observations genuinely paired?
If you measured the same units twice and want to compare a continuous outcome, a paired t-test may fit. If you measured different units, an independent two-sample test may fit. For pass/fail outcomes, use a method for proportions, with a different approach for paired observations.
The words “before and after” describe timing. They do not, by themselves, identify the correct test.
Start with the outcome you want to improve
Write the operational question before opening your statistical software.
- “Did average setup time decrease?” concerns a continuous measurement.
- “Did the percentage of orders requiring rework decrease?” concerns a binary outcome: rework or no rework.
- “Did defects per 1,000 units decrease?” concerns an event rate. One unit may contain several defects.
These questions need different analyses. A count of defects is not the same as a count of defective units. Minitab distinguishes continuous measurements, binary outcomes, and counts when selecting an analysis. See its guidance for two-sample t-tests and Poisson rate comparisons.
Also define a worthwhile improvement. Saving five seconds and saving five minutes may lead to different business decisions even when both changes are statistically detectable.
Decide whether the observations are really paired
Paired data have a meaningful one-to-one connection. Examples include:
- The same technician completing a standardized task before and after training.
- The same reusable component measured before and after cleaning.
- The same participant receiving a pass/fail assessment at both time points.
Each unit needs an identifier so its before value stays with its own after value. NIST describes paired comparisons as measurements with this natural correspondence, analyzed through their within-pair differences. See NIST's paired observations guidance.
By contrast, 50 orders processed last month and 50 different orders processed this month are not automatically paired. Neither are Monday's first 20 parts and Tuesday's first 20 parts. Equal sample sizes, the same production line, and matching spreadsheet row numbers do not create pairs.
If the same people or units appear in both periods but you cannot identify the matches, do not simply declare the samples independent. Recover the identifiers or get advice on an analysis suited to the available data.
Use this starting guide
These are candidate methods, subject to the assumptions below.
| Outcome and design | Candidate analysis | Example |
|---|---|---|
| Continuous outcome, genuine pairs | Paired t-test for the mean change | Setup time for the same technicians before and after training |
| Continuous outcome, independent samples | Welch two-sample t-test for the difference in means | Processing time for different orders sampled before and after a change |
| Binary outcome, independent samples | Two-proportion test; consider an exact method for sparse counts | Percentage of different shipments delivered late in each period |
| Binary outcome, genuine pairs | McNemar's test | The same participants classified as pass or fail before and after training |
The independent-sample Welch approach does not require equal population variances. NIST explains the unequal-variance two-sample t-test. For categorical outcomes, Minitab documents two independent proportions and Fisher's exact and McNemar's tests.
For paired pass/fail data, keep the four transition counts: pass to pass, pass to fail, fail to pass, and fail to fail. McNemar's test uses the two kinds of change to assess whether the marginal proportions differ. Two overall percentages alone lose the information about who changed.
Counts, exposure-adjusted rates, more than two categories, and more than two time points may require other methods. This guide is not a complete test-selection tree.
Check independence before trusting a result
An independent two-sample test assumes independent observations within and between the samples. A paired test allows dependence within each pair, but the different pairs still need to be independent.
That distinction matters in improvement work:
- Twenty readings from one part are not twenty independent parts.
- Several orders from the same customer may be related.
- Hourly measurements can carry patterns from one hour into the next.
Plot observations in time order and record the batch, machine, customer, or person where relevant. NIST explains how autocorrelation identifies dependence across time.
If observations are clustered or time-dependent, a basic t-test or proportions test may misstate uncertainty. You may need a model that accounts for repeated measurements, groups, or time. Collecting more dependent readings does not fix that problem.
For ongoing processes, a suitable control chart can help examine whether a change persists and whether the process is stable. It does not automatically correct dependence or establish causation. ASQ describes the purpose and use of control charts.
Check the distribution and the question being tested
For a paired t-test, examine the differences within pairs, not just the separate before and after columns. With a small sample, the normality assumption for those differences deserves particular attention. Look for strong skewness and unusual observations. Larger samples can make t procedures more robust to some departures from normality, but there is no sample-size shortcut that fixes every dataset. Minitab explains that the paired t procedure applies a one-sample t-test to the paired differences.
A nonparametric test is not an automatic rescue for questionable data. For example, a Wilcoxon signed-rank approach on paired differences has a symmetry assumption. A sign test does not require symmetry, but it answers a different question from a test of the mean. Choose based on the decision you need to make, not whichever test produces a smaller p-value. See Minitab's guidance on Wilcoxon and sign tests.
Keep measurement definitions, sampling rules, and exclusions consistent across periods. Decide the primary outcome, sample size, significance level, and whether the test is one-sided or two-sided before reviewing the result.
A fictional example of a paired comparison
Suppose 12 technicians each complete the same standardized setup task before and after a procedure change. Technician identifiers preserve the pairs, and the analysis treats the technicians' changes as independent. There are no missing pairs.
The team defines improvement as before time minus after time, so a positive number means a faster setup. The fictional reductions, in minutes, are:
0.8, 1.6, 1.1, 2.0, 0.6, 1.4, 1.8, 0.9, 1.3, 2.2, 0.7, 1.5.
A two-sided paired t-test against a mean change of zero gives:
- Number of pairs: 12.
- Estimated mean reduction: 1.33 minutes.
- 95% confidence interval for the mean reduction: approximately 0.99 to 1.66 minutes.
- P-value: less than 0.001.
These results support a nonzero mean reduction under the test assumptions. They do not mean that every future technician will save 1.33 minutes. The confidence interval concerns the population mean change, not the range of individual changes.
Suppose the team set a minimum worthwhile reduction of 1.25 minutes before collecting data. The estimate exceeds that threshold, but the confidence interval includes smaller reductions. A significant result against zero does not establish that the business target has been exceeded.
There is another limit: everyone performed the task a second time. Practice could explain some of the reduction. The team should describe an observed improvement after the change, rather than claiming this analysis proves the procedure caused it.
Report the size of the change and its limits
A useful conclusion includes the estimated difference, confidence interval, sample sizes, method, and relevant assumptions. A p-value alone leaves too much unanswered.
For a before-and-after study, consider alternative explanations such as a different product mix, seasonal demand, staffing changes, or a changed measurement system. A statistically significant comparison cannot isolate these explanations by itself. This follows the broader distinction between association and causation described by NIST.
A nonsignificant result also does not prove “no change.” The data may be too imprecise to detect an improvement that matters. Review the confidence interval before deciding what the result supports.
What software and preparation do you need
Use statistical software that supports the selected test, its assumptions, and a confidence interval for the effect you want to estimate. In Minitab, the paired t-test is available under Stat > Basic Statistics > Paired t. McNemar's test is available through Cross Tabulation and Chi-Square > Other Stats. See the official paired t example and categorical test options.
Before running the analysis, write down:
- The outcome and its measurement units.
- What counts as one observation.
- Whether observations are paired, independent, clustered, or time-dependent.
- The smallest improvement that would change your decision.
- The sampling plan, candidate test, and assumptions to check.
- How you will report the effect, uncertainty, and remaining limitations.
The RootCause Applied Statistics Planning Worksheets provide a six-page PDF with hypothesis-testing and simple linear regression planning/results worksheets and fictional worked examples. The worksheets cost $7 and are non-fillable. Statistical software is not included.
Use them to organize your analysis plan and results alongside your chosen software. The most useful first step is to decide what evidence would support an improvement before collecting the next set of measurements.