Q: What are post-hoc analyses?

Answer: Post-hoc analyses are statistical tests conducted after an experiment to explore findings not specified in the original hypothesis. They are typically used to identify which specific group differences drove a significant result.

What Makes Post-Hoc Analyses Different

When researchers run an experiment, they usually begin with a primary hypothesis: a specific question they intend to answer. But data often reveals unexpected patterns. Post-hoc analyses (from the Latin after this) are follow-up investigations performed after looking at the data, rather than before. This distinguishes them sharply from a priori (pre-planned) analyses, where hypotheses are locked in before data collection begins. The key tension is this: the more comparisons you test in a dataset, the higher your chance of finding something that looks significant purely by accident. This is known as the multiple comparisons problem, and it is the central reason post-hoc analyses require careful handling.  

Why Researchers Use Post-Hoc Analyses

Post-hoc analyses serve several legitimate purposes in research:

1. Decomposing omnibus test results: When an ANOVA (Analysis of Variance) reveals that some group differs from others, post-hoc tests pinpoint which groups are different.
2. Exploring unexpected patterns: Data sometimes tells a more interesting story than the original hypothesis anticipated. Post-hoc exploration allows researchers to follow those threads.
3. Generating new hypotheses: Discoveries made post-hoc are not conclusions; they are starting points for future confirmatory studies.
4. Understanding subgroup effects: In clinical trials, a treatment may perform differently across age groups or sexes, and post-hoc analyses can surface these variations.
5. Correcting for multiple testing: Specialised post-hoc correction methods help control the false discovery rate when many comparisons are made simultaneously.

Common Post-Hoc Tests and When to Use Them

Different post-hoc tests are designed for different scenarios. Choosing the wrong one can inflate or deflate your findings.

Test	Best Used When	Correction Approach
Tukey's HSD	Comparing all pairs of group means	Controls family-wise error rate
Bonferroni Correction	Small number of planned comparisons	Divides alpha by number of tests
Scheffé's Test	Complex contrasts beyond pairwise	Very conservative; suits exploratory work
Dunnett's Test	Comparing multiple groups to one control	Protects only the control comparison
Benjamini-Hochberg	Large number of comparisons (e.g., genomics)	Controls false discovery rate (FDR)
Games-Howell	Unequal group sizes or variances	Does not assume equal variances

 

The Risk of p-Hacking

The most serious misuse of post-hoc analysis is p-hacking: running many tests until a statistically significant result appears and then presenting that result as if it were always the intended finding. This practice inflates false-positive rates and has contributed significantly to the replication crisis in social and medical sciences. Common p-hacking patterns include:

1. Collecting more data after peeking at results until p < 0.05 is reached.
2. Testing multiple outcome variables and reporting only the significant one.
3. Removing "outliers" selectively until significance is achieved.
4. Switching between statistical tests until one produces a desired result.
5. Splitting data into subgroups and searching for segments that show an effect.

The solution is transparency: researchers must clearly label any analysis as post-hoc and apply appropriate statistical corrections.

Post-Hoc vs. Exploratory vs. Confirmatory Analysis

These three terms are frequently confused. The table below clarifies the distinctions:

Type	Timing	Hypothesis	Purpose	Requires Replication?
Confirmatory	Pre-registered before data collection	Fixed, specific	Test a defined prediction	No (if pre-registered)
Exploratory	Before or during analysis, no rigid hypothesis	Open-ended	Find patterns and generate ideas	Yes
Post-Hoc	After observing results	Suggested by the data	Explain unexpected findings	Yes

  The core principle: confirmatory analysis proves; exploratory and post-hoc analysis discover. Discoveries must be confirmed in independent datasets before being treated as established findings.  

How to Report Post-Hoc Analyses Responsibly

Transparent reporting is what separates legitimate post-hoc analysis from data manipulation. Responsible practice includes:

1. Label analyses explicitly: State clearly that a finding was unplanned and arose after examining the data.
2. Report all tests conducted: Not just the significant ones. Selective reporting is a form of bias.
3. Apply appropriate corrections: Use Bonferroni, FDR control, or another suitable method when multiple comparisons are made.
4. Distinguish effect size from statistical significance: A p-value below 0.05 in a post-hoc analysis is a weak signal without a meaningful effect size to support it.
5. Frame findings as hypothesis-generating: Use language like "this warrants further investigation" rather than "this proves."
6. Pre-register future studies: Turn the post-hoc finding into the primary hypothesis of a new, pre-registered study.

 

Real-World Applications

Post-hoc analyses appear across nearly every research-intensive field:

Field	Example Use Case
Medicine	Identifying which patient subgroup benefited most from a drug trial
Psychology	Pinpointing which experimental condition caused a behaviour change
Marketing	Determining which customer segment responded to a campaign
Genomics	Screening thousands of gene variants after an initial association study
Education	Exploring which school demographic showed unexpected learning gains
Economics	Investigating which time period drove an anomalous trend

 

The Bottom Line

Post-hoc analyses are powerful, valid, and necessary, but only when conducted and communicated with rigour. They are tools for discovery, not confirmation. When a post-hoc finding is presented as a primary result without correction, disclosure, or replication, it becomes misleading. When used transparently, with appropriate statistical safeguards and honest framing, post-hoc analysis is one of the most valuable instruments a researcher has for understanding complex data.