Within Ignorance
When No Effect Has Not Been Proved
A study can fail to find an effect because the effect is absent, or because the test was too small or too noisy to detect it.
On this page
- Why non significant results can mislead
- Power, sample size, and noisy measurements
- Better ways to argue for genuine absence
Page outline Jump by section
Introduction
A common form of appeal to ignorance appears when a study fails to find a statistically significant effect and people treat that outcome as proof that no effect exists. In reality, a null result often means only that the study did not detect a difference under the conditions tested. The effect may truly be absent, but it may also be too small, too variable, or too difficult for the study design to detect. Statistical researchers have warned for decades that “non-significant” should not be read as “no effect,” because a failed detection and a demonstrated absence are different conclusions. [PubMed]pubmed.ncbi.nlm.nih.govPubMedAbsence of evidence is not evidence of absenceby DG Altman · 1995 · Cited by 2470 — When statistical analysis of the study data fin… [PMC]pmc.ncbi.nlm.nih.govPMCAbsence of evidence is not evidence of absencePMC - NIHby P Alderson · 2004 · Cited by 329 — Altman and Bland considered the dangers of misinterpreting differences that do not reach s…
Within the broader family of appeal-to-ignorance errors, this mistake occurs when missing evidence is treated as decisive evidence. The reasoning jumps from “the study did not show an effect” to “the effect does not exist,” even though the study may not have been capable of ruling the effect out. [PubMed]pubmed.ncbi.nlm.nih.govPubMedAbsence of evidence is not evidence of absenceby DG Altman · 1995 · Cited by 2470 — When statistical analysis of the study data fin…
Why Non-Significant Results Can Mislead
Scientific papers often report whether an observed difference reached a threshold for statistical significance, commonly expressed through a p-value. When a result is not statistically significant, the formal conclusion is usually that the study failed to reject the null hypothesis. That is not the same thing as proving the null hypothesis correct. [Taylor & Francis Online]tandfonline.comTaylor & Francis OnlineThe ASA Statement on p-Values: Context, Process, and…by RL Wasserstein · 2016 · Cited by 8584 — P-values do not…
The distinction matters because many readers unconsciously convert a cautious statement into a stronger one:
- What the study showed: the evidence was insufficient to establish a detectable effect.
- What the mistaken interpretation claims: there is no effect.
Altman and Bland’s influential discussion of “absence of evidence” pointed out that many supposedly “negative” studies may still be compatible with clinically or practically important effects. A non-significant result merely indicates that the data did not provide strong enough evidence under the chosen test. [PubMed]pubmed.ncbi.nlm.nih.govPubMedAbsence of evidence is not evidence of absenceby DG Altman · 1995 · Cited by 2470 — When statistical analysis of the study data fin…
This confusion mirrors the structure of an appeal to ignorance. The lack of proof becomes treated as proof of the opposite position. In scientific debates, that can prematurely close inquiry, discourage replication, or create unwarranted confidence in a claim of no difference. [PMC]pmc.ncbi.nlm.nih.govPMCStatistically significant results from low-power analysessignificant results from low-power analyses - PMCby C Jaksic · 2026 — A non-statistically significant result means that the test failed t…
Power, Sample Size, and Noisy Measurements
Why Small Studies Miss Real Effects
The most common reason a null result fails to settle a question is insufficient statistical power. Power refers to the probability that a study will detect an effect if the effect genuinely exists. Low-powered studies have a substantial risk of false negatives: real effects remain hidden because the experiment lacks enough information to reveal them. PMC [PubMed]pubmed.ncbi.nlm.nih.govPubMedAbsence of evidence is not evidence of absenceby DG Altman · 1995 · Cited by 2470 — When statistical analysis of the study data fin…
A study with only a small number of participants may observe a genuine difference, but random variation can easily obscure it. Researchers may therefore report a non-significant result even when an effect is present. Sample-size planning exists largely to reduce this problem. [PMC]pmc.ncbi.nlm.nih.govPMCStatistics in Brief: The Importance of Sample SizePMC - NIHby DJ Biau · 2008 · Cited by 732 — The size of the sample studied is a major determinant of the risk of reporting false-negative… [ScienceDirect]sciencedirect.comScienceDirectBest (but oft forgotten) practices: sample size planning for…by SF Anderson · 2019 · Cited by 52 — This article aims to p…
Noise Can Hide Signals
Sample size is only part of the story. Measurements can be noisy because of inconsistent instruments, biological variation, unreliable surveys, or poorly controlled conditions. Statistical power depends on the balance between signal and noise. Even relatively large studies can struggle if measurements are imprecise. [PMC]pmc.ncbi.nlm.nih.govPMCSample size, power and effect size revisitedPMC - NIHby CC Serdar · 2020 · Cited by 2154 — Use of a statistically incorrect sample size may lead to inadequate results in both clinic…
Consider two medical treatments that differ only slightly in effectiveness. If patient outcomes vary widely for unrelated reasons, the treatment difference may disappear within the background noise. A non-significant result in that situation does not establish equality between treatments; it may simply reveal the limits of the measurement process. [PMC]pmc.ncbi.nlm.nih.govPMCHow to Calculate Sample Size and WhyPMCby J Kim · 2013 · Cited by 143 — Many researchers want to show that the two groups are truly distinct, but they will fail to find sign…
Historical Lessons from Underpowered Research
Concerns about underpowered studies have been particularly prominent in fields such as neuroscience and psychology. Reviews of research practices have repeatedly found that many studies were too small to reliably detect the effects they sought to measure. The consequence is not only missed effects but also confusion about which findings should be trusted. [PubMed]pubmed.ncbi.nlm.nih.govPubMedAbsence of evidence is not evidence of absenceby DG Altman · 1995 · Cited by 2470 — When statistical analysis of the study data fin…
This history provides a cautionary lesson: when a field routinely relies on small studies, a collection of null results may tell us more about methodological limitations than about the true absence of an effect. [PubMed]pubmed.ncbi.nlm.nih.govPubMedAbsence of evidence is not evidence of absenceby DG Altman · 1995 · Cited by 2470 — When statistical analysis of the study data fin…
A Concrete Example of the Reasoning Error
Imagine a study investigating whether a new educational programme improves examination scores. Researchers compare two groups of students and find that the programme group scores slightly higher, but the difference is not statistically significant.
A careful interpretation would be: [pmc.ncbi.nlm.nih.gov]pmc.ncbi.nlm.nih.govPMCUse of Confidence Intervals in Interpreting Nonstatistically…by AT Hawkins · 2021 · Cited by 32 — Careful use of confidence interva…
The study did not provide strong evidence that the programme improved scores.
An appeal-to-ignorance interpretation would be:
The programme has no effect on scores.
The second statement assumes more than the data justify. The study may have involved too few students, too much variation in teaching quality, or an effect too modest for the chosen design to detect. Until those possibilities are addressed, the claim of “no effect” remains unproven. [PubMed]pubmed.ncbi.nlm.nih.govPubMedAbsence of evidence is not evidence of absenceby DG Altman · 1995 · Cited by 2470 — When statistical analysis of the study data fin… [PMC]pmc.ncbi.nlm.nih.govPMCSample Size and its Importance in ResearchSize and its Importance in Research - PMCby C Andrade · 2020 · Cited by 1101 — This article discusses sample size and how it relates to m…
Better Ways to Argue for Genuine Absence
The fact that null results do not automatically prove no effect does not mean evidence for absence is impossible. Researchers have developed stronger approaches for situations where the goal is to show that an effect is genuinely absent or too small to matter.
Use Confidence Intervals
Confidence intervals show the range of effect sizes compatible with the data. A non-significant result accompanied by a very wide interval remains inconclusive because large positive or negative effects may still fit the evidence. A narrow interval centred near zero provides a much stronger basis for claiming that any effect is likely to be trivial. [PMC]pmc.ncbi.nlm.nih.govPMCPower to the People: Power, Negative Results and Sample Sizeby BN Gaskill · 2020 · Cited by 64 — The conventional wisdom that statisti… [BMJ]bmj.comabsence evidence and importance confidence intervalsBMJAbsence of evidence and the importance of confidence…26 Feb 2004 — Since our 1995 BMJ note[1] the title “Absence of evidence is not…
For example:
- A confidence interval ranging from a large benefit to a large harm suggests uncertainty.
- A confidence interval tightly clustered around no difference suggests practical absence.
The distinction is often more informative than the significance label alone. [PMC]pmc.ncbi.nlm.nih.govPMCListen to the data when results are not significantPMCby CE Hewitt · 2008 · Cited by 80 — When randomised controlled trials show a difference that is not statistically significant there is…
Design Studies to Rule Out Meaningful Effects
If researchers want to argue that an effect is absent, they should first define what size of effect would matter. Studies can then be designed with enough precision to detect effects of that magnitude. If no such effect appears, the resulting conclusion is far stronger. [JAMA Network]jamanetwork.comJAMA NetworkLack of Treatment Efficacy From Statistically Nonsignificant…by T Perneger · 2023 · Cited by 19 — Many statistically nonsi…
This shifts the question from “Did we find significance?” to “Have we gathered enough evidence to exclude effects that would be important?” [JAMA Network]jamanetwork.comJAMA NetworkLack of Treatment Efficacy From Statistically Nonsignificant…by T Perneger · 2023 · Cited by 19 — Many statistically nonsi…
Use Equivalence Testing and Related Methods
Modern statistical approaches such as equivalence testing and Bayes factors are specifically designed to evaluate evidence for absence rather than merely evidence against a null hypothesis. Recent analyses of replication projects have shown that many studies labelled as having “null results” are actually inconclusive when examined with these methods. [eLife]elifesciences.orgreviewed preprintseLifeReplication of “null results” – Absence of evidence or…by S Pawel · 2024 · Cited by 12 — We show how methods, such as equivalence…
These approaches recognise a crucial logical point: failing to find evidence for an effect is different from obtaining evidence that the effect is absent.
What This Means for Logical Reasoning
Null results occupy an awkward position in argumentation. They are often informative, but only when interpreted in light of study quality, sample size, measurement precision, and the expected detectability of the effect. Treating every non-significant result as proof of no effect turns uncertainty into certainty without justification.
The appeal-to-ignorance mistake occurs when the absence of detected evidence is allowed to carry more weight than the evidence itself supports. Sound reasoning asks an additional question: if the effect were real, should this study have been able to find it? Only when the answer is clearly yes does a null result begin to function as meaningful evidence of absence rather than merely an absence of evidence. PMC [PubMed]pubmed.ncbi.nlm.nih.govPubMedAbsence of evidence is not evidence of absenceby DG Altman · 1995 · Cited by 2470 — When statistical analysis of the study data fin… [PMC]pmc.ncbi.nlm.nih.govPMCHow sample size influences research outcomesVery small samples undermine the internal and external validity of a study.Read more…
Amazon book picks
Further Reading
Books and field guides related to When No Effect Has Not Been Proved. Use these as the next step if you want deeper reading beyond the article.
The Art of Statistics
Explains uncertainty, significance, and interpretation of evidence.
The Skeptics' Guide to the Universe
Explains why lack of evidence is not always evidence of absence.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCAbsence of evidence is not evidence of absence
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC351831/Source snippet
PMC - NIHby P Alderson · 2004 · Cited by 329 — Altman and Bland considered the dangers of misinterpreting differences that do not reach s...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCStatistically significant results from low-power analyses
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC12856849/Source snippet
significant results from low-power analyses - PMCby C Jaksic · 2026 — A non-statistically significant result means that the test failed t...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCStatistics in Brief: The Importance of Sample Size
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC2493004/Source snippet
PMC - NIHby DJ Biau · 2008 · Cited by 732 — The size of the sample studied is a major determinant of the risk of reporting false-negative...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCSample size, power and effect size revisited
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC7745163/Source snippet
PMC - NIHby CC Serdar · 2020 · Cited by 2154 — Use of a statistically incorrect sample size may lead to inadequate results in both clinic...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCHow to Calculate Sample Size and Why
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC3758995/Source snippet
PMCby J Kim · 2013 · Cited by 143 — Many researchers want to show that the two groups are truly distinct, but they will fail to find sign...
-
Source: sciencedirect.com
Link: https://www.sciencedirect.com/science/article/pii/S0002916522011777Source snippet
ScienceDirectBest (but oft forgotten) practices: sample size planning for...by SF Anderson · 2019 · Cited by 52 — This article aims to p...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCSample Size and its Importance in Research
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC6970301/Source snippet
Size and its Importance in Research - PMCby C Andrade · 2020 · Cited by 1101 — This article discusses sample size and how it relates to m...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC6978577/Source snippet
PMCPower to the People: Power, Negative Results and Sample Sizeby BN Gaskill · 2020 · Cited by 64 — The conventional wisdom that statisti...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCListen to the data when results are not significant
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC2174757/Source snippet
PMCby CE Hewitt · 2008 · Cited by 80 — When randomised controlled trials show a difference that is not statistically significant there is...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCHow sample size influences research outcomes
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC4296634/Source snippet
Very [small samples]({{ 'small-samples/' | relative_url }}) undermine the internal and external validity of a study.Read more...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC9365504/Source snippet
PMCUse of Confidence Intervals in Interpreting Nonstatistically...by AT Hawkins · 2021 · Cited by 32 — Careful use of confidence interva...
-
Source: bmj.com
Title: absence evidence and importance confidence intervals
Link: https://www.bmj.com/rapid-response/2011/10/30/absence-evidence-and-importance-confidence-intervalsSource snippet
BMJAbsence of evidence and the importance of confidence...26 Feb 2004 — Since our 1995 BMJ note[1] the title “Absence of evidence is not...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC12567074/Source snippet
'no' with confidence: statistical approaches to test for...by LG Halsey · 2025 — Here, I provide a quick-and-easy guide to simple yet po...
-
Source: bmj.com
Link: https://www.bmj.com/ -
Source: bmj.com
Link: https://www.bmj.com/content/311/7003/485/relatedSource snippet
Statistics notes: Absence of evidence is not...Interpretation of CIs in clinical trials with non-significant results: systematic review...
-
Source: bmj.com
Link: https://www.bmj.com/content/342/bmj.d3126Source snippet
Absence of evidence is not evidence of absence. BMJ 1995;311:485. OpenUrlFREE Full TextGoogle Scholar · View Abstract.Read more...
-
Source: bmj.com
Link: https://www.bmj.com/content/328/7438/476?page=1&panels_ajax_tab_tab=bmj_related_rapid_responses&panels_ajax_tab_trigger=rapid-responsesSource snippet
He suggests that one should never claim that there is “no effect” but rather that authors shouldRead more...
-
Source: bmj.com
Link: https://www.bmj.com/content/326/7401/1267.1Source snippet
“Evidence of absence” can be importantby M Joffe · 2003 · Cited by 4 —. Absence of evidence is not evidence of absence. BMJ 1995; 311: 4...
-
Source: bmjopen.bmj.com
Link: https://bmjopen.bmj.com/content/7/7/e017288Source snippet
not statistically significant is 'negative' or 'inconclusive'. In conclusion... Absence of evidence is not evidence of absence. BMJ 2004...
-
Source: sciencedirect.com
Link: https://www.sciencedirect.com/science/article/pii/S2590113326000088Source snippet
Statistically significant results from low-power analysesby C Jaksic · 2026 — At low power, results are either accurate but statistically...
-
Source: pubmed.ncbi.nlm.nih.gov
Link: https://pubmed.ncbi.nlm.nih.gov/7647644/Source snippet
PubMedAbsence of evidence is not evidence of absenceby DG Altman · 1995 · Cited by 2470 — When statistical analysis of the study data fin...
-
Source: tandfonline.com
Link: https://www.tandfonline.com/doi/full/10.1080/00031305.2016.1154108Source snippet
Taylor & Francis OnlineThe ASA Statement on p-Values: [Context]({{ 'context/' | relative_url }}), Process, and...by RL Wasserstein · 2016 · Cited by 8584 — P-values do not...
-
Source: elifesciences.org
Title: reviewed preprints
Link: https://elifesciences.org/reviewed-preprints/92311Source snippet
eLifeReplication of “null results” – Absence of evidence or...by S Pawel · 2024 · Cited by 12 — We show how methods, such as equivalence...
-
Source: pubmed.ncbi.nlm.nih.gov
Link: https://pubmed.ncbi.nlm.nih.gov/23571845/Source snippet
PubMedPower failure: why small sample size undermines the...by KS Button · 2013 · Cited by 9954 — A study with low statistical power has...
-
Source: jamanetwork.com
Link: https://jamanetwork.com/journals/jama/fullarticle/2806151Source snippet
JAMA NetworkLack of Treatment Efficacy From Statistically Nonsignificant...by T Perneger · 2023 · Cited by 19 — Many statistically nonsi...
-
Source: elifesciences.org
Link: https://elifesciences.org/articles/92311Source snippet
Replication of null results: Absence of evidence or...by S Pawel · 2024 · Cited by 12 — This work provides a valuable contribution and a...
-
Source: pubmed.ncbi.nlm.nih.gov
Link: https://pubmed.ncbi.nlm.nih.gov/41002082/Source snippet
Would Be the Effect of Lowering the Threshold...by Y Shimozono · 2026 · Cited by 5 — Lowering the p value threshold to 0.005 would requi...
Additional References
-
Source: oamonitor.ireland.openaire.eu
Link: https://oamonitor.ireland.openaire.eu/national/search/publication?pid=10.1136%2Fbmj.311.7003.485Source snippet
notes: Absence of evidence is not evidence of absenceStatistics notes: Absence of evidence is not evidence of absence · Octreotide infusi...
-
Source: acsu.buffalo.edu
Link: https://www.acsu.buffalo.edu/~wdmccall/os512d/EvidAbs.htmlSource snippet
Douglas G Altman, head,a... Similar evidence of the dangers of misinterpretation of non-significant results...Read more...
-
Source: ncbi.nlm.nih.gov
Title: NCBIHypothesis Testing, P Values, Confidence Intervals
Link: https://www.ncbi.nlm.nih.gov/books/NBK557421/Source snippet
NCBIby J Shreffler · 2023 · Cited by 105 — Thus, while the p-value used to detect statistical significance may result in "not significant...
-
Source: anesthesia.healthsci.mcmaster.ca
Link: https://anesthesia.healthsci.mcmaster.ca/wp-content/uploads/2022/08/absence-of-evidence-is-not-evidence-of-absence.pdfSource snippet
"not significant." Randomised controlled clinical trials that do not show a significant difference...Read more...
-
Source: blog.minitab.com
Title: the american statistical associations statement on the use of p values
Link: https://blog.minitab.com/en/blog/adventures-in-statistics-2/the-american-statistical-associations-statement-on-the-use-of-p-valuesSource snippet
American Statistical [Association]({{ 'association/' | relative_url }})'s Statement on...23 Mar 2016 — Using P values in conjunction with a significance level to decide when t...
-
Source: researchgate.net
Title: (PDF) Absence of Evidence Is Not Evidence of Absence
Link: https://www.researchgate.net/publication/232268309_Absence_of_Evidence_Is_Not_Evidence_of_AbsenceSource snippet
The non-equivalence of statistical significance and clinical importance has long been recognised, but this error of interpretation remain...
-
Source: journals.sagepub.com
Title: Sage Journals With Low Power Comes Low Credibility?
Link: https://journals.sagepub.com/doi/10.1177/25152459241296397Source snippet
Toward a...Jan 28, 2025 — Researchers should be motivated to adequately power statistical tests because tests with low power have a low...
-
Source: youtube.com
Title: Reject or Fail to Reject Null Hypothesis | Decision Rule Explained
Link: https://www.youtube.com/watch?v=nzBrVjvCBOsSource snippet
The video Why do we say Fail to Reject the Null Hypothesis? Why can't I accept the Null Hypothesis? explicitly outlines the fallacy of tr...
-
Source: linkedin.com
Link: https://www.linkedin.com/pulse/p-value-trap-why-significant-does-mean-effect-absence-filip-poscic-2trcfSource snippet
The p-value trap: why “not significant” does not mean “no...A high p-value only indicates your data is compatible with there being no ef...
-
Source: statalist.org
Link: https://www.statalist.org/forums/forum/general-stata-discussion/general/1620255-the-use-of-p-values-has-been-criticized-by-the-american-statistical-associationSource snippet
Geoff Cumming advocates confidence intervals over p-values.Read more...
Topic Tree