Within Evidence Gaps
When One Study Hides the Rest
A single favorable study can mislead when stronger, larger, or contradictory evidence is left out.
On this page
- How selective evidence changes the picture
- What missing trials and reviews can reveal
- How readers can check whether support is balanced
Page outline Jump by section
Introduction
A common form of cherry-picking in health claims occurs when a speaker, advertiser, influencer, or campaign highlights one favourable study while ignoring larger, better-designed, or contradictory research. The cited study may be genuine and accurately described, yet the overall argument becomes misleading because the audience sees only a selected fragment of the available evidence. In the context of logical fallacies, the error is not necessarily that the chosen study is false, but that the evidence presented is incomplete in a way that creates a distorted impression of scientific support. Systematic reviews were developed partly to reduce exactly this problem by examining the total body of relevant evidence rather than isolated findings. [EN Testing Treatments interactive]en.testingtreatments.orgEN Testing Treatments interactiveReducing biases in systematic reviewsFor example, researchers can simply 'cherry pick' those studies whi…
Health decisions are particularly vulnerable to this fallacy because people may change diets, buy supplements, delay treatment, or adopt medical practices based on claims that appear scientific but rest on a carefully selected subset of research. Regulatory and research organisations have repeatedly warned that health claims should be evaluated against the full evidence base rather than individual studies that happen to support a preferred conclusion. [Federal Trade Commission]ftc.govFederal Trade Commission Health Products Compliance GuidanceFederal Trade CommissionHealth Products Compliance GuidanceDecember 20, 2022 — 20 Dec 2022 — This document provides guidance from FTC sta…
How Selective Evidence Changes the Picture
A single study can appear highly persuasive. It may report a statistically significant result, be published in a respected journal, or involve a striking improvement in patients. Yet one study rarely settles a health question on its own.
Medical research often produces mixed findings. Small trials may show benefits that disappear in larger studies. Early observations may not survive replication. Different populations may respond differently to the same intervention. When communicators present only the favourable studies, they create the impression that the science is settled when it may actually be uncertain or divided. [Cochrane]cochrane.orgChapter 1CochraneChapter 1: IntroductionSystematic reviews aim to minimize bias by using explicit, systematic methods documented in advance with a…
This can happen in several ways:
- Selecting positive studies while ignoring negative ones.
- Highlighting small preliminary trials while omitting larger trials.
- Quoting subgroup results while ignoring the main outcome.
- Presenting an early finding without mentioning later research.
- Citing studies that support a marketing claim while excluding systematic reviews that reach a different conclusion.
The result is a skewed picture that exaggerates confidence and may make a weak claim appear strongly supported.
Dietary supplements provide a frequent example. Marketing materials often emphasise favourable studies or biological theories while downplaying larger reviews that find limited or uncertain benefits. Researchers examining supplement advertising have repeatedly identified misleading health claims that rely on selective presentation of evidence. PMC [Springer]link.springer.comMarketing of Dietary Supplements: A Canadian Perspectiveby NJ Temple · 2013 · Cited by 29 — This paper presents an overview of commonly-u…
Why One Positive Study Feels Convincing
People naturally respond to concrete examples. A claim such as “a clinical study showed significant improvement” sounds more persuasive than a cautious statement that dozens of studies produced mixed results.
This psychological effect is strengthened when the cited study is easy to understand. A dramatic result from one trial is memorable. A systematic review weighing dozens of studies, varying methodologies, and uncertainty estimates is harder to communicate. Selective evidence therefore exploits a mismatch between what is scientifically strongest and what is rhetorically strongest.
Health marketers often rely on this imbalance. Regulatory guidance on health-product advertising stresses that claims should be supported by the overall scientific evidence rather than isolated findings chosen because they support a desired message. [Federal Trade Commission]ftc.govFederal Trade Commission Health Products Compliance GuidanceFederal Trade CommissionHealth Products Compliance GuidanceDecember 20, 2022 — 20 Dec 2022 — This document provides guidance from FTC sta…
What Missing Trials and Reviews Can Reveal
The most important question is often not “What study was shown?” but “What studies were left out?”
Systematic reviews and meta-analyses attempt to answer that question by collecting all relevant studies according to predefined criteria. Their purpose is to reduce the influence of selective evidence selection. Cochrane, one of the leading organisations in evidence synthesis, specifically identifies missing studies and non-reporting biases as major threats to reliable conclusions. [Cochrane]cochrane.orgeview, covering both biases in the results of included studies and biases in the…Read more…
A review can reveal that:
- Several unpublished or overlooked trials found little benefit.
- Positive effects shrink when larger samples are included.
- Apparently strong findings depend heavily on a few small studies.
- Different studies reach conflicting conclusions.
- Evidence quality is too weak to support confident claims.
Sometimes the omitted evidence changes the conclusion entirely. Research on publication bias has shown that studies with favourable or statistically significant results are more likely to be published and cited, creating a risk that visible evidence overstates effectiveness. PMC [Wiley Online Library]onlinelibrary.wiley.comWiley Online LibraryInvestigating and dealing with publication bias and other…by MJ Page · 2021 · Cited by 296 — In this article, we s…
The Problem of Selective Outcome Reporting
Cherry-picking can occur within a single study as well as across multiple studies.
Researchers often measure many outcomes. If only the favourable outcomes are highlighted while disappointing results are omitted or minimised, readers receive a distorted picture of the intervention’s performance. This practice is known as selective outcome reporting or outcome reporting bias. Studies examining clinical research have documented discrepancies between planned outcomes and reported outcomes, leading to concerns that published results may overstate benefits. PMC [BMJ Open]bmjopen.bmj.comBMJ OpenSelective reporting bias in randomised controlled trials…by EKC Wong · 2019 · Cited by 32 — The most common discrepancy was no…
Reporting standards such as CONSORT were developed to improve transparency and make it easier to determine whether researchers are presenting all planned results rather than only the most favourable ones. [BMJ]bmjopen.bmj.comBMJ OpenSelective reporting bias in randomised controlled trials…by EKC Wong · 2019 · Cited by 32 — The most common discrepancy was no…
A Familiar Pattern in Nutrition and Supplement Claims
Nutrition and supplement debates often illustrate the mechanics of cherry-picking because evidence develops gradually and findings can be inconsistent.
For example, health products may continue promoting benefits based on early or limited evidence even after larger trials produce weaker results. Analyses of fish-oil supplement marketing found widespread cardiovascular health claims despite later large randomised trials failing to show broad heart-health benefits for the general population. The claims were often built around selective emphasis on supportive evidence rather than the full research record. [Health]health.comStudy: Majority of Fish Oil Supplements Make Unfounded Health Claimsmake health claims not supported by scientific evidence, particularly regarding heart health. Conducted by researchers from UT Southweste…
The same pattern appears when preliminary findings are presented as established fact. Early studies may justify further investigation, but persuasive claims sometimes treat them as conclusive proof while overlooking the need for replication and larger trials. Recent debates over dietary approaches for mental-health conditions have highlighted concerns that small or preliminary studies can be oversold long before stronger evidence is available. [The Washington Post]washingtonpost.comThe Washington Post RFK Jrclaimed that keto diets cured schizophrenia. Here's what science says.February 17, 2026 — Health Secretary Robert F. Kennedy Jr. has clai…
The fallacy is not that the preliminary studies exist. The fallacy lies in presenting them as though they represent the complete state of knowledge.
How Readers Can Check Whether Support Is Balanced
Readers do not need specialist training to recognise warning signs of selective evidence.
When encountering a health claim, useful questions include:
- Is the claim based on one study or many?
A single study is usually less informative than a body of evidence reviewed systematically.
- Are systematic reviews or meta-analyses mentioned?
These are designed to reduce cherry-picking by examining all relevant research. [Cochrane]cochrane.orgMeta-analyses are at risk of bias due to missing evidence when results of some…Read more…
- Does the source discuss contradictory findings?
Honest summaries acknowledge disagreement and uncertainty.
- How large was the study?
Small studies are often more vulnerable to chance findings and exaggerated effects.
- Were the outcomes pre-specified?
Transparent reporting reduces the risk that only favourable results were selected for publication. [BMJ]bmj.combmj 2024 081123BMJCONSORT 2025 statement: updated guideline for reporting…by S Hopewell · 2025 · Cited by 1511 — The CONSORT 2025 statement consists…
- Who benefits from the claim?
Commercial incentives do not automatically invalidate evidence, but they increase the importance of examining the complete research record.
A balanced evidence presentation rarely claims certainty from a single paper. Instead, it situates individual studies within the wider literature, acknowledges limitations, and explains how the new findings compare with existing research.
Why Cherry-Picked Studies Matter
Cherry-picking is powerful because it can create the appearance of scientific support without providing a fair account of the science itself. In health communication, the difference between one supportive study and the total body of evidence can influence medical choices, purchasing decisions, and public understanding of risk.
The logical fallacy emerges when a conclusion is presented as evidence-based even though key evidence has been excluded. A favourable study may be real, rigorous, and worth discussing. The problem begins when it is treated as the whole story while contradictory trials, negative findings, and broader reviews remain out of view. In evidence-based health assessment, what is missing can be just as important as what is shown.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: cochrane.org
Title: Chapter 1
Link: https://www.cochrane.org/authors/handbooks-and-manuals/handbook/archive/v5.2.0Source snippet
CochraneChapter 1: IntroductionSystematic reviews aim to minimize bias by using explicit, systematic methods documented in advance with a...
-
Source: cochrane.org
Link: https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-07Source snippet
eview, covering both biases in the results of included studies and biases in the...Read more...
-
Source: cochrane.org
Link: https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-13Source snippet
Meta-analyses are at risk of bias due to missing evidence when results of some...Read more...
-
Source: onlinelibrary.wiley.com
Link: https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1468Source snippet
Wiley Online LibraryInvestigating and dealing with publication bias and other...by MJ Page · 2021 · Cited by 296 — In this article, we s...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCFalse and misleading health-related claims in food
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC11082796/Source snippet
PMCby C Muela-Molina · 2021 · Cited by 36 — The aim of this research is to analyse the presence of false and misleading claims in food su...
-
Source: link.springer.com
Link: https://link.springer.com/article/10.1007/s13668-013-0057-zSource snippet
Marketing of Dietary Supplements: A Canadian Perspectiveby NJ Temple · 2013 · Cited by 29 — This paper presents an overview of commonly-u...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCToo Little, Too Late: Ineffective Regulation of Dietary
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC4330859/Source snippet
PMCby RR Starr · 2015 · Cited by 261 — Moreover, they have applied a higher standard for demonstrating that a claim is misleading, arguin...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCAssessment of publication bias and outcome reporting
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC6992172/Source snippet
PMCby AA Ayorinde · 2020 · Cited by 128 — We found that 43% (n = 85) of the reviews mentioned publication bias and 10% (n = 19) formally...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC10683727/Source snippet
CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials.Read more...
-
Source: bmjopen.bmj.com
Link: https://bmjopen.bmj.com/content/9/9/e031138Source snippet
BMJ OpenSelective reporting bias in randomised controlled trials...by EKC Wong · 2019 · Cited by 32 — The most common discrepancy was no...
-
Source: bmj.com
Title: bmj 2024 081123
Link: https://www.bmj.com/content/389/bmj-2024-081123Source snippet
BMJCONSORT 2025 statement: updated guideline for reporting...by S Hopewell · 2025 · Cited by 1511 — The CONSORT 2025 statement consists...
-
Source: health.com
Title: Study: Majority of Fish Oil Supplements Make Unfounded Health Claims
Link: https://www.health.com/fish-oil-supplements-for-heart-health-7852475Source snippet
make health claims not supported by scientific evidence, particularly regarding heart health. Conducted by researchers from UT Southweste...
-
Source: methods.cochrane.org
Title: 2010 SMG training cardiff day2 session3 dwan altman
Link: https://methods.cochrane.org/statistics/sites/methods.cochrane.org.statistics/files/uploads/SMG_training_course_cardiff/2010_SMG_training_cardiff_day2_session3_dwan_altman.pdfSource snippet
Outcome Reporting(iii) Better reporting - CONSORT statement, submission of protocol with manuscript... Outcome reporting bias in randomi...
-
Source: en.testingtreatments.org
Link: https://en.testingtreatments.org/book/8-assessing-relevant-reliable-evidence/reducing-biases-in-systematic-reviews/Source snippet
EN Testing Treatments interactiveReducing biases in systematic reviewsFor example, researchers can simply 'cherry pick' those studies whi...
-
Source: ftc.gov
Title: Federal Trade Commission Health Products Compliance Guidance
Link: https://www.ftc.gov/business-guidance/resources/health-products-compliance-guidanceSource snippet
Federal Trade CommissionHealth Products Compliance GuidanceDecember 20, 2022 — 20 Dec 2022 — This document provides guidance from FTC sta...
Published: December 20, 2022
-
Source: washingtonpost.com
Title: The Washington Post RFK Jr
Link: https://www.washingtonpost.com/health/2026/02/17/keto-diet-schizophrenia-rfk-jr/Source snippet
claimed that keto diets cured schizophrenia. Here's what science says.February 17, 2026 — Health Secretary Robert F. Kennedy Jr. has clai...
Published: February 17, 2026
Additional References
-
Source: eprints.whiterose.ac.uk
Link: https://eprints.whiterose.ac.uk/id/eprint/214582/1/Selective%20outcome%20reporting%20in%20trials%20of%20behavioural%20health%20interventions%20in%20health%20psychology%20and%20behavioural%20medicine%20journals%20%20a%20review.pdfSource snippet
outcome reporting in trials of behavioural health...by K Matvienko-Sikar · 2024 · Cited by 7 — While all reviewed journals instruct auth...
-
Source: cov.com
Link: https://www.cov.com/en/news-and-insights/insights/2023/01/ftc-issues-new-guidance-on-health-related-claims-to-replace-the-dietary-supplements-advertising-guideSource snippet
FTC Issues New Guidance on Health-Related Claims to...5 Jan 2023 — The new guidance broadly applies to all health-related claims, as opp...
-
Source: researchgate.net
Link: [https://www.researchgate.net/publication/272891129Influence_of%27soft%27versus%27scientific%27health_information_framing_and_contradictory_information_on_consumers%27_health_inferences_and_attitudes_towards_a_food_supplement](https://www.researchgate.net/publication/272891129_Influence_of%27soft%27versus%27scientific%27_health_information_framing_and_contradictory_information_on_consumers%27_health_inferences_and_attitudes_towards_a_food_supplement)Source snippet
e effects for these claims on consumers' preferences and purchase behavior [9][10] [...Read more...
-
Source: pearl.plymouth.ac.uk
Link: https://pearl.plymouth.ac.uk/cgi/viewcontent.cgi?article=1348&context=pds-researchSource snippet
of bias, publication bias, and reasons for rating up certaintyby G Guyatt · 2025 · Cited by 52 — To decide whether to rate down certainty...
-
Source: ouci.dntb.gov.ua
Link: https://ouci.dntb.gov.ua/en/works/lDojKdX4/Source snippet
However, the results of previous studies have been contradictory regarding...Read more...
-
Source: equator-network.org
Link: https://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-clinical-trials/Source snippet
Reporting guidelines under development for clinical trialsThe 2022 CONSORT update will provide guidance for reporting all randomized, con...
-
Source: journals.stfm.org
Link: https://journals.stfm.org/primer/2025/venkatesh-2024-0129/Source snippet
Food Packaging Health Claims Depict Healthiness or...by S Venkatesh · 2025 — In our sample, packaging health claims did not consistently...
-
Source: cebm.ox.ac.uk
Link: https://www.cebm.ox.ac.uk/news/views/outcome-reporting-bias-if-you-say-youre-going-to-do-something-do-itSource snippet
Outcome reporting bias: if you say you're going to do...21 Dec 2015 — However, although many journals endorse the CONSORT statement, not...
-
Source: sciencedirect.com
Link: https://www.sciencedirect.com/science/article/pii/S2405457726000422Source snippet
Exploring nutritional myths and [fake news]({{ 'fake-news/' | relative_url }}): Impact and...by M Capocasa · 2026 · Cited by 4 — Misleading dietary claims can influence peop...
-
Source: tandfonline.com
Link: https://www.tandfonline.com/doi/full/10.1080/17437199.2024.2367613Source snippet
Selective outcome reporting in trials of behavioural health...by K Matvienko-Sikar · 2024 · Cited by 7 — Accuracy in detecting inadequat...
Topic Tree