Within Bad Samples
How Many Examples Are Enough?
A few real examples can raise a useful question, but they rarely prove a broad claim about a whole group or trend.
On this page
- Tiny samples and chance variation
- Matching claim size to evidence size
- When small samples justify caution, not certainty
Page outline Jump by section
Introduction
A few examples can be enough to make us curious, concerned, or alert. They are rarely enough to prove a sweeping claim. One of the most common forms of hasty generalisation occurs when people move directly from a small number of cases to a conclusion about an entire group, product, policy, or trend. The reasoning feels persuasive because the examples are real, vivid, and easy to remember. The problem is that a handful of observations often tells us more about chance, circumstance, or selection than about the wider population. Statistical research and decades of work in cognitive psychology show that people routinely overestimate how much a small sample reveals about a larger reality. PMC [2stats.org.uk]stats.org.ukBELIE F IN THE LAW OF SMALL NUMBERSIn particular, they regard a sample randomly drawn from a population as highly representative.Read more…
The key question is not whether the examples happened. It is whether the number and type of examples justify the size of the conclusion being drawn from them.
How a Few Cases Turn into a Broad Claim
The mechanism behind this fallacy is surprisingly simple. A person observes several similar cases and assumes they reveal a stable pattern.
Consider these everyday arguments:
- “Three friends had problems with that airline, so the airline is terrible.”
- “Two teenagers from that neighbourhood were arrested, so the area is full of criminals.”
- “I met several rude tourists from that country, so people from that country are rude.”
- “Five customers posted complaints online, so the product must be defective.”
Each example contains genuine observations. The mistake occurs when those observations are treated as representative of a much larger population without sufficient evidence.
Psychologists Amos Tversky and Daniel Kahneman described a related tendency as the “belief in the law of small numbers”: people often expect small samples to resemble the larger population much more closely than they actually do. In reality, small samples are highly variable and can produce misleading patterns purely by chance. [stats.org.uk]stats.org.ukBELIE F IN THE LAW OF SMALL NUMBERSIn particular, they regard a sample randomly drawn from a population as highly representative.Read more… [PMC]pmc.ncbi.nlm.nih.govPMCHow sample size influences research outcomesVery small samples undermine the internal and external validity of a study.Read more…
The leap from “these cases happened” to “this is generally true” is the critical step where the fallacy occurs.
Tiny Samples and Chance Variation
Small samples are especially dangerous because randomness has a stronger effect when there are few observations.
Imagine flipping a fair coin ten times. Getting eight heads is unusual but not shocking. Flip the same coin ten thousand times and the proportion of heads will usually move much closer to fifty percent. Small samples naturally produce more extreme outcomes than large ones. [Wikipedia]WikipediaInsensitivity to sample sizeAugust 13, 2025 — Insensitivity to sample size is a cognitive bias where people estimate the probability of obtaining a sample statistic…
The same principle applies outside gambling and statistics:
- A new restaurant might receive several negative reviews during its first week.
- A small class might have an unusually high failure rate one year.
- A company might experience several product defects in a short period.
These events may signal a genuine problem. They may also reflect ordinary variation that becomes less dramatic when more cases are observed.
Research on sample size consistently shows that very small samples provide unstable estimates and are more likely to generate misleading conclusions. Findings based on limited observations are therefore more vulnerable to exaggeration and false patterns. PMC [2catalogofbias.org]catalogofbias.orgWrong sample size biasWhen small sample size is used, the risk is high that observations will be due to chance, something studies with la…
This is why striking anecdotes often feel more convincing than they deserve. A dramatic cluster of cases attracts attention precisely because it stands out from what normally happens.
Why Vivid Examples Feel More Convincing Than Statistics
A single memorable story often has more emotional impact than a table of numbers.
Suppose a person hears one powerful account of a medical treatment apparently working wonders. That story may feel more persuasive than a study involving thousands of patients. Yet anecdotal evidence lacks the comparison groups and broader context needed to determine whether the treatment truly caused the outcome. The same issue applies to crime stories, consumer experiences, investment successes, and political examples. [National Academies]nationalacademies.orgNational Academies Chapter: Reference Guide on Statistics–David HKaye and…Anecdotal evidence usually amounts to reports that events of one kind are followed by events of another kind. Typically, the…
Research on persuasion shows that people frequently give substantial weight to individual examples, even when statistical evidence provides a stronger basis for general conclusions. Anecdotes are easier to imagine, easier to remember, and easier to discuss. [Taylor & Francis Online]tandfonline.comTaylor & Francis OnlineCombining Anecdotal and Statistical Evidence in Real-Life…by J Hornikx · 2018 · Cited by 42 — The present artic…
This does not make anecdotes worthless. They can:
- Highlight possible problems.
- Suggest new hypotheses.
- Reveal experiences that deserve investigation.
- Draw attention to overlooked issues.
What they usually cannot do by themselves is establish how common, typical, or representative those experiences are.
Matching Claim Size to Evidence Size
A useful way to evaluate a generalisation is to compare the breadth of the claim with the breadth of the evidence.
The larger the claim, the stronger the evidence must be.
EvidenceReasonable conclusionUnreasonable conclusionThree delayed deliveriesThese deliveries were delayed.The company is always unreliable.Four poor product reviewsSome customers had problems.The product is defective for most users.Several rude encountersThose individuals were rude.People from that group are generally rude.A few policy failuresThe policy may have weaknesses.The policy never works.
The problem is often not that the conclusion is impossible. It is that the evidence is too limited to justify confidence in such a broad statement.
Statistical methodology emphasises that sample adequacy depends on the question being asked, the variability of the population, and the precision required. There is no universal sample size that automatically makes a conclusion reliable. Nevertheless, larger and more representative samples generally support broader claims more effectively than smaller ones. [PMC]pmc.ncbi.nlm.nih.govPMCCan we shift belief in the 'Law of Small Numbers'?PMC - NIHby DVM Bishop · 2022 · Cited by 8 — One cognitive bias demonstrated by Tversky & Kahneman [1] is the 'belief in the law of small… [Wikipedia]WikipediaSample size determinationSample size determination or estimation is the act of choosing the number of observations or replicates to in…
When Small Samples Justify Caution, Not Certainty
Not every response to a small sample is fallacious.
Sometimes a small number of cases is enough to justify concern or further investigation.
For example:
- One confirmed contamination event may justify testing other batches.
- Several unexpected equipment failures may justify an inspection.
- A handful of complaints may justify a review of customer service practices.
Notice the difference. These responses are cautious. They treat the evidence as a warning sign rather than proof of a sweeping conclusion.
A careful reasoner says:
- “This may indicate a problem.”
- “We should investigate further.”
- “More evidence is needed.” [necsus-ejms.org]necsus-ejms.orgAnecdotal evidenceNECSUSby S Cubitt — In the anecdotal method we seek distinctions at least as much as we seek shared features, the nuances that make one r…
A hasty generaliser says:
- “The problem is proven.”
- “This always happens.”
- “The entire group is like this.”
The first approach recognises uncertainty. The second assumes certainty before the evidence warrants it.
Spotting the Fallacy in Everyday Arguments
Several warning signs often indicate that small samples are being turned into sweeping claims:
- The sample can be counted on one hand.
- The examples are highly emotional or memorable.
- No information is provided about the wider population.
- Words such as “all”, “most”, “always”, or “never” appear suddenly.
- The argument relies mainly on personal experience.
- No attempt is made to check whether the examples are typical.
When these features appear together, the risk of hasty generalisation increases substantially.
The Core Lesson
Small samples are valuable starting points. They can reveal possibilities, raise questions, and identify risks that deserve attention. What they usually cannot do is settle broad questions about large populations, social trends, products, or policies.
The central mechanism of this fallacy is a mismatch between evidence and conclusion. A few observations are treated as though they capture the whole picture. The result is overconfidence in claims that extend far beyond what the available evidence can actually support. Statistical reasoning, survey methodology, and research on human judgement all point to the same lesson: a handful of cases may suggest a pattern, but it rarely proves one. [journals.sagepub.com]journals.sagepub.comRevisiting representativeness heuristic classic paradigmsKahneman and Tversky showed that when people make probability judgements, they t… [PMC]pmc.ncbi.nlm.nih.govPMCHow sample size influences research outcomesVery small samples undermine the internal and external validity of a study.Read more… [3stats.org.uk]stats.org.ukBELIE F IN THE LAW OF SMALL NUMBERSIn particular, they regard a sample randomly drawn from a population as highly representative.Read more…
Amazon book picks
Further Reading
Books and field guides related to How Many Examples Are Enough?. Use these as the next step if you want deeper reading beyond the article.
Thinking, Fast and Slow
Famously discusses the tendency to trust small samples too much.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCHow sample size influences research outcomes
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC4296634/Source snippet
Very small samples undermine the internal and external validity of a study.Read more...
-
Source: stats.org.uk
Title: BELIE F IN THE LAW OF SMALL NUMBERS
Link: https://www.stats.org.uk/statistical-inference/TverskyKahneman1971.pdfSource snippet
In particular, they regard a sample randomly drawn from a population as highly representative.Read more...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCCan we shift belief in the ‘Law of Small Numbers’?
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC8889191/Source snippet
PMC - NIHby DVM Bishop · 2022 · Cited by 8 — One cognitive bias demonstrated by Tversky & Kahneman [1] is the 'belief in the law of small...
-
Source: Wikipedia
Title: Insensitivity to sample size
Link: https://en.wikipedia.org/wiki/Insensitivity_to_sample_sizeSource snippet
August 13, 2025 — Insensitivity to sample size is a cognitive bias where people estimate the probability of obtaining a sample statistic...
Published: August 13, 2025
-
Source: catalogofbias.org
Link: https://catalogofbias.org/biases/wrong-sample-size-bias/Source snippet
Wrong sample size biasWhen small sample size is used, the risk is high that observations will be due to chance, something studies with la...
-
Source: Wikipedia
Link: https://en.wikipedia.org/wiki/Sample_size_determinationSource snippet
Sample size determinationSample size determination or estimation is the act of choosing the number of observations or replicates to in...
-
Source: journals.sagepub.com
Link: https://journals.sagepub.com/doi/10.1177/17470218241255916Source snippet
Revisiting representativeness heuristic classic paradigmsKahneman and Tversky showed that when people make probability judgements, they t...
-
Source: Wikipedia
Title: Representativeness heuristic
Link: https://en.wikipedia.org/wiki/Representativeness_heuristicSource snippet
Representativeness heuristicThe representativeness heuristic works by comparing an event to a prototype or stereotype that we already...
-
Source: methods.sagepub.com
Title: 22 nonprobability sampling
Link: https://methods.sagepub.com/hnbk/edvol/the-sage-handbook-of-survey-methodology/chpt/22-nonprobability-samplingSource snippet
sagepub.comNon-probability SamplingAnother advantage of probability samples – which is of extreme practical value – is that the confidenc...
-
Source: youtube.com
Title: Why Small Samples Fool Us: The Law of Small Numbers Explained
Link: https://www.youtube.com/watch?v=A4p3QILDFsoSource snippet
The Law of Small Numbers...
-
Source: youtube.com
Title: The Law of Small Numbers
Link: https://www.youtube.com/watch?v=HoMb4nKTZwgSource snippet
What is Anecdotal Evidence? (Easiest Explanation)...
-
Source: nationalacademies.org
Title: National Academies Chapter: Reference Guide on Statistics–David H
Link: https://www.nationalacademies.org/read/13163/chapter/7Source snippet
Kaye and...Anecdotal evidence usually amounts to reports that events of one kind are followed by events of another kind. Typically, the...
-
Source: tandfonline.com
Link: https://www.tandfonline.com/doi/full/10.1080/0163853X.2017.1312195Source snippet
Taylor & Francis OnlineCombining Anecdotal and Statistical Evidence in Real-Life...by J Hornikx · 2018 · Cited by 42 — The present artic...
-
Source: necsus-ejms.org
Title: Anecdotal evidence
Link: https://necsus-ejms.org/anecdotal-evidence/Source snippet
NECSUSby S Cubitt — In the anecdotal method we seek distinctions at least as much as we seek shared features, the nuances that make one r...
-
Source: jstor.org
Link: https://www.jstor.org/stable/2982094Source snippet
Statistical Assessments as Evidenceby SE Fienberg · 1982 · Cited by 36 — Although there have been other isolated uses of probabilistic an...
Additional References
-
Source: itl.nist.gov
Link: https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htmSource snippet
nist.gov7.2.2.2. Sample sizes requiredThe table below gives sample sizes for a two-sided test of hypothesis that the mean is a given valu...
-
Source: fs.blog
Link: https://fs.blog/mental-model-bias-from-insensitivity-to-sample-size/Source snippet
Farnam StreetMental Model: Bias from Insensitivity to Sample SizeOur bias from insensitivity to sample size, (aka the law of small number...
-
Source: leanscape.io
Link: https://leanscape.io/the-importance-of-identifying-the-right-sample-size-for-business-improvementSource snippet
How to Choose the Right Sample Size for ImprovementA study with a small sample size may not have enough power to detect statistically sig...
-
Source: beyonduxdesign.com
Link: https://www.beyonduxdesign.com/cognition-catalog/insensitivity-to-sample-size/Source snippet
Insensitivity to Sample SizeTversky and Kahneman showed that people often ignore the size of the sample when making judgments based on st...
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/226764072_Anecdotal_Statistical_and_Causal_Evidence_Their_Perceived_and_Actual_PersuasivenessSource snippet
(PDF) Anecdotal, Statistical, and Causal Evidence: Their...In this article, we define anecdotal evidence (also known as narrative eviden...
-
Source: lpwm.com
Link: https://lpwm.com/behavioral-finance/sample-size-neglectSource snippet
Sample Size NeglectSample size neglect is a bias where one evaluates statistical information and arrives at an erroneous conclusion after...
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/322862980_Significance_Errors_Power_and_Sample_Size_The_Blocking_and_Tackling_of_Statistics -
Source: facebook.com
Link: https://www.facebook.com/groups/529807464111948/posts/2317017425390934/Source snippet
Small and large sample sizes in statisticsSmall Sample: If the sample size n is less than 30 (n<30), it is known as small sample. In case...
-
Source: andrewclark.co.uk
Link: https://andrewclark.co.uk/all-media/belief-in-the-law-of-small-numbers -
Source: statisticsbyjim.com
Link: https://statisticsbyjim.com/basics/sample-size/Source snippet
Sample Size Essentials: The Foundation of Reliable StatisticsUnderstand the importance of sample size in statistical analysis. Learn how...
Topic Tree