Within Evidence Gaps
When Five Voices Become Everyone
A few easy-to-reach voices can reveal a real concern while still failing to show what most people think.
On this page
- What convenience samples can and cannot prove
- How representative sampling changes the claim
- Warning words that make small evidence overreach
Page outline Jump by section
Introduction
A convenience sample is a group of people chosen because they are easy to reach: readers of one website, followers of one social-media account, customers in one shop, or the first few people willing to answer a question. Such samples can reveal genuine experiences and concerns. The problem begins when a speaker treats those easy-to-reach voices as if they represent the wider public.
Within logical fallacies, this is a common evidence gap. The evidence may be real, but the claim stretches beyond what the evidence can support. Five conversations, a viral comment thread, or a quick online poll can show that some people hold a view. They cannot automatically show what most people think. Researchers distinguish between convenience samples and representative samples precisely because the route by which people enter a sample affects how safely conclusions can be generalised. [PMC]pmc.ncbi.nlm.nih.govPMCThe Inconvenient Truth About Convenience and Purposive…by C Andrade · 2020 · Cited by 3081 — This sample, however, may not be repre…
What Convenience Samples Can and Cannot Prove
Convenience sampling selects participants based on availability rather than a process that gives members of the target population a known chance of selection. Because of this, the sample may differ systematically from the population it is supposed to represent. [PMC]pmc.ncbi.nlm.nih.govPMCThe Inconvenient Truth About Convenience and Purposive…by C Andrade · 2020 · Cited by 3081 — This sample, however, may not be repre…
What a convenience sample can often show:
- A concern exists.
- A viewpoint is present.
- A problem has occurred.
- Some people are motivated enough to speak up.
- Topics worth investigating further may exist.
What it usually cannot show by itself:
- How common the view is.
- Whether the view represents a majority.
- Whether the view is typical across regions, ages, incomes, or other groups.
- What the wider public would say if measured systematically.
Consider a statement such as: “Everyone is furious about the new policy. Just read the comments.” The comments may indeed contain hundreds of angry responses. Yet the people most motivated to comment are often those with the strongest feelings. Silent readers, indifferent citizens, and people outside that online community are absent from the evidence. The sample demonstrates dissatisfaction among commenters, not necessarily among the public as a whole.
This is why convenience samples frequently become the foundation for hasty generalisations. The leap is not from false evidence to a false conclusion. It is from limited evidence to an overconfident conclusion.
When Visible Voices Distort Public Opinion
Many modern examples arise because visibility is mistaken for representativeness.
A social-media trend may create the impression that “everyone is talking about” an issue. In reality, the discussion may be concentrated among a particular platform’s users, a political community, a professional group, or people with unusually strong interests. The audience is visible, but it is not necessarily representative.
The same problem appears in:
- Website reader polls.
- Call-in radio programmes.
- Open online questionnaires.
- Customer feedback forms.
- Street interviews conducted in a single location.
- Voluntary response surveys.
Researchers often classify these as nonprobability samples because participation is not determined through a probability-based selection process. Pew Research Center notes that many online opt-in surveys are easier and cheaper to conduct, but they raise questions about how accurately they represent the broader population. [Pew Research Center]pewresearch.orgwhat are nonprobability surveysThese polls are generally easier and cheaper for polling organizations to conduct.Read more…
The persuasive force of these samples comes from their vividness. Readers see real people expressing real opinions. What they do not see are the people who were never included.
How Representative Sampling Changes the Claim
The crucial question is not simply how many people were surveyed but how they were selected.
A common misunderstanding is that a large sample automatically solves the problem. It does not. A huge sample drawn in a biased way can still produce misleading conclusions.
The classic example is the 1936 Literary Digest presidential poll in the United States. The magazine mailed millions of ballots and received roughly 2.4 million responses, an enormous number for the period. Yet the poll incorrectly predicted a decisive victory for Alf Landon. Later analysis found that both the sampling method and response patterns were biased, producing a sample that did not represent the electorate. [OUP Academic]academic.oup.comOUP AcademicWHY THE 1936 LITERARY DIGEST POLL FAILEDby P SQUIRE · 1988 · Cited by 314 — Despite its importance, no empirical research has…
The lesson was striking: millions of responses did not compensate for a flawed sample. Meanwhile, scientific polling methods using far smaller but more carefully selected samples produced much better estimates. [mathcenter.oxford.emory.edu]mathcenter.oxford.emory.eduhistorical BlundersFamous Statistical Blunders in HistoryIn 1936, Literary Digest, a national magazine of the time, sent out 10 million "straw" ballots aski…
Representative sampling changes the strength of the claim because it aims to ensure that different parts of the population have a fair chance of inclusion. Probability-based approaches are designed to reduce the risk that the results merely reflect whoever was easiest to contact or most eager to respond. [Pew Research Center]pewresearch.orgwhat are nonprobability surveysThese polls are generally easier and cheaper for polling organizations to conduct.Read more…
As a result, a representative survey may support claims about broader public opinion that a convenience sample cannot.
Why Large Numbers Do Not Automatically Fix the Problem
People often assume that sample size is the only thing that matters. This intuition is understandable: hearing from 10,000 people sounds more convincing than hearing from 100.
However, sampling experts have long emphasised that representativeness and sample selection are fundamental. A large convenience sample can repeatedly measure the wrong thing with great precision if the people included differ from the population of interest. [Pew Research Center]pewresearch.orgwhat are nonprobability surveysThese polls are generally easier and cheaper for polling organizations to conduct.Read more…
Imagine surveying 50,000 members of a single enthusiast forum about a controversial issue. The result may accurately describe that community. It may even describe it extremely accurately. Yet it still may not describe the wider public.
The key distinction is between measuring a group well and measuring the correct group.
Warning Words That Make Small Evidence Overreach
Certain phrases often signal that a convenience sample is being stretched beyond its evidential limits.
Watch for claims such as:
- “Everyone thinks…”
- “People are saying…”
- “The public wants…”
- “Nobody supports…”
- “Voters believe…”
- “Most people agree…”
These phrases describe broad populations. To justify them, the evidence must also be broad and representative.
A more defensible claim might be:
- “Among the people who responded…”
- “Among readers of this site…”
- “Among customers surveyed…”
- “Among commenters on this post…”
Notice how the revised wording stays close to what the evidence actually shows.
The difference may seem minor, but it changes the logic of the argument. One version reports observed responses. The other claims knowledge about a much larger population.
A Quick Test for Readers
When confronted with a claim about public opinion, ask three simple questions:
- Who was actually surveyed or observed?
- How were those people selected?
- Does the conclusion describe only that group, or does it make claims about a wider population?
If the evidence comes from people who were merely easiest to reach, volunteered themselves, or belonged to a single visible community, caution is warranted. The voices may be genuine and important, but they do not automatically become the voice of everyone.
The central evidence gap is straightforward: convenience samples can reveal that an opinion exists, but they cannot by themselves establish how widely that opinion is shared. When arguments ignore that distinction, a small and accessible group is transformed into “the public”, and the evidence is asked to carry more weight than it can support.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC8295573/Source snippet
PMCThe Inconvenient Truth About Convenience and Purposive...by C Andrade · 2020 · Cited by 3081 — This sample, however, may not be repre...
-
Source: academic.oup.com
Link: https://academic.oup.com/poq/article-abstract/52/1/125/1878544Source snippet
OUP AcademicWHY THE 1936 LITERARY [DIGEST POLL]({{ 'digest-poll/' | relative_url }}) FAILEDby P SQUIRE · 1988 · Cited by 314 — Despite its importance, no empirical research has...
-
Source: academic.oup.com
Link: https://academic.oup.com/poq/article/52/1/125/1878544Source snippet
The Literary Digest poll of 1936 holds an infamous place in the history of survey research. Despite its importance, no empirical research...
-
Source: mathcenter.oxford.emory.edu
Title: historical Blunders
Link: https://mathcenter.oxford.emory.edu/site/math117/historicalBlunders/Source snippet
Famous Statistical Blunders in HistoryIn 1936, Literary Digest, a national magazine of the time, sent out 10 million "straw" ballots aski...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC11343077/Source snippet
of combined probability and nonprobability samplesby W Xi · 2022 · Cited by 7 — In scientific studies with low-prevalence outcomes, proba...
-
Source: academic.oup.com
Link: https://academic.oup.com/jssam/article/8/1/4/5699631Source snippet
Instead, most...Read more...
-
Source: academic.oup.com
Link: https://academic.oup.com/poq/article-pdf/52/1/125/5310718/52-1-125.pdfSource snippet
Literary Digest poll gained an infamous place in the history of survey research. Almost every book on presidential elections or survey me...
-
Source: pewresearch.org
Title: what are nonprobability surveys
Link: https://www.pewresearch.org/short-reads/2018/08/06/what-are-nonprobability-surveys/Source snippet
These polls are generally easier and cheaper for polling organizations to conduct.Read more...
-
Source: pewresearch.org
Title: Pew Research Center Nonprobability Surveys
Link: https://www.pewresearch.org/topic/methodological-research/survey-methods/online-surveys/nonprobability-surveys/Source snippet
Pew Research CenterNonprobability Surveys - Research and data from Pew...Many online surveys are conducted using “nonprobability” or “op...
-
Source: pewresearch.org
Title: Pew Research Center Methods 101: Random Sampling
Link: https://www.pewresearch.org/methods/2017/05/12/methods-101-video-random-sampling/Source snippet
Pew Research CenterMethods 101: Random SamplingMay 12, 2017 — The first video in Pew Research Center's Methods 101 series helps explain r...
Published: May 12, 2017
-
Source: pewresearch.org
Link: https://www.pewresearch.org/u-s-survey-methodology/Source snippet
Pew Research CenterU.S. Survey MethodologyPew Research Center has conducted surveys online in the United States using our American Trends...
-
Source: pewresearch.org
Title: building pew research centers american trends panel
Link: https://www.pewresearch.org/methods/2015/04/08/building-pew-research-centers-american-trends-panel/Source snippet
Pew Research CenterBuilding Pew Research Center's American Trends PanelApr 8, 2015 — Panels recruited through non-probability methods or...
-
Source: pewresearch.org
Title: Evaluating Online Nonprobability Surveys
Link: https://www.pewresearch.org/methods/2016/05/02/evaluating-online-nonprobability-surveys/Source snippet
May 2, 2016 — Pew Research Center conducted a study in which an identical 56-item questionnaire was administered to nine samples supplied...
Published: May 2, 2016
-
Source: pewresearch.org
Link: https://www.pewresearch.org/methods/2018/08/06/video-explainer-what-are-nonprobability-surveys/Source snippet
m traditional probability-based polls...
-
Source: pewresearch.org
Title: race lgbtq in schools methodology
Link: https://www.pewresearch.org/social-trends/2024/02/22/race-lgbtq-in-schools-methodology/Source snippet
MethodologyFeb 22, 2024 — The margin of sampling error for the full sample of 2,531 teachers is plus or minus 2.4 percentage points. The...
-
Source: pewresearch.org
Title: assessing the accuracy of online nonprobability surveys
Link: https://www.pewresearch.org/methods/2016/05/02/assessing-the-accuracy-of-online-nonprobability-surveys/Source snippet
May 2, 2016 — Well-designed surveys using either probability or nonprobability samples can provide accurate measures of political prefere...
Published: May 2, 2016
-
Source: pewresearch.org
Title: Here’s how it works
Link: https://www.pewresearch.org/methods/2017/05/12/video-explainer-understanding-random-sampling-for-public-opinion-surveys/Source snippet
Video Explainer: Understanding random sampling for public...The first video in our “Methods 101” series is about random sampling, a conc...
-
Source: pewresearch.org
Title: privacy and information sharing methodology
Link: https://www.pewresearch.org/internet/2016/01/14/privacy-and-information-sharing-methodology/Source snippet
Methodology14 Jan 2016 — Sampling errors and statistical tests of significance take into account the effect of weighting at each of these...
-
Source: pewresearch.org
Title: methodology koerber 2025
Link: https://www.pewresearch.org/global/2025/11/24/methodology-koerber-2025/Source snippet
MethodologyNov 24, 2025 — Panel members are randomly selected so that survey results can properly represent the U.S. population with a me...
-
Source: pewresearch.org
Title: methodology 100 2
Link: https://www.pewresearch.org/internet/2008/02/13/methodology-100-2/Source snippet
MethodologyFeb 13, 2008 — For results based on the total sample, one can say with 95% confidence that the error attributable to sampling...
-
Source: pewresearch.org
Title: comparing two types of online survey samples
Link: https://www.pewresearch.org/methods/2023/09/07/comparing-two-types-of-online-survey-samples/Source snippet
Sep 7, 2023 — Pew Research Center designed this study to assess the current state of online survey sampling methods – both probability-ba...
Additional References
-
Source: uca.edu
Link: https://uca.edu/psychology/files/2013/08/Ch7-Sampling-Techniques.pdfSource snippet
Chapter 7. Sampling TechniquesBefore a sample is taken, we must first define the population to which we want to generalize our results. T...
-
Source: saylordotorg.github.io
Link: https://saylordotorg.github.io/text_principles-of-sociological-inquiry-qualitative-and-quantitative-methods/s10-sampling.htmlSource snippet
Chapter 7: SamplingSampling has to do with selecting some subset of one's group of interest (in this case, babies) and drawing conclusion...
-
Source: pressbooks.openeducationalberta.ca
Link: https://pressbooks.openeducationalberta.ca/communicationsresearchmethods/chapter/6-sampling/Source snippet
– Navigating Communication ResearchA representative sample is one that resembles the population from which it was drawn in all the ways t...
-
Source: researchgate.net
Title: 315940924 Roosevelt Predicted to Win Revisiting the 1936 Literary Digest Poll
Link: https://www.researchgate.net/publication/315940924_Roosevelt_Predicted_to_Win_Revisiting_the_1936_Literary_Digest_PollSource snippet
Roosevelt Predicted to Win: Revisiting the 1936 Literary...The Literary Digest poll of 1936, which incorrectly predicted that Landon wou...
-
Source: pivotalresearch.ca
Title: the poll that changed polling selection bias and the 1936 us election
Link: https://www.pivotalresearch.ca/2024/07/03/the-poll-that-changed-polling-selection-bias-and-the-1936-us-election/Source snippet
The Poll that Changed Polling (Selection bias and the 1936...Jul 3, 2024 — In advance of the 1936 presidential election, the Literary Di...
-
Source: ozanozbey.medium.com
Title: two lessons of sampling bias from 1936 us election e4e96bd42be
Link: https://ozanozbey.medium.com/two-lessons-of-sampling-bias-from-1936-us-election-e4e96bd42beSource snippet
Lessons of Sampling Bias from 1936 US Elections | by OÖ“Literary Digest Poll is on!” announced confidently the editors of Literary Digest...
-
Source: mrisimmons.com
Title: demographics, such samples often fail to be nationally representative.Read more
Link: https://www.mrisimmons.com/2020/08/24/the-inconvenience-of-online-non-probability-surveys/Source snippet
The Inconvenience of Online Non-probability SurveysAug 24, 2020 — While responses from these panelists are typically weighted to match U.S...
-
Source: medium.com
Link: https://medium.com/data-science/nonresponse-bias-how-the-silent-majority-decided-an-election-and-sunk-a-beloved-publisher-90a40e73d3a1Source snippet
oosevelt in tomorrow's voting…” There was only one caveat: the...
-
Source: ecampusontario.pressbooks.pub
Title: pub6.3: Non-probability sampling –
Link: https://ecampusontario.pressbooks.pub/introductiontomarketresearch/chapter/non-probability-sampling/Source snippet
pressbooks.pub6.3: Non-probability sampling – Introduction to Market...One challenge with a convenience sample is selection bias, as the...
-
Source: cambridge.org
Link: https://www.cambridge.org/core/journals/social-science-history/article/president-landon-and-the-1936-literary-digest-poll/E360C38884D77AA8D71555E7AB6B822CSource snippet
“President” Landon and the 1936 Literary Digest Pollby D Lusinchi · 2012 · Cited by 36 — The disastrous prediction of an Alf Landon victo...
Topic Tree