Within False Cause
Did the Policy Work, or Just Arrive First?
A policy followed by better outcomes may deserve attention, but background trends and other changes can explain the same result.
On this page
- Why before and after comparisons feel persuasive
- Background trends, common causes, and rival explanations
- Better ways to judge policy effects
Page outline Jump by section
Introduction
Public debates often treat policy success as obvious: a law is introduced, a statistic improves, and the policy receives the credit. This is a specific form of post hoc reasoning. Because the improvement happened after the policy, people assume the policy caused it. Yet social and economic outcomes are influenced by many forces at once. A falling crime rate, rising employment, improved health outcome, or stronger economy may reflect trends that were already under way, broader national changes, demographic shifts, seasonal patterns, or unrelated events occurring at the same time. The central question is not whether the outcome changed after the policy, but whether it changed because of the policy. Modern policy evaluation exists largely because simple before-and-after comparisons are often misleading. [World Bank]documents1.worldbank.orgWorld Bank Impact Evaluation in PracticeWorld BankImpact Evaluation in Practice - World Bank Documentby PJ Gertler · Cited by 2943 — Its main goal is to expand the evidence base…
Did the Policy Work, or Just Arrive First?
The fallacy appears when an argument follows a simple pattern:
- A policy was introduced.
- An outcome improved or worsened afterwards.
- Therefore, the policy caused the change.
At first glance this reasoning feels sensible. Causes normally precede effects. A policy cannot improve employment before it exists. But timing is only a necessary condition for causation, not sufficient proof of it.
Suppose a city launches a new anti-crime initiative in January and reported crime falls by December. The decline may have been caused by the programme. It may also reflect a long-term decline already visible before January, demographic changes, economic improvements, altered reporting practices, or several influences working together. Looking only at the before-and-after numbers cannot separate these possibilities. [Institute for Fiscal Studies]ifs.org.ukInstitute for Fiscal Studiesoxb08_pol_eval.pptCorrelation is not causality! “Post hoc, ergo propter hoc”: looking at what happens after…
This is why policy analysts distinguish between observed change and attributable change. Observed change is what happened. Attributable change is the portion that can reasonably be linked to the intervention itself. The gap between those two concepts is where false credit and false blame often arise. [Better Evaluation]betterevaluation.orgBetter EvaluationImpact evaluationAn impact evaluation provides information about the observed changes or 'impacts' produced by an interv…
Why Before-and-After Comparisons Feel Persuasive
Before-and-after stories are powerful because they are easy to understand. They offer a simple narrative with a beginning, an intervention, and an outcome.
Several psychological features make these arguments attractive:
- Visible action receives visible credit. People tend to focus on the most recent policy change rather than slower background forces.
- Human beings prefer clear causes. A single explanation is easier to communicate than a complex mixture of influences.
- Political incentives reward certainty. Supporters highlight positive changes after their preferred policy, while opponents emphasise negative developments after policies they dislike.
- Counterfactuals are invisible. We can observe what happened after a policy. We cannot directly observe what would have happened without it.
The last point is particularly important. The real policy question is not whether conditions improved after implementation. It is whether conditions improved more than they would have improved anyway. Impact evaluation literature consistently emphasises the need for a counterfactual—an estimate of what would have occurred in the absence of the intervention. World Bank [JRC Publications]publications.jrc.ec.europa.euJRC PublicationsA note on the impact evaluation of public policiesby M LOI · Cited by 58 — This report describes concisely, and in an int…
Background Trends, Common Causes, and Rival Explanations
When the Trend Started Before the Policy
One common error occurs when a policy receives credit for a trend that was already moving in the same direction.
Imagine unemployment falling steadily for two years before a new jobs programme begins. If unemployment continues falling afterwards, the programme may deserve some credit, but the continuation of an existing trend is not proof of effectiveness. The key question becomes whether the decline accelerated beyond what earlier patterns would predict. Evaluators therefore examine pre-policy trends rather than focusing solely on the implementation date. [World Bank Blogs]blogs.worldbank.orgrevisiting difference differences parallel trends assumption part i pre trendWorld Bank BlogsRevisiting the Difference-in-Differences Parallel Trends…Jan 21, 2020 — Difference-in-differences (DiD) analysis is on…
When Several Things Change at Once
Policies rarely operate in isolation. Economic growth, technological shifts, demographic changes, weather conditions, cultural changes, and other government actions may all affect the same outcome.
For example, a health policy introduced during a period of rising public awareness, medical innovation, and economic improvement may coincide with better health outcomes. Crediting the policy alone ignores the possibility that several causes contributed simultaneously. Policy evaluation frameworks therefore stress examining coherence, interactions, and competing explanations rather than assuming a single cause. [OECD]legalinstruments.oecd.orgOECD Legal InstrumentsRecommendation of the Council on Public Policy EvaluationPublic policy evaluations aim to promote understanding of…
Regression to the Mean
Another source of false credit appears when intervention follows an unusually bad period.
Governments often introduce policies in response to crises: spikes in crime, unusually high unemployment, poor test scores, or disease outbreaks. Extreme outcomes frequently move closer to normal levels over time even without intervention. If a policy is introduced near the peak of a problem, some improvement may occur naturally. Observers may then mistakenly attribute this expected rebound to the policy itself.
This does not mean the policy had no effect. It means that before-and-after comparisons alone cannot determine how much of the improvement resulted from the intervention and how much reflected a return toward typical conditions.
How False Credit Shapes Public Debate
The consequences extend beyond academic disputes.
When policymakers receive credit for outcomes they did not cause, ineffective programmes can survive for years. Resources may be directed toward interventions that merely coincided with favourable trends. Conversely, useful policies may be abandoned if adverse events occur after implementation even when those events were driven by unrelated factors.
Public discussions about economic performance frequently illustrate the problem. Governments inherit economies shaped by previous administrations, international markets, central bank decisions, demographic trends, and global events. Yet political rhetoric often treats any improvement or deterioration after taking office as direct evidence of success or failure. Such claims may contain some truth, but timing alone does not establish the magnitude—or even the existence—of a policy effect.
The same reasoning appears in debates about education reforms, public-health measures, environmental regulations, taxation, and criminal justice policies. In each case, the danger is confusing sequence with causation.
Better Ways to Judge Policy Effects
Recognising the weakness of before-and-after comparisons does not mean policy effects are unknowable. Researchers have developed methods specifically designed to address the post hoc problem.
Compare Against a Similar Group
One of the most common approaches is to compare areas affected by a policy with similar areas that were not affected.
If employment rises where a programme was introduced but rises equally in comparable places without the programme, the policy may not deserve much credit. If the treated areas improve substantially more, the evidence becomes stronger. This logic underlies many quasi-experimental methods used in economics and public policy. [dimewiki.worldbank.org]dimewiki.worldbank.orgDifference-in-Differences | Dime WikiWorld Bank7 Aug 2023 — Difference-in-differences combines these two methods to compare the before-and-after changes in outcomes for treat…
Difference-in-Differences
A widely used method called difference-in-differences compares changes over time in a treated group with changes in a comparison group. Rather than asking whether outcomes improved after a policy, it asks whether they improved more than they did elsewhere over the same period. [World Bank Blogs]blogs.worldbank.orgrevisiting difference differences parallel trends assumption part i pre trendWorld Bank BlogsRevisiting the Difference-in-Differences Parallel Trends…Jan 21, 2020 — Difference-in-differences (DiD) analysis is on…
This approach is not perfect and relies on assumptions that must be examined carefully, but it is generally far more informative than a simple before-and-after comparison. Researchers continue refining these methods precisely because causal attribution is difficult. [PMC]pmc.ncbi.nlm.nih.govLearned From CDC's Prevention Research Centersby S Honeycutt · 2015 · Cited by 99 — Evaluating policy, systems, and environmental change…
Look for Multiple Lines of Evidence
Strong policy evaluation often combines several forms of evidence: [oecd.org]oecd.orgPublic Policy EvaluationImplementation ToolkitIt creates a robust framework of incentives, responsibilities and accountability of different government, encouragi…
- Trends before implementation.
- Comparisons with similar groups or regions.
- Independent datasets.
- Mechanisms explaining how the policy should work.
- Tests for alternative explanations.
- Evidence from different times and locations.
The more a claimed effect survives these checks, the more confidence we can have that the policy genuinely contributed to the outcome. [OECD Legal Instruments]legalinstruments.oecd.orgOECD Legal InstrumentsRecommendation of the Council on Public Policy EvaluationPublic policy evaluations aim to promote understanding of… [OECD]oecd.orgOECDApplying Evaluation Criteria ThoughtfullyThese guidelines provide a framework and advice to help evaluators consider interconnections…
A Practical Question for Readers
When someone claims that a policy succeeded because conditions improved afterwards, a useful response is not immediate acceptance or rejection. Instead, ask three questions:
- Was the trend already moving in that direction before the policy?
- What else changed at the same time?
- How did comparable places or groups perform without the policy?
These questions do not guarantee the right answer, but they shift the discussion from mere timing to evidence of causation. That shift is the essential defence against the logical fallacy of giving a policy credit—or blame—simply because the outcome came later.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: documents1.worldbank.org
Title: World Bank Impact Evaluation in Practice
Link: https://documents1.worldbank.org/curated/en/823791468325239704/pdf/Impact-evaluation-in-practice.pdfSource snippet
World BankImpact Evaluation in Practice - World Bank Documentby PJ Gertler · Cited by 2943 — Its main goal is to expand the evidence base...
-
Source: legalinstruments.oecd.org
Link: https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0478Source snippet
OECD Legal InstrumentsRecommendation of the Council on Public Policy EvaluationPublic policy evaluations aim to promote understanding of...
-
Source: worldbank.org
Title: World Bank Impact Evaluation in Practice
Link: https://www.worldbank.org/en/programs/sief-trust-fund/publication/impact-evaluation-in-practiceSource snippet
Impact Evaluation in Practice - Second EditionJan 19, 2016 — The book incorporates real-world examples to present practical guidelines fo...
-
Source: blogs.worldbank.org
Title: revisiting difference differences parallel trends assumption part i pre trend
Link: https://blogs.worldbank.org/en/impactevaluations/revisiting-difference-differences-parallel-trends-assumption-part-i-pre-trendSource snippet
World Bank BlogsRevisiting the Difference-in-Differences Parallel Trends...Jan 21, 2020 — Difference-in-differences (DiD) analysis is on...
-
Source: blogs.worldbank.org
Title: when your difference differences has too many differences
Link: https://blogs.worldbank.org/en/impactevaluations/when-your-difference-differences-has-too-many-differencesSource snippet
World Bank BlogsWhen your difference-in-differences has too many differencesOct 14, 2020 — Recent papers have proposed methods to robusti...
-
Source: oecd.org
Link: https://www.oecd.org/en/publications/applying-evaluation-criteria-thoughtfully_543e84ed-en/full-report/understanding-the-six-criteria-definitions-elements-for-analysis-and-key-challenges_2843ff7d.htmlSource snippet
OECDApplying Evaluation Criteria ThoughtfullyThese guidelines provide a framework and advice to help evaluators consider interconnections...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC4611860/Source snippet
Learned From CDC's Prevention Research Centersby S Honeycutt · 2015 · Cited by 99 — Evaluating policy, systems, and environmental change...
-
Source: dimewiki.worldbank.org
Title: Difference-in-Differences | Dime Wiki
Link: https://dimewiki.worldbank.org/Difference-in-DifferencesSource snippet
World Bank7 Aug 2023 — Difference-in-differences combines these two methods to compare the before-and-after changes in outcomes for treat...
-
Source: blogs.worldbank.org
Link: https://blogs.worldbank.org/en/impactevaluations/what-are-we-estimating-when-we-estimate-difference-differencesSource snippet
World Bank BlogsWhat Are We Estimating When We Estimate Difference-in-...Sep 30, 2019 — In DD estimation, a researcher compares the chan...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC11305929/Source snippet
PMCAdvances in Difference-in-differences Methods for Policy...by G Wang · 2024 · Cited by 108 — Difference-in-differences (DiD) is a pow...
-
Source: oecd.org
Title: Public Policy Evaluation
Link: https://www.oecd.org/content/dam/oecd/en/publications/reports/2025/02/implementation-toolkit-for-the-oecd-recommendation-on-public-policy-evaluation_f24516be/77faa4fe-en.pdfSource snippet
Implementation ToolkitIt creates a robust framework of incentives, responsibilities and accountability of different government, encouragi...
-
Source: legalinstruments.oecd.org
Link: https://legalinstruments.oecd.org/api/download/?name=policy-evaluation-comparative-study-highlights.pdf&uri=%2Fprivate%2Ftemp%2F7ecfe9af-5b05-4992-9a10-ebdc45a944a8.pdfSource snippet
It can help governments improve the design and implementation of public policies...Read more...
-
Source: oecd.org
Link: https://www.oecd.org/en/about/projects/oecd-ec-project-on-policy-impact-evaluation-through-the-use-of-linked-administrative-and-survey-data.htmlSource snippet
OECD-EC project on policy impact evaluation through the...These evaluations (especially counterfactual impact evaluations, CIEs) provide...
-
Source: oecd.org
Title: ai in policy evaluation c88cc2fd
Link: https://www.oecd.org/en/publications/2025/06/governing-with-artificial-intelligence_398fa287/full-report/ai-in-policy-evaluation_c88cc2fd.htmlSource snippet
AI in policy evaluation: Governing with Artificial Intelligence18 Sept 2025 — Some OECD countries have developed ways in which different...
-
Source: documents1.worldbank.org
Title: Impact evaluation in practice
Link: https://documents1.worldbank.org/curated/en/698441474029568469/pdf/Impact-evaluation-in-practice.pdfSource snippet
Evaluation in Practice - World Bank Documentoutcome trends are different for the treatment and comparison groups, then the estimated trea...
-
Source: openknowledge.worldbank.org
Link: https://openknowledge.worldbank.org/entities/publication/dd5a1c37-56a7-5b90-a1f9-508e7a3a7632Source snippet
worldbank.orgPublication: Impact Evaluation in International Developmentby P Glewwe — The Impact Evaluation in Practice handbook is a com...
-
Source: blogs.worldbank.org
Title: can we trust shoestring evaluations
Link: https://blogs.worldbank.org/en/impactevaluations/can-we-trust-shoestring-evaluationsSource snippet
we trust shoestring evaluations?Mar 7, 2012 — Probably the thing that practitioners would most like to avoid is the need for baseline dat...
-
Source: openknowledge.worldbank.org
Link: https://openknowledge.worldbank.org/entities/publication/1b84ccc9-7f78-5398-8495-5daa1c85327cSource snippet
worldbank.orgPublication: Reconstructing Baseline Data for Impact...This note discusses the reasons why baseline studies are often not c...
-
Source: openknowledge.worldbank.org
Link: https://openknowledge.worldbank.org/entities/publication/e9a20ecb-ad43-58ee-b66e-67ac6164c048Source snippet
worldbank.orgPublication: Behind the ScenesThis paper aims at filling this gap by describing and discussing experiences from four large i...
-
Source: worldbank.org
Title: impact evaluations
Link: https://www.worldbank.org/en/topic/education/publication/impact-evaluationsSource snippet
for Education PolicyOct 10, 2024 — Covering early childhood to secondary education, this publication compiles a decade's worth of educati...
-
Source: ifs.org.uk
Link: https://ifs.org.uk/sites/default/files/output_url_files/oxb08_pol_eval.ppt -
Source: betterevaluation.org
Link: https://www.betterevaluation.org/methods-approaches/themes/impact-evaluationSource snippet
Better EvaluationImpact evaluationAn impact evaluation provides information about the observed changes or 'impacts' produced by an interv...
-
Source: publications.jrc.ec.europa.eu
Link: https://publications.jrc.ec.europa.eu/repository/handle/JRC74778Source snippet
JRC PublicationsA note on the impact evaluation of public policiesby M LOI · Cited by 58 — This report describes concisely, and in an int...
-
Source: fsnnetwork.org
Title: world bank impact evaluation in practice
Link: [https://fsnnetwork.org/sites/default/files/world_bank_-impact_evaluation_in_practice.pdf](https://fsnnetwork.org/sites/default/files/world_bank-_impact_evaluation_in_practice.pdf)Source snippet
Impact Evaluation in Practiceby PJ Gertler · Cited by 2943 — It is also dependent on the availability of data with sufficient coverage of...
-
Source: Wikipedia
Link: https://en.wikipedia.org/wiki/WorldSource snippet
WorldThe world or universe is commonly defined as "the totality of all space and time; all that is, has been, and will be"...
-
Source: ieg.worldbankgroup.org
Link: https://ieg.worldbankgroup.org/sites/default/files/Data/reports/oed_impact_evaluation_methods.pdfSource snippet
of Impact Evaluation Methodologies Used By The...by AG Kapoor · Cited by 22 — This method is often used in situations where an intervent...
-
Source: betterevaluation.org
Link: https://www.betterevaluation.org/methods-approaches/methods/difference-differenceSource snippet
Difference-in-differenceDifference-in-difference involves comparing the before-and-after difference for the group receiving the intervent...
Additional References
-
Source: nationalacademies.org
Link: https://www.nationalacademies.org/read/25631/chapter/4Source snippet
Chapter: 3 Policy-Relevant Evidence for Population HealthRead chapter 3 Policy-Relevant Evidence for Population Health: Promise and Chall...
-
Source: dmeo.gov.in
Link: https://dmeo.gov.in/sites/default/files/2022-01/Difference%20in%20Differences.docxSource snippet
Difference in Differences.docxA treatment group is enrolled in a program, and a comparison group is not enrolled. The before-and-after ou...
-
Source: goltc.org
Link: https://goltc.org/publications/difference-in-differences-approach/Source snippet
Difference-in-Differences approachDifference-in-differences (DiD) is a quasi-experimental method used to estimate the effect of an interv...
-
Source: emerald.com
Link: https://www.emerald.com/rausp/article/54/4/519/359801/Impact-evaluation-using-Difference-in-DifferencesSource snippet
Impact evaluation using Difference-in-DifferencesThis double difference can be calculated whenever treatment and control group data on th...
-
Source: congreso.es
Link: https://www.congreso.es/docu/docum/ddocum/dosieres/sleg/legislatura_14/spl_52/pdfs/28.pdfSource snippet
Improving Governance with Policy EvaluationFailing to rise to the policy evaluation challenge may entail serious consequences for economi...
-
Source: scienceetbiencommun.pressbooks.pub
Link: https://scienceetbiencommun.pressbooks.pub/pubpolevaluation/chapter/difference-in-differences-method/Source snippet
pressbooks.pubDifference-in-differences Method – Policy Evaluationby D Fougère · Cited by 5 — The difference-in-differences method is a q...
-
Source: americanprogress.org
Title: measuring the health outcomes of social economic and environmental policies
Link: https://www.americanprogress.org/article/measuring-the-health-outcomes-of-social-economic-and-environmental-policies/Source snippet
Measuring the Health Outcomes of Social, Economic, and...18 Apr 2022 — A variety of analytic tools can clarify public health priorities...
-
Source: cdc.gov
Link: https://www.cdc.gov/polaris/media/pdfs/2024/09/UsingEvaluationtoInformCDCsPolicyProcess.pdfSource snippet
• Understand the importance of using evaluation findings to inform the...Read more...
-
Source: planapp.gov.pt
Link: https://www.planapp.gov.pt/wp-content/uploads/2023/01/OECD-PLANAPP-Evaluation-M2.pdfSource snippet
A majority of countries have developed a legal framework that guides policy evaluation across...Read more...
-
Source: cdn.ymaws.com
Link: https://cdn.ymaws.com/www.safestates.org/resource/resmgr/evaluation_resources_webpage/CDC_Policy_Evaluation_Briefs.pdfSource snippet
1: Overview of Policy EvaluationThis brief provides a definition of policy evaluation, including a description of the evaluation framewor...
Topic Tree