Centuries of public policy variation across thousands of jurisdictions (countries, states, cities) constitute a massive natural experiment. The data to identify which policies maximize welfare exists but has not been systematically harvested.
The Optimal Policy Generator (OPG) applies causal inference methods (synthetic control, difference-in-differences, regression discontinuity) and Bradford Hill criteria to this cross-jurisdictional data, measuring policy impact on two terminal general-welfare metrics: real after-tax median income growth and median healthy life years.
For any jurisdiction, OPG produces four categories of public policy recommendations: ENACT (evidence-supported policies the jurisdiction lacks), REPLACE (policies set at suboptimal levels), REPEAL (policies with net welfare harm), and MAINTAIN (policies aligned with evidence). Each recommendation includes expected effects on both metrics, confidence grades, and blocking factors including freedom and autonomy constraints.
Thousands of jurisdictions have been running different policies for centuries. Some of them got richer. Some of them got healthier. Some of them did both. The data to figure out which policies worked already exists. Nobody has systematically harvested it. This is like having centuries of clinical trial data in a filing cabinet and never opening the cabinet.
This specification describes the Optimal Policy Generator (OPG), a framework for producing jurisdiction-specific policy recommendations from quasi-experimental evidence. OPG measures policy impact on two terminal general-welfare metrics: real after-tax median income growth (economic welfare) and median healthy life years (health welfare). These metrics capture the primary welfare effects of most policies while remaining directly interpretable.
OPG answers four questions: “What should we add? Change? Remove? Keep?” The framework operates at any jurisdiction level (country, state, county, city) and produces four outputs:
Enact: New policies the jurisdiction should adopt
Replace: Existing policies to modify
Repeal: Harmful policies to remove
Maintain: Current policies aligned with evidence
Each recommendation includes expected effects on both metrics, confidence grades, and blocking factors (see Blocking Factors).
OPG measures policy impact using the two-metric welfare function defined in the Optimocracy2 Framework. These are the terminal general-welfare metrics, while the terminal values are pleasant consciousness and unpleasant consciousness:
Real after-tax median income growth (pp/year) - economic welfare
Median healthy life years (years) - health welfare
See the Optimocracy paper for full justification of these metric choices, data sources, and the welfare function formula.
The two terminal general-welfare metrics: having money and being alive to spend it. You’d think this would be obvious, but governments often forget the second bit.
Why Only Two Metrics?
Simplicity: These two metrics capture the primary welfare dimensions affected by most policies while remaining directly interpretable. No complex conversion factors (VSL, QALY→$) are needed.
Coverage gap: Freedom and autonomy concerns are handled as blocking factors rather than adding metric complexity. A policy that improves income and health but restricts freedom is flagged, not silently scored. Environmental impacts and distributional effects are tracked as supplementary indicators where data permits.
Income Metric Definition
Real after-tax median income growth is defined narrowly as: wages, salaries, and self-employment income, minus taxes paid. This metric captures what appears in household budgets.
What counts as income effects:
Wage increases from productivity gains (e.g., fewer sick days → measurably higher wages)
Tax changes that directly affect take-home pay
Employment effects that translate to wage income
What does NOT count as income effects:
Healthcare cost savings (these are health system efficiency gains, not personal income)
Reduced insurance premiums (unless they translate to higher take-home pay via employer pass-through)
Quality-of-life improvements that don’t appear in wages
Implication for policy analysis: This creates genuine tradeoffs that the two-metric framework makes explicit. A tobacco tax, for example, may show:
Income effect: Negative for smokers (direct tax burden), partially offset by productivity gains for those who quit
Health effect: Positive (reduced smoking → longer healthy life)
The framework does not hide this tradeoff by claiming healthcare cost savings are “income gains.” If a policy improves health but costs money, both effects are reported honestly. This is a feature, not a bug: it prevents corner solutions (an infinite tobacco tax would maximize health but devastate income for smokers) and surfaces the welfare tradeoff for democratic deliberation.
Outcome Translation Methodology
While OPG uses only two terminal metrics (income growth and healthy life years), evidence often measures surrogate outcomes (smoking rates, traffic deaths, crime rates). This section specifies how surrogate outcomes are translated to the terminal metrics.
Where CV is the coefficient of variation at each translation step. A three-step translation with 30% uncertainty at each step yields ~52% uncertainty in the terminal metric.
Important clarification: The claim that “no complex conversion factors are needed” in the abstract refers to the terminal metrics themselves (income and health are directly interpretable, unlike utility or welfare indices). Translation from surrogate outcomes to terminal metrics does require conversion factors, which must be documented and include uncertainty bounds.
The Evidence Base: Centuries of Natural Policy Experiments
Every jurisdiction that enacted a policy created a natural experiment. The evidence to know what works already exists, scattered across thousands of jurisdictions and hundreds of years. OPG systematically harvests this evidence.
Scale of Available Natural Experiments
Level
Jurisdictions
Years
Policy-Years
US States
50
70+
3,500+
Countries
200+
230+
46,000+
EU Regions
300+
50+
15,000+
US Counties
3,000+
50+
150,000+
Cities worldwide
10,000+
varies
millions
Each policy change creates a before/after comparison. Each jurisdiction that didn’t adopt creates a control group. This represents a vast, largely untapped evidence base.
US states give you 3,500 policy-years of data. Cities worldwide give you millions. It’s like comparing a cookbook to the entire history of food.
The OPG Pipeline
Data goes in, gets organized, analyzed, scored, then spits out recommendations. It’s a sausage factory, but for telling politicians what works instead of what kills you.
┌─────────────────────────────────────────────────────────────────┐
│ OPG EVIDENCE PIPELINE │
├─────────────────────────────────────────────────────────────────┤
│ │
│ 1. INGEST │
│ └── All policy changes with dates, jurisdictions, details │
│ │
│ 2. ALIGN │
│ └── Match policies to outcome time series by jurisdiction │
│ │
│ 3. ANALYZE │
│ └── Apply quasi-experimental methods (synth control, DiD) │
│ │
│ 4. SCORE │
│ └── Compute Policy Impact Scores using Bradford Hill │
│ │
│ 5. RANK │
│ └── Generate jurisdiction-specific recommendations │
│ │
└─────────────────────────────────────────────────────────────────┘
Why This Hasn’t Been Done Before
Data fragmentation: Policy records scattered across legislative databases, government archives, academic papers
Computational limits: Meta-analysis at this scale requires modern infrastructure
Methodological advances: Synthetic control (2003), modern DiD (2021) are recent
Incentive structures: No existing institution has mandate + capability + incentive
OPG aggregates fragmented evidence, applies modern causal inference at scale, and produces actionable output.
Four reasons this was impossible before: scattered data, slow computers, bad methods, nobody cared. Now: fast computers, good methods, some people care. Progress is three steps forward, four barriers removed.
System Overview
What Policymakers See
A jurisdiction-specific dashboard showing which policies to enact, replace, repeal, or maintain, ranked by expected welfare impact:
See Appendix A for a complete worked example showing jurisdiction-specific recommendations.
What Policy Analysts See
Eight different types of data combine to tell you if a policy actually works. Like ingredients in a recipe, except this one tells you which recipes poison people.
Effect estimates with standard errors, confidence intervals, and heterogeneity statistics
Policy Impact Scores (PIS) for each policy-outcome relationship (intermediate metric)
Bradford Hill criteria scores for causality assessment
Analysis method used (synthetic control, DiD, RDD) with quality diagnostics
Confounders controlled and potential threats to validity
Natural experiments identified for validation opportunities
Jurisdiction-specific adjustments based on demographics, existing policies, and context
Introduction
Why Policy Ranking Fails Today
Current policy adoption follows a process dominated by political economy dynamics well-documented in the public choice literature5,6:
Lobbying intensity: Policies that benefit concentrated interests (with resources to lobby) are adopted over policies that benefit diffuse majorities7,8
Ideological priors: Policymakers filter evidence through pre-existing beliefs, accepting studies that confirm priors and rejecting those that don’t
Anecdote-driven reasoning: Vivid individual cases drive policy more than systematic evidence (“If it saves one child…”)
Status quo bias: Existing policies persist regardless of evidence because change requires political capital
The result: welfare losses from documented policy failures. Evidence-based policy movements have attempted to address these failures9,10, but lack systematic, jurisdiction-specific recommendation generation.
Evidence says policy X works. But lobbying, fear of change, and shiny distractions filter it out. It’s like having the cure but drinking the poison because the bottle is prettier.
Scale of Available Evidence
The evidence base comprises millions of policy-years of natural experiments across all jurisdictional levels (see The Evidence Base: Centuries of Natural Policy Experiments for detailed counts). Systematically analyzing this data provides a far better basis for policy adoption than the current system of lobbying-driven, ideology-filtered decision-making.
Current system: decide based on feelings, maybe 10 examples. New system: decide based on millions of examples. It’s the difference between astrology and astronomy, but for governance.
Contributions
Methodological: A systematic framework for translating quasi-experimental evidence into jurisdiction-specific policy recommendations, extending beyond generic evidence ratings to actionable output in four categories (enact/replace/repeal/maintain).
Evidence becomes a score. Score tells you: do this new thing, swap that old thing, stop doing that terrible thing, or keep doing that good thing. It’s like Marie Kondo, but for laws.
Taxonomic: We formalize the four recommendation types and introduce the Policy Impact Score (PIS) as an intermediate metric combining effect magnitude, causal confidence (Bradford Hill criteria), and methodological quality. This provides a standardized approach to evidence aggregation.
Applied: We demonstrate the complete framework with a worked example for Texas traffic safety policy, showing how generic effect estimates are translated into context-adjusted, prioritized recommendations with blocking factors and tracking guidance.
Performance Benchmarking
OPG is designed to generate deployable, evidence-weighted recommendations. Retrospective and prospective benchmarking refine the thresholds, context adjustments, and prioritization rules over time (see Benchmarking Framework).
Related Work
Existing Policy Evaluation Frameworks
Regulatory Impact Analysis (RIA): Required by executive order for US federal regulations since 198111. RIA estimates costs and benefits of proposed rules but: (1) applies only to new regulations, not the existing policy stock; (2) lacks systematic cross-jurisdiction evidence aggregation; (3) does not produce jurisdiction-specific recommendations for subnational governments.
What Works Clearinghouse (WWC): The Institute of Education Sciences operates WWC to review education interventions against methodological standards12. WWC demonstrates that systematic evidence synthesis is feasible, but: (1) covers only education; (2) provides generic intervention ratings, not jurisdiction-specific recommendations; (3) does not quantify expected welfare gains.
Cochrane and Campbell Collaborations: These systematic review organizations cover healthcare13 and social policy14 respectively. They represent the gold standard for evidence synthesis but: (1) produce narrative reviews rather than quantitative recommendations; (2) provide no jurisdiction-specific output; (3) operate on slow update cycles (years between reviews).
Congressional Budget Office (CBO): CBO provides nonpartisan fiscal scoring of proposed legislation. While valuable for budget discipline, CBO: (1) estimates budgetary effects rather than welfare; (2) evaluates what is proposed rather than what should be proposed; (3) is reactive rather than proactive.
Benefit-Cost Analysis Tradition: The broader benefit-cost literature15,16 provides theoretical foundations for policy evaluation but typically focuses on individual project or regulation assessment rather than systematic cross-jurisdiction recommendation generation.
This Framework’s Contribution
How OPG is different from traditional evaluation: it’s personalized, comprehensive, and uses actual data instead of vibes.
OPG differs from existing approaches by:
Producing jurisdiction-specific recommendations rather than generic evidence ratings
Covering the full policy stock (enact/replace/repeal/maintain) not just new proposals
Aggregating quasi-experimental evidence via meta-analysis with heterogeneity quantification
Applying Bradford Hill criteria systematically to assess causality confidence
Including subsidiarity guidance (optimal jurisdictional level) and tracking for continuous improvement
Theoretical Framework
The Policy Optimization Problem
Let \(\mathcal{P}\) denote the set of available policies. For jurisdiction \(j\), let \(P_j \subseteq \mathcal{P}\) denote the current policy bundle. Welfare under policy bundle \(P\) is defined using the two core metrics:
\(\text{IncomeGrowth}_j(P)\) = Real after-tax median income growth (pp/year)
\(\text{HealthyYears}_j(P)\) = Median healthy life expectancy (years)
\(\alpha = 0.5\) (default equal weighting; can be adjusted for jurisdiction priorities)
The social planner’s problem: \[
P_j^* = \arg\max_{P \subseteq \mathcal{P}} W_j(P) \quad \text{subject to feasibility constraints}
\]
Assumption 1 (Additive Separability): For tractability, assume each metric is approximately additively separable across policies: \[
\text{IncomeGrowth}_j(P) \approx \sum_{p \in P} \beta^{\text{inc}}_{jp} + \varepsilon_{\text{inc}}
\]\[
\text{HealthyYears}_j(P) \approx \sum_{p \in P} \beta^{\text{hlth}}_{jp} + \varepsilon_{\text{hlth}}
\]
where \(\beta^{\text{inc}}_{jp}\) and \(\beta^{\text{hlth}}_{jp}\) are the marginal effects of policy \(p\) on each metric in jurisdiction \(j\), and interaction terms are assumed to be second-order.
Two circles: what you do now, what you should do. The bits that don’t overlap are where people are dying unnecessarily. Venn diagrams finally do something useful.
Justification and limitations: Additive separability is a standard simplifying assumption in policy analysis (see11 for regulatory impact analysis applications). This assumption is most valid when: (1) policies operate through distinct mechanisms, (2) jurisdictions have not reached saturation in any policy domain, and (3) policies do not create complementarities or substitution effects. When these conditions fail (for example, when a carbon tax interacts with renewable energy subsidies), the marginal effects may be mis-estimated.
Policy Interaction Detection:
OPG flags potential interaction effects using the following heuristics:
Effect heterogeneity test: If a policy’s effect varies significantly depending on whether another policy is present, flag the pair as potentially interacting.
Known interaction database: Documented policy complementarities and substitutes:
Policy A
Policy B
Interaction Type
Evidence
Seat belt law
Speed limit
Complementary
Both target crash fatalities
Nutrition labeling
School lunch programs
Complementary
Both improve dietary outcomes
Tobacco tax
Smoking ban
Complementary
Reinforce each other
Income tax cut
Sales tax increase
Substitutable
Offsetting fiscal effects
Sensitivity analysis recommendation: For high-priority recommendations, report: “How would this recommendation change if policies X and Y interact?” with bounds on combined effect.
Proposition 1 (Policy Gap Characterization): Under Assumption 1, the welfare-optimal policy set satisfies: \[
P_j^* = \{p \in \mathcal{P} : w_j(p) > 0\}
\]
and the policy gap for jurisdiction \(j\) is: \[
\Delta_j = (P_j^* \setminus P_j) \cup (P_j \setminus P_j^*)
\]
where \((P_j^* \setminus P_j)\) represents beneficial policies the jurisdiction lacks (enact candidates) and \((P_j \setminus P_j^*)\) represents harmful policies the jurisdiction has (repeal candidates). See Policy Gap Analysis for the operational implementation.
Proof: Direct consequence of additive separability. Include policy \(p\) if and only if \(w_j(p) > 0\). ∎
Evidence Aggregation Properties
Proposition 2 (PIS as Precision-Weighted Evidence): Under random-effects meta-analysis with between-jurisdiction variance \(\tau^2\), the pooled effect estimate \(\hat{\beta}_{\text{pooled}}\) is (see Global (Aggregate) PIS Calculation for implementation): \[
\hat{\beta}_{\text{pooled}} = \frac{\sum_j \frac{1}{\text{SE}_j^2 + \tau^2} \hat{\beta}_j}{\sum_j \frac{1}{\text{SE}_j^2 + \tau^2}}
\]
with variance: \[
\text{Var}(\hat{\beta}_{\text{pooled}}) = \frac{1}{\sum_j \frac{1}{\text{SE}_j^2 + \tau^2}}
\]
Proof: Standard random-effects meta-analysis derivation (DerSimonian-Laird). ∎
meaning the pooled estimate explains less than 25% of cross-jurisdiction variation. Context-specific estimates are required rather than direct application of the pooled effect. This constraint is operationalized in Evidence Grading.
Proof: By definition, \(I^2 = \frac{\tau^2}{\tau^2 + \bar{\sigma}^2}\) where \(\bar{\sigma}^2\) is typical within-study variance. When \(I^2 > 0.75\), between-study variance dominates, and the pooled estimate provides limited information about any individual jurisdiction’s true effect. ∎
Information Value
Proposition 4 (Value of Additional Evidence): The expected value of information from an additional jurisdiction study is: \[
\text{VOI} = E[\max_{a \in \{adopt, reject\}} U(a | \text{new data})] - \max_{a} E[U(a | \text{current data})]
\]
which is maximized when prior uncertainty is high and decision stakes are large.
Corollary 1 (Trial Prioritization): Policies with (1) high prior variance in effect estimates, (2) large potential welfare impact, and (3) low trial cost should be prioritized for experimental validation. See Trial Prioritization for implementation.
Core Methodology
Policy-Outcome Data Structure
The OPG system uses a relational database schema. The following is a reference implementation showing the conceptual data model; production deployments may vary.
How the database connects policies to outcomes. It’s plumbing, but for knowledge instead of waste. Although some policies are also waste.
Core Tables
-- Hierarchical jurisdictions (country > state > county > city)jurisdictions (id, name, jurisdiction_type, -- 'country', 'state', 'county', 'city' parent_id, -- FK to parent jurisdiction (e.g., Texas -> USA) iso_code, population, gdp_per_capita, constitution_type, -- constraints on policy space data_quality_score, -- how complete is our policy inventory? latitude, longitude, ...)-- Policy types (canonical definitions)policy_types (id, name, policy_category_id, policy_type, is_continuous, typical_onset_delay_days, typical_duration_of_effect_years, canonical_text, ...)-- Current policy inventory by jurisdictionjurisdiction_policies ( jurisdiction_id, policy_type_id, has_policy BOOLEAN, policy_strength, -- e.g., tobacco tax amount, not just yes/no implementation_date, policy_details_json, data_source, last_verified)-- Two core welfare metrics (fixed schema)outcome_metrics (id, metric_type ENUM('income', 'health'), -- Only two types jurisdiction_id, measurement_date,value, -- pp/year for income; years for health confidence_interval_low, confidence_interval_high, data_source -- Census/BLS for income; WHO/BRFSS for health)-- Policy recommendations (generated output)policy_recommendations ( jurisdiction_id, policy_type_id, recommendation_type, -- 'enact', 'replace', 'repeal', 'maintain' current_status, -- what they have now (NULL if nothing) recommended_target, -- what evidence suggests-- Two-metric effects income_effect_pp, -- Expected effect on median income growth (pp/year) income_effect_ci_low, income_effect_ci_high, health_effect_years, -- Expected effect on healthy life years health_effect_ci_low, health_effect_ci_high, evidence_grade, priority_score, blocking_factors, -- 'constitutional', 'federal_preemption', 'political', 'autonomy', etc. similar_jurisdictions,-- Jurisdictional level guidance minimum_effective_level, recommended_level,-- Tracking for feedback loop tracking_frequency, tracking_baseline_method, last_generated)
Policy Types
Type
Description
Example
Measurement
law
Statutory law passed by legislature
Environmental regulation law
Binary (exists/not)
regulation
Administrative rule by agency
Agency emission standards
Continuous (stringency)
tax_policy
Tax rate, bracket, credit, deduction
Investment income tax rate
Continuous (rate)
budget_allocation
Spending decision
Education spending per pupil
Continuous ($/capita)
executive_order
Executive action
Enforcement priority directive
Binary
court_ruling
Judicial precedent
Constitutional interpretation
Binary
treaty
International agreement
Multilateral cooperation treaty
Binary
local_ordinance
Municipal rule
Land use restrictions
Categorical
Analysis Methods
Different ways to figure out if policies work when you can’t run proper experiments because ethics committees get upset about randomly killing control groups.
The OPG system supports multiple quasi-experimental designs, reflecting the “credibility revolution” in applied economics17. Each method is appropriate for different data structures18:
Synthetic Control Method
Use case: Single treated jurisdiction, good donor pool of similar untreated jurisdictions.
Method: Construct a “synthetic” control as a weighted average of untreated jurisdictions that matches the treated jurisdiction’s pre-treatment outcome trajectory. Post-treatment divergence estimates the causal effect.
Quality metrics:
pre_treatment_rmse: How well does synthetic control match pre-treatment? (Lower is better)
placebo_p_value: Permutation test comparing treated effect to placebo effects (Lower is better)
Example: Effect of a state tobacco tax increase on smoking rates, using similar states without tax changes as donors19,20. For comprehensive reviews of the synthetic control method, see21.
Two lines run parallel, then one gets the policy and diverges. The gap between them is how much the policy helped or hurt. It’s like twins, but one gets vegetables.
Method: Compare pre-post change in treated jurisdictions to pre-post change in control jurisdictions. Difference of differences estimates treatment effect. For settings with staggered adoption, modern estimators account for heterogeneous treatment effects across cohorts22.
Quality metrics:
parallel_trends_test_stat: Test statistic for pre-treatment trend equality
parallel_trends_p_value: P-value for parallel trends test (Higher is better, want to fail to reject)
Example: Effect of occupational licensing reforms across states with different adoption timing.
Regression Discontinuity Design (RDD)
Use case: Sharp eligibility threshold determines treatment assignment.
Dots on either side of a line, big jump at the cutoff. People just above the line do better. It’s like being born one day later and getting free healthcare.
Method: Compare outcomes just above vs. just below the threshold. If other characteristics are smooth across the threshold, the discontinuity in outcomes estimates the causal effect.
Quality metrics:
Bandwidth selection diagnostics
McCrary density test for manipulation
Covariate balance at threshold
Example: Effect of program eligibility on outcomes at an income or age threshold (e.g., retirement benefits at age 65).
Event Study / Interrupted Time Series
Use case: Need to visualize pre-trends and dynamic treatment effects.
Nothing happens, nothing happens, nothing happens, policy hits, then things change. It’s like a heart rate monitor, but for legislation instead of life.
Method: Estimate treatment effects at each time period relative to treatment, including leads (pre-treatment) and lags (post-treatment).
Quality metrics:
Pre-treatment coefficients should be near zero (no anticipation)
Post-treatment coefficients show effect dynamics
Example: Effect of unemployment insurance extensions on job search behavior, showing both anticipation effects (before benefits expire) and persistence of impact (after return to baseline).
Confidence Weighting by Method
The following weights reflect default values based on the methodological rigor hierarchy in applied economics.
Method
Base Confidence Weight
Rationale
Randomized experiment
1.00
Gold standard; rare for policies
Regression discontinuity
0.90
Local randomization at threshold
Synthetic control
0.85
Good pre-treatment fit implies validity
Difference-in-differences
0.80
Requires untestable parallel trends
Event study
0.75
Descriptive of dynamics; less rigorous
Interrupted time series
0.65
Single-unit; history threats
Simple before-after
0.40
No control group; confounding likely
Cross-sectional
0.25
Snapshot; severe confounding
Bradford Hill Criteria Scoring Functions
Bradford Hill’s criteria for causality23, originally developed for epidemiology, are operationalized here as explicit scoring functions. Each criterion maps to a saturation function that produces a score in \([0, 1]\).
Take nine different ways to check if something causes something else, squish them into numbers between 0 and 1. Science loves turning confidence into decimals.
Strength of Association
Larger effect estimates provide stronger evidence. We use an exponential saturation function:
Where \(|\hat{\beta}_{\text{std}}|\) is the absolute standardized effect size and \(\beta_{\text{sig}} = 0.3\) is the saturation parameter.
Parameter justification: The threshold \(\beta_{\text{sig}} = 0.3\) corresponds to Cohen’s convention for a “medium” effect size in social science24. This is a starting point; sensitivity analysis shows PIS changes by ±15% when \(\beta_{\text{sig}}\) varies from 0.2 to 0.4. A standardized effect of 0.3 yields \(S_{\text{strength}} \approx 0.63\); effects of 0.6+ yield scores \(>0.86\).
Consistency Across Jurisdictions
Replication across contexts provides stronger evidence. Scored by number of independent jurisdiction studies:
Where \(N_j\) is the number of jurisdictions with concordant effect direction and \(N_{\text{sig}} = 10\) is the saturation parameter.
Parameter justification: The threshold \(N_{\text{sig}} = 10\) reflects that replication across 10+ independent jurisdictions provides strong evidence against idiosyncratic local effects. This aligns with meta-analytic conventions where 10+ studies enable reliable heterogeneity estimation25. Sensitivity analysis shows PIS varies by ±12% when \(N_{\text{sig}}\) ranges from 7 to 15. Five concordant jurisdictions yield \(S_{\text{consistency}} \approx 0.39\); ten yield \(\approx 0.63\).
Temporality (Required)
Policy adoption must precede outcome change. This is binary (either satisfied or not):
Where \(\delta\) is the lag between policy implementation and outcome measurement. If temporality is violated, the overall CCS is zeroed regardless of other criteria.
Where \(r_{\text{dose}}\) is the correlation between policy intensity and outcome magnitude, and \(r_{\text{sig}} = 0.5\) is the saturation parameter.
Parameter justification: The threshold \(r_{\text{sig}} = 0.5\) reflects that a correlation of 0.5 between policy intensity and outcome represents moderate dose-response evidence. This is analogous to toxicological dose-response standards where monotonic relationships strengthen causal inference26. Sensitivity analysis shows PIS varies by ±8% when \(r_{\text{sig}}\) ranges from 0.3 to 0.7. A dose-response correlation of 0.5 yields \(S_{\text{gradient}} = 0.5\); correlation of 0.7 yields \(\approx 0.66\).
Binary policies: For binary (yes/no) policies, dose-response cannot be assessed. Rather than defaulting to a neutral score of 0.5, binary policies are marked as “N/A” for gradient and this criterion is excluded from the CCS calculation (weights are renormalized across remaining criteria). This prevents binary policies from being systematically penalized relative to continuous policies.
Experiment Quality
Quality of the quasi-experimental design, weighted by validity diagnostic violations:
Where \(w_{\text{method}}\) is the base method weight and \(v_{\text{violations}} \in [0, 1]\) is the proportion of validity checks failed (parallel trends, pre-treatment fit, placebo tests).
Plausibility (Mechanistic)
Economic or behavioral mechanism linking policy to outcome. Scored by expert-validated mechanism database:
Where \(m_i \in \{0, 1\}\) indicates whether mechanism component \(i\) is satisfied and \(w_i\) are component weights.
Mechanism component checklist:
Component
Weight
Assessment Criterion
Economic theory predicts direction
0.30
Peer-reviewed theory paper supports predicted sign
Behavioral response documented
0.25
Empirical evidence of behavioral change in response to similar policies
No implausible required assumptions
0.20
Mechanism doesn’t require assumptions contradicted by evidence
Timing consistent with mechanism
0.15
Effect onset matches expected mechanism timeline
Magnitude plausible
0.10
Effect size within range predicted by mechanism
Scoring procedure: Each component is scored binary (0 or 1) by literature review. The weighted sum yields \(S_{\text{plausibility}} \in [0, 1]\). When expert-validated mechanism assessments are unavailable, this score defaults to 0.5 with a note that mechanism plausibility is unassessed.
Coherence with Literature
Consistency with broader economic and social science evidence:
Where \(N_{\text{studies}}\) is the count of supporting studies in the literature and \(N_{\text{sig}} = 5\). Three supporting studies yield \(S_{\text{coherence}} \approx 0.45\); ten yield \(\approx 0.86\).
Specificity
Whether the policy affects specific outcomes rather than everything:
Where \(N_{\text{outcomes}}\) is the number of outcome categories with significant effects. A policy affecting 1-2 outcomes has \(S_{\text{specificity}} > 0.7\); a policy affecting 10+ outcomes has \(S_{\text{specificity}} < 0.3\). Lower specificity suggests confounding or measurement artifact.
Causal Confidence Score (CCS) Calculation
The aggregate CCS combines the eight non-temporality criteria with explicit weights, gated by temporality:
\(S_{\text{temporality}}\) acts as a binary gate: if temporality fails (policy doesn’t precede outcome), the entire CCS is zero regardless of other criteria scores.
Proposed default criterion weights:
These weights reflect the relative importance of each criterion for causal inference in policy contexts. They can be adjusted based on domain expertise, sensitivity analysis, or implementation experience. The weights are adapted from the epidemiological Bradford Hill framework.
Criterion
Weight
Role
Temporality
Gate
Binary prerequisite (must be 1.0 to proceed)
Experiment
0.225
Method quality is primary for causal inference
Consistency
0.19
Replication across jurisdictions crucial
Strength
0.15
Effect magnitude matters for welfare
Gradient
0.125
Dose-response is strong causal evidence
Coherence
0.10
Literature support adds confidence
Plausibility
0.09
Mechanism existence supports causation
Specificity
0.06
Targeted effects more credible
Analogy
0.06
Transfer learning from similar policies
Weights for the eight scored criteria sum to 1.0. Temporality is not weighted because it is a binary gate, not a continuous score.
Jurisdiction Policy Inventory
Tracking Current Policies by Jurisdiction
Before generating recommendations, OPG must know what policies each jurisdiction currently has. The jurisdiction_policies table tracks:
Field
Description
Example
has_policy
Whether jurisdiction has this policy type
TRUE/FALSE
policy_strength
For continuous policies, the current level
$1.41/pack (tobacco tax)
implementation_date
When current policy took effect
2009-01-01
policy_details_json
Structured details about implementation
{“primary_enforcement”: false}
data_source
Where this information came from
“Texas Tax Code §154.021”
last_verified
When this was last confirmed accurate
2024-06-15
Data Sources for Policy Status
Jurisdiction Level
Primary Sources
Update Frequency
Country
WTO, OECD, IMF policy databases
Annual
US State
NCSL, state legislative databases, LexisNexis
Continuous
EU Member
EUR-Lex, national legal databases
Continuous
US City/County
Municipal code databases, Municode
Varies
Other Subnational
National statistics offices, academic datasets
Varies
Handling Missing Data
Data completeness varies by jurisdiction and policy type:
Data Quality Score
Interpretation
Recommendation Confidence
> 0.9
Comprehensive inventory
Full confidence
0.7 - 0.9
Most major policies tracked
High confidence
0.5 - 0.7
Significant gaps
Medium confidence; flag gaps
< 0.5
Sparse data
Low confidence; prioritize data collection
Recommendations are only generated when policy status is known with reasonable confidence.
Policy Gap Analysis
Comparing Current to Optimal
For each jurisdiction \(j\), the policy gap for policy type \(p\) is:
\(M_{jp}\) = Monetized annual welfare impact, adjusted for jurisdiction \(j\)’s population and context
Priority tiers:
Tier
Priority Score
Interpretation
Critical
\(\geq 0.80\)
Immediate action recommended
High
\([0.50, 0.80)\)
Strong candidate for adoption
Medium
\([0.25, 0.50)\)
Consider if political capital available
Low
\(< 0.25\)
Monitor for better evidence
High-priority recommendations have: 1. Large gap between current and optimal 2. Strong evidence (Grade A or B; high PIS) 3. Large expected welfare impact (high M)
Context Adjustment
Effect estimates are adjusted for jurisdiction characteristics:
This reflects increased uncertainty when extrapolating beyond the observed evidence distribution.
Recommendation Generation
Recommendation Types
Type
Question
When to Use
Example
Enact
“Add this?”
New policy the jurisdiction doesn’t have
“ENACT primary seat belt law”
Replace
“Change this?”
Modify existing policy level or approach
“REPLACE tobacco tax: $1.41 → $2.50”
Repeal
“Remove this?”
Remove policy with negative evidence
“REPEAL [harmful policy]”
Maintain
“Keep this?”
Current policy is evidence-supported
“MAINTAIN DUI threshold at 0.08 BAC”
For continuous policies (taxes, spending levels), Replace specifies the change from current to optimal level. Enact is reserved for truly new policies that don’t exist in the jurisdiction.
Blocking Factors
Recommendations flag constraints that may impede adoption:
Blocking Factor
Severity
Description
Example
Constitutional Constraint
Hard
Requires constitutional amendment
Takings Clause limits on land use regulations
Federal Preemption
Hard
Federal law prevents state/local action
Federal minimum wage floor
Treaty Obligation
Hard
International agreement constrains policy
WTO rules on tariffs
Autonomy Concern
Soft
Restricts individual freedom/choice
Mandatory helmet laws
Political Feasibility
Soft
Strong organized opposition
Industry lobbying
Implementation Cost
Soft
High fixed costs to implement
New regulatory agency needed
Design rationale: Why blocking factors are metadata only
OPG produces evidence-based rankings, not political forecasts. Blocking factors are flagged but do not affect algorithmic priority scores, for three reasons:
Political feasibility shifts over time. A policy “impossible” in 2020 may be mainstream by 2025. Filtering by current political feasibility would lock in the status quo and fail to surface the evidence-supported set.
Politicians know their context. An elected official in Texas understands local political dynamics better than any algorithm. OPG provides the evidence; filtering is left to policymaker judgment.
Autonomy tradeoffs require human judgment. A universal helmet law may save lives but restrict freedom. This is a value judgment, not an evidence question. OPG surfaces the health/income effects; the autonomy tradeoff is for democratic deliberation.
Hard vs. Soft blocking factors:
Hard blockers (constitutional, preemption, treaty): These represent legal impossibility at the current jurisdictional level. Recommendations with hard blockers are marked distinctly but still shown, as they may inform advocacy for constitutional change or higher-level policy.
Soft blockers (political, cost, autonomy): These represent practical difficulty, not impossibility. Many transformative policies faced “impossible” political opposition before adoption.
Important: The full evidence-supported recommendation set is always shown. Users can filter by blocking factor severity if desired, but the default view shows all recommendations ranked by expected welfare impact.
Similar Jurisdictions
For each recommendation, OPG identifies jurisdictions that: 1. Had similar characteristics to the target jurisdiction 2. Adopted the recommended policy 3. Experienced the predicted effects
This provides concrete examples for policymakers: “Vermont (similar demographics, adopted this in 2015, saw -7.1 pp smoking reduction).”
How to find good examples to copy: find places like you, who did the thing, and didn’t collapse. It’s like plagiarism, but encouraged.
Computing Jurisdiction Similarity
Similarity between jurisdictions \(j_1\) and \(j_2\) is computed as a weighted sum across three dimensions:
Where \(z\) values are z-scores and the denominator normalizes to [0,1] (4 SD maximum difference).
Institutional Similarity (\(\text{sim}_I\)):
Feature
Comparison
Federal vs. unitary
Binary match (1.0 if same, 0.5 if different)
Legal tradition
Common law, civil law, mixed (1.0/0.5/0.0)
Enforcement capacity
World Bank governance indicator proximity
Corruption level
Transparency International CPI proximity
Usage: Jurisdictions with \(\text{sim}(j_1, j_2) > 0.7\) are considered “similar” for evidence transfer purposes. Effect estimates from similar jurisdictions receive higher weight in context adjustment.
Recommended Tracking (for OPG Feedback)
Each recommendation includes minimal tracking guidance to enable continuous OPG improvement:
Field
Description
Example
Primary metric
The outcome variable to track
Traffic deaths per 100K
Data source
Where to get it
State vital statistics
Measurement frequency
How often
Annual
Comparison baseline
What to compare against
Pre-implementation 3-year average
This creates a learning loop: OPG recommends → jurisdiction implements → reports outcomes → OPG improves future recommendations.
OPG suggests thing, place does thing, place reports how it went, OPG learns. It’s a feedback loop, except it actually uses the feedback instead of filing it.
Optimal Jurisdictional Level for Policy Implementation
The Subsidiarity Principle for Evidence Generation
OPG recommends policies be implemented at the lowest jurisdictional level where the policy can be effective, for two reasons:
Maximize experimental data: 50 states experimenting > 1 federal policy. 3,000+ counties > 50 states. More jurisdictions = more natural experiments = faster evidence accumulation.
Federal level: little data, big risk. County level: lots of data, small risk. It’s safer to experiment in Shropshire than with the entire country.
Minimize harm from policy failures: A failed city ordinance affects thousands; a failed federal policy affects hundreds of millions. Lower-level experimentation bounds downside risk.
When Higher Levels Are Necessary
Some policies require higher jurisdictional levels:
Reason
Example
Recommendation
Externalities
Pollution crosses borders
State or federal
Race-to-bottom risk
Labor standards, tax competition
Federal floor, state variation above
Network effects
Infrastructure standards
Federal coordination
Economies of scale
Defense, diplomacy
National
Jurisdictional Level in Recommendations
For each policy recommendation, OPG specifies:
Field
Example
Minimum effective level
“City or higher”
Recommended level
“City (maximize data collection)”
Current adoption
“12 states, 47 cities have this”
Level constraints
“Federal preemption prevents city-level”
Policy Impact Score (Intermediate Metric)
Overview
The Policy Impact Score (PIS) is the intermediate metric used to generate recommendations. It quantifies the strength of evidence that a policy affects an outcome, combining effect magnitude, causal confidence, and analysis quality into a single score.
Jurisdiction-Level PIS Calculation
How to calculate if a policy works: add up how big the effect is, how sure we are, and how good the data is, for both money and health. Then argue about the number.
For each jurisdiction \(j\) and policy \(p\), compute PIS separately for each of the two metrics:
Where \(\sigma_{\text{income}}\) is the cross-jurisdictional SD of median income growth (typically ~1.5 pp/year) and \(\sigma_{\text{health}}\) is the cross-jurisdictional SD of healthy life expectancy (typically ~3-5 years).
I²: Percentage of variance due to heterogeneity (vs. sampling error)
\(I^2 < 25\%\): Low heterogeneity
\(25\% \leq I^2 < 75\%\): Moderate heterogeneity
\(I^2 \geq 75\%\): High heterogeneity (effects vary substantially across jurisdictions)
τ²: Estimated between-study variance
Q statistic: Cochran’s test for heterogeneity
High heterogeneity suggests moderators (policy effects vary by context) rather than a single true effect.
Evidence Grading
Evidence grades are assigned using explicit thresholds on PIS, heterogeneity (\(I^2\)), and jurisdiction count (\(N_j\)):
\[
\text{Grade} = \begin{cases}
A & \text{if } \text{PIS} \geq 0.80 \text{ AND } I^2 < 0.50 \text{ AND } N_j \geq 5 \\
B & \text{if } \text{PIS} \geq 0.60 \text{ AND } I^2 < 0.50 \text{ AND } N_j \geq 3 \\
C & \text{if } \text{PIS} \geq 0.40 \text{ AND } I^2 < 0.75 \text{ AND } N_j \geq 2 \\
D & \text{if } \text{PIS} \geq 0.20 \\
F & \text{otherwise}
\end{cases}
\]
Grade interpretation:
Grade
PIS Threshold
Heterogeneity
Jurisdictions
Interpretation
A
\(\geq 0.80\)
\(I^2 < 50\%\)
\(\geq 5\)
Strong evidence; ready for implementation
B
\(\geq 0.60\)
\(I^2 < 50\%\)
\(\geq 3\)
Good evidence; consider piloting
C
\(\geq 0.40\)
\(I^2 < 75\%\)
\(\geq 2\)
Suggestive evidence; suitable for targeted pilots
D
\(\geq 0.20\)
Any
Any
Weak evidence; exploratory only
F
\(< 0.20\)
Any
Any
Insufficient evidence
Threshold benchmarking methodology:
These thresholds can be benchmarked against historical policy adoption outcomes. The benchmarking procedure:
Historical validation: Apply OPG grading to policies adopted 10+ years ago with known outcomes
Target validation rates: Grade A recommendations should validate at 70%+ rate; Grade B at 50%+
Threshold adjustment: If observed validation rates differ from targets, adjust PIS and \(I^2\) thresholds
Heterogeneity threshold rationale:
The \(I^2 < 50\%\) threshold for Grades A and B follows Cochrane Collaboration guidance that heterogeneity above 50% indicates “substantial” variability across studies13. Grade C allows heterogeneity up to 75% (the “high” threshold) with explicit acknowledgment that effects are context-dependent. Above 75%, pooled estimates provide limited guidance for any specific jurisdiction.
Evidence grading decision rule (text summary): start with PIS threshold, then apply heterogeneity threshold (\(I^2\)), then jurisdiction count (\(N_j\)). The canonical Grade A/B threshold in this spec is \(I^2 < 50\%\).
Additional grade modifiers:
Conflicting evidence: Downgrade by 1 letter if direction of effect differs across high-quality studies
High-quality RCT: Automatic Grade A if RCT with low risk of bias, regardless of other criteria
Single jurisdiction: Maximum Grade C unless effect is extraordinarily large (\(|\hat{\beta}| > 1.0\) SD)
Context-Specific Confidence
Effects may vary by jurisdiction characteristics. We report confidence separately for:
Context
Description
Example Modifier
High-income countries
OECD members, GDP/capita > $30K
Tax policy effects
Low-income countries
GDP/capita < $5K
Different institutional capacity
Federal systems
Policy set at national level
vs. subnational variation
Subnational
States, provinces, cities
Local policy autonomy
Quality Requirements & Benchmarking
Minimum Thresholds for Inclusion
Criterion
Minimum
Rationale
Pre-treatment periods
4
Need to assess pre-trends
Post-treatment periods
2
Need to observe effect
Outcome observations
20
Statistical power
Control jurisdictions (for DiD)
5
Donor pool size
Pre-treatment RMSE (synthetic control)
< 2 SD
Acceptable pre-treatment fit
Parallel Trends Testing (DiD)
For difference-in-differences analyses, we test whether treated and control jurisdictions had parallel outcome trends before treatment:
Estimate event study with pre-treatment leads
Test joint significance of pre-treatment coefficients
If p < 0.10, flag as potential parallel trends violation
Report sensitivity: how different would trends need to be to explain away the effect?
Parallel trends test workflow: estimate event-study leads, run a joint significance test on pre-treatment leads, flag when \(p < 0.10\), then report sensitivity to plausible alternative trends.
Pre-Treatment Fit (Synthetic Control)
How to check if your fake control group is good enough: measure error, try fake treatments, reject if it’s rubbish. Quality control for imaginary things.
For synthetic control analyses:
Calculate RMSE of synthetic vs. actual treated unit pre-treatment
Compare to distribution of placebo RMSEs (treating each donor as “treated”)
If treated RMSE is in top 10% of placebo RMSEs, flag as poor fit
Report ratio of post-treatment effect to pre-treatment RMSE
Placebo and Robustness Tests
Test
Purpose
Implementation
In-time placebo
Does “treatment” show effect before it happened?
Assign fake treatment date before actual
In-space placebo
Do untreated units show similar effects?
Apply analysis to control jurisdictions
Leave-one-out
Is result driven by single jurisdiction?
Re-estimate dropping each jurisdiction
Bandwidth sensitivity
(For RDD) Is result robust to bandwidth choice?
Estimate with multiple bandwidths
Covariate adjustment
Does controlling for confounders change result?
Add covariates, compare estimates
Interpreting Recommendations
Priority Tiers
Tier
Criteria
Action
Quick Wins
High impact, low blocking factors, Grade A evidence
Immediate adoption recommended
Major Reforms
High impact, significant blocking factors
Requires political capital; strategic timing
Long-Term
Moderate impact, constitutional or treaty constraints
Requires structural change
Monitor
Moderate impact, Grade C/D evidence
Watch for better evidence
Political Feasibility Notes
While OPG does not filter by political feasibility, it provides context:
Organized opposition: Industries or groups likely to lobby against
Public opinion: Polling data on similar policies where available
Adjacent jurisdictions: Whether neighbors have adopted (diffusion effects)
Historical attempts: Previous failed attempts and why
Sequencing Guidance
Start with easy wins, build momentum, bundle things together, hit critical mass. It’s like a diet plan, but for governance and with better success rates.
Some policies are easier to adopt after others:
Quick wins first: Build political capital with easy, high-impact changes
Complementary bundles: Some policies work better together
Threshold effects: Some benefits only appear after critical mass of policies
Effect Size Benchmarks
Effect sizes are calibrated to cross-jurisdictional variation to aid interpretation:
Size
Income (pp/year)
Health (years)
Example
Small
< 0.05
< 0.1
Minor regulatory changes
Medium
0.05 - 0.15
0.1 - 0.3
Typical tax policy effects
Large
0.15 - 0.30
0.3 - 0.5
Major reform programs
Very Large
> 0.30
> 0.5
Transformative policies (rare)
Calibration basis: US states vary by ~1.5 pp/year in median income growth and ~3-5 years in healthy life expectancy. A “medium” effect represents ~10% of cross-state variation.
Confidence interval interpretation:
Narrow (< 25% of effect): Precise estimate; high confidence
High heterogeneity (I² > 75%) suggests context-dependence rather than universal effects.
Same policy, different places, different results. Turns out context matters. Who knew, apart from everyone who’s ever tried anything anywhere.
Jurisdiction-Specific Caveats
Caveat
Description
Mitigation
Data completeness
Policy inventory may be incomplete
Flag data quality; recommend verification
Context transfer
Effect in State A may not transfer to State B
Adjust for observable differences; widen CIs
Implementation variation
Same policy, different enforcement
Track implementation quality where possible
Interaction effects
Effect depends on other policies in place
Model policy bundles, not just single policies
Time-Varying Effects
Short-run vs. long-run: Immediate effects may differ from sustained effects
Policy drift: Implementation changes over time (amendment_notes tracking)
Adaptation: Jurisdictions and individuals adapt to policies
The event study design explicitly models dynamic effects; we report both immediate and sustained impact estimates.
Immediate effect, people adapt, effect drifts, long-run effect settles. Policies age like milk, not wine.
Publication Bias
Studies that find nothing don’t get published, so we think everything works. Funnel plots fish the failures out of the file drawer. Science learns to count its zeros.
The policy evaluation literature suffers from systematic publication bias:
Null effects underreported: Studies finding “no significant effect” are less likely to be published
Positive framing: Researchers may frame results to emphasize statistically significant findings
File drawer problem: Failed replications rarely published
Jurisdiction selection: Jurisdictions with cleaner natural experiments are overrepresented
Mitigation strategies:
Weight by inverse probability of publication (using funnel plot asymmetry tests)
Require pre-registration of analysis protocols before data access
Include unpublished working papers and government reports
Apply trim-and-fill or PET-PEESE corrections for funnel plot asymmetry
Report null findings prominently in the database
Operational Scope
OPG is built to rank policies by expected welfare impact, generate jurisdiction-specific recommendations, identify strong candidates for piloting, quantify heterogeneity, and flag likely harms. It relies on quasi-experimental designs with explicit diagnostics, robustness checks, and jurisdiction-level context adjustment.
Randomized trials, local implementation knowledge, and expert review complement OPG by deepening the evidence base where higher-resolution evidence is valuable.
Benchmarking Framework
The Critical Question
The decisive question is straightforward: Do jurisdictions that adopt high-priority OPG recommendations see better outcomes than those that don’t?
Addressing Adoption Bias
A naive retrospective comparison suffers from adoption bias: jurisdictions that voluntarily adopt policies may differ systematically from those that don’t. States adopting tobacco tax increases may already have anti-smoking momentum, overstating the causal effect of the tax itself.
Instrumental variable approach:
To address adoption bias, validation should exploit exogenous shocks to adoption:
Exogenous Shock
Example
Rationale
Court rulings
State court strikes down previous policy
Adoption forced by legal ruling, not political choice
Federal mandates
Clean Air Act state implementation
Compliance driven by federal law, not state preference
Close electoral outcomes
Ballot measure passes 51-49%
Near-randomization around threshold
Leadership turnover
New governor from different party
Adoption reflects leadership change, not underlying trends
These quasi-random adoption events provide cleaner tests of OPG predictions than voluntary adoption comparisons.
Retrospective Benchmark Study
Design: Retrospective prediction using instrumental variable identification.
Check if the system would have been right in the past: compute old data, identify policies, compare predictions to reality, grade yourself. It’s like marking your own homework, but honest.
Method:
Compute OPG recommendations for all jurisdictions using only data available before a cutoff date (e.g., 2015)
Identify exogenously-induced policy adoptions (court rulings, mandates, close votes) after the cutoff
Compare actual outcome changes in adopting jurisdictions to OPG predictions
Assess prediction accuracy and prioritization value
Success Metrics (strengthened from initial draft):
Metric
Definition
Target
Discrimination (AUC)
Does adopting recommendations predict “welfare improved”?
AUC > 0.70
Calibration
Correlation between predicted effect and actual effect
r > 0.5
Prioritization value
High-priority validation rate vs. low-priority rate
Ratio > 2:1
False positive rate
High-priority recommendations that harmed welfare
< 10%
Expected Outcomes:
If high-priority recommendations show validation rate of 60%+ and low-priority show rate < 30%, the system has practical utility
If no discrimination observed, the methodology needs recalibration or fundamental revision
Prospective Pre-Registration
To prevent hindsight bias, OPG should publish recommendations before adoption decisions are made:
Quarterly publication of jurisdiction-specific recommendations with timestamps
Public pre-commitment to methodology (no post-hoc adjustments)
Tracking of which recommendations were subsequently adopted
Comparison of pre-registered predictions to actual outcomes
This creates an auditable record that prevents retrofitting methodology to match observed outcomes.
Promise what you’ll measure before you measure it, then stick to the promise. Prevents ‘we meant to test that all along’ syndrome.
Benchmarking Questions
Context adjustment accuracy: Do jurisdiction-specific adjustments improve prediction?
Blocking factor impact: Are recommendations with blocking factors less likely to be adopted?
Evidence grade thresholds: Are the A-F grade cutoffs appropriately calibrated?
Heterogeneity interpretation: Does high I² actually indicate context-dependence vs. measurement noise?
Translation pipeline accuracy: Do surrogate→terminal metric conversions introduce systematic bias?
Continuous Improvement via Adoption Feedback
OPG improves through a learning loop:
OPG generates recommendation with expected effect ± uncertainty
Jurisdiction adopts policy at recommended level
Jurisdiction tracks primary metric per tracking guidance
Jurisdiction reports outcomes to OPG feedback system
OPG incorporates new data point into meta-analysis
Future recommendations reflect updated evidence
This transforms OPG from a static evidence aggregator into a self-improving system where every adoption strengthens the evidence base. The tracking guidance included with each recommendation standardizes what data jurisdictions should collect and report.
Recommend policy, place tries it, place reports results, analysis updates, better recommendations. It’s machine learning, but for government instead of cat pictures.
Adoption feedback improves the system but does not substitute for exogenous-shock benchmarking. Jurisdictions that implement OPG-recommended policies and report outcomes may differ systematically from those that do not, so instrumental-variable designs remain the cleanest performance test.
Future Directions
Benchmarking Priorities
Ways to check if predictions work, ranked by importance: retrospective studies, prospective trials, cross-validation, expert review. Trust in descending order.
Retrospective validation study (highest priority): Test OPG predictions against subsequent outcomes
Prospective prediction pre-registration: Publicly commit to recommendations before policy adoption decisions
Domain expert review: Have policy experts assess face validity of rankings
Cross-validation: Hold out jurisdictions, predict their outcomes from others
Data Infrastructure
Collect laws, teach computers to read them, standardize the results, give researchers access. It’s a library, but the books are alive and the librarian is an algorithm.
Automated policy tracking: NLP pipeline to detect policy changes from legislative databases
Outcome harmonization: Standardized outcome definitions across jurisdictions
API access: Enable researchers to query OPG data programmatically
Version control: Track how recommendations change as new data arrives
Integration with Decision-Making
Show data, model scenarios, get feedback, repeat. It’s a disciplined operating loop instead of vibes with better branding.
Policy dashboard: Real-time recommendations for policymakers
Uncertainty communication: Visualizations that convey confidence appropriately
Scenario modeling: “What if” analysis for proposed policies based on similar historical policies
Feedback mechanisms: Track whether recommendations were actually adopted and outcomes realized
Conclusion
The Optimal Policy Generator provides a systematic framework for translating policy-outcome evidence into jurisdiction-specific recommendations. By comparing each jurisdiction’s current policy inventory to the evidence-supported set, OPG produces actionable recommendations in four categories (enact/replace/repeal/maintain) ranked by expected welfare impact. The framework transforms scattered natural experimental evidence into actionable, jurisdiction-specific guidance.
Acknowledgments
[To be added: acknowledgments for seminar participants, reviewers, and colleagues who provided feedback.]
Appendix A: Worked Example - Texas Policy Recommendations
Warning SYNTHETIC DATA - NOT EMPIRICAL FINDINGS
All numbers in this appendix are fabricated for illustration. The effect sizes (+0.15 years, +0.02 pp/year, etc.), confidence intervals, and Bradford Hill scores are synthetic placeholders demonstrating the OPG framework’s output format. They were not derived from actual data analysis.
Do not cite these numbers as empirical findings. Actual policy effects would require jurisdiction-specific evidence analysis using real data from the sources described in this specification.
Overview
This worked example demonstrates the complete OPG output for a specific jurisdiction: Texas. It shows how generic policy evidence is translated into jurisdiction-specific recommendations.
Texas Policy Inventory (Sample)
Policy
Status
Income Effect
Health Effect
Recommendation
Grade
Primary seat belt
Missing
+0.02 pp/yr
+0.15 years
ENACT
A
Motorcycle helmet (all ages)
Partial
+0.01 pp/yr
+0.08 years
ENACT
A
Speed limit (85→70 mph)
Excessive
+0.01 pp/yr
+0.06 years
REPLACE
B
DUI threshold (0.08 BAC)
Optimal
N/A
N/A
MAINTAIN
A
Graduated licensing
Optimal
N/A
N/A
MAINTAIN
A
Step 1: Calculate Policy Impact Scores
Example: Primary Seat Belt Law
From meta-analysis of 47 US states (2000-2020):
Metric
Effect
SE
I²
Grade
Income (pp/yr)
+0.025
0.008
28%
A
Health (years)
+0.18
0.04
28%
A
Income effect derives from reduced healthcare costs and fewer disability-related productivity losses. Health effect converts mortality reduction to median healthy life years.
Bradford Hill Criteria Scores (applies to both metrics):
Criterion
Score
Rationale
Strength
0.75
Moderate standardized effects on both metrics
Consistency
0.82
I² = 28%, consistent across states
Temporality
0.95
Clear temporal ordering
Plausibility
0.90
Clear mechanism (increased compliance)
Experiment
0.85
Multiple synthetic control studies
CCS = 0.81 → Grade A
Step 2: Apply Context Adjustment for Texas
Factor
Texas Value
Adjustment
Current seat belt use
91.5%
Effect may be smaller (already high)
Rural driving proportion
High
Effect may be larger (more severe crashes)
Population
29.5M
Scale up total state-level impact
Adjusted expected effects for Texas:
Income: +0.02 pp/year (slightly smaller due to already-high compliance)
Health: +0.15 years
Step 3: Generate Recommendations
OPG Recommendations for Texas
ENACT (New Policies to Adopt)
1. Primary Seat Belt Enforcement Law (Current: secondary enforcement only)
Metric
Expected Effect
95% CI
Income
+0.02 pp/year
[+0.01, +0.03]
Health
+0.15 years
[+0.10, +0.20]
Evidence grade: A
Priority: High
Blocking factors: None identified
Similar jurisdictions: Florida adopted 2009; saw +0.18 years health effect
Blocking factors: Political (autonomy concerns from rider groups)
Similar jurisdictions: California (all ages since 1992)
REPLACE (Policies to Modify)
3. Maximum Speed Limit: 85 mph → 70 mph
Metric
Expected Effect
95% CI
Income
+0.01 pp/year
[+0.005, +0.02]
Health
+0.06 years
[+0.03, +0.09]
Current level: 85 mph (highest in US)
Recommended level: 70 mph
Evidence grade: B
Priority: Low
Blocking factors: Political (driver opposition), autonomy
REPEAL (Policies to Remove)
No high-priority repeal recommendations for Texas at this time.
(Example format: If Texas had a policy shown to cause net harm, it would appear here with expected welfare gain from removal.)
MAINTAIN (No Change Needed)
5. DUI Threshold at 0.08 BAC ✓
Current level: 0.08 BAC (national standard)
Evidence: Aligned with evidence-supported level
Status: Continue current policy
6. Graduated Driver Licensing Program ✓
Current level: Three-stage system with night/passenger restrictions
Evidence: Consistent with best practices
Status: Continue current policy
Step 4: Summary Dashboard
Total Expected Welfare Gain by Recommendation Type
Type
Recommendation
Income Effect
Health Effect
Grade
ENACT
Primary seat belt
+0.02 pp/yr
+0.15 years
A
ENACT
Universal helmet
+0.01 pp/yr
+0.08 years
A
REPLACE
Speed limit: 85→70 mph
+0.01 pp/yr
+0.06 years
B
MAINTAIN
DUI threshold, GDL
N/A
N/A
A
Total from changes
+0.04 pp/yr
+0.29 years
Note: MAINTAIN items confirm evidence alignment and require no action. REPEAL section empty for Texas; no harmful policies identified with strong evidence.
Interpretation
This example demonstrates how OPG transforms generic evidence into actionable, jurisdiction-specific recommendations using the two-metric framework:
If Texas adopted all recommendations, expected effects are:
+0.04 pp/year increase to median income growth
+0.29 years added to median healthy life expectancy
How policies affect money versus how they affect not dying. Ideally, both go up. Often, you have to pick one.
The two-metric format enables direct interpretation without complex conversion factors. When genuine tradeoffs exist (health gains with income costs, or vice versa), both effects are reported explicitly rather than hidden behind aggregated scores.
Appendix B: OPG Analysis Workflow
Complete OPG Pipeline
┌─────────────────────────────────────────────────────────────────┐
│ OPTIMAL POLICY GENERATOR WORKFLOW │
└─────────────────────────────────────────────────────────────────┘
Phase 1: DATA COLLECTION
─────────────────────────
1. Policy database ingestion
├── Parse legislative text
├── Record implementation dates by jurisdiction
└── Classify policy type and category
2. Jurisdiction policy inventory
├── Pull current policy status for each jurisdiction
├── Record policy strength (for continuous policies)
├── Flag data quality and gaps
└── Identify last verification date
3. Outcome data collection
├── Pull from primary sources (World Bank, WHO, etc.)
├── Harmonize units and definitions
├── Identify missing data patterns
└── Flag measurement quality issues
4. Confounder data collection
├── Economic indicators (GDP, unemployment)
├── Demographic variables (age structure, education)
├── Political variables (regime type, election cycles)
└── Geographic variables (neighbors' policies)
Phase 2: EVIDENCE ANALYSIS (Quasi-Experimental)
───────────────────────────────────────────────
5. Policy-outcome pair identification
├── Match policies to plausible outcome categories
├── Filter by minimum data requirements
└── Identify applicable quasi-experimental methods
6. Method selection
├── Synthetic control: single treated, good donors
├── Difference-in-differences: multiple treated, parallel trends
├── Regression discontinuity: sharp threshold exists
├── Event study: need dynamic effects
└── Interrupted time series: fallback
7. Effect estimation
├── Run primary analysis
├── Calculate standard errors (clustered)
├── Compute confidence intervals
└── Store jurisdiction-level results
8. Robustness checks
├── In-time placebo tests
├── In-space placebo tests
├── Leave-one-out sensitivity
└── Covariate adjustment sensitivity
Phase 3: AGGREGATION & PIS CALCULATION
──────────────────────────────────────
9. Meta-analysis
├── Pool jurisdiction estimates (random effects)
├── Calculate I², τ², Q statistics
├── Test for publication bias (funnel plot)
└── Apply trim-and-fill if needed
10. Bradford Hill scoring
├── Score each criterion (0-1)
├── Apply criterion weights
├── Compute CCS (causal confidence score)
└── Document evidence for each criterion
11. PIS calculation
├── Standardize effect estimate
├── Calculate quality adjustment
├── Compute final PIS
└── Assign evidence grade (A-F)
Phase 4: RECOMMENDATION GENERATION
──────────────────────────────────
12. Policy gap analysis (per jurisdiction)
├── Compare current inventory to evidence-supported
├── Calculate gap magnitude
├── Identify gap type (missing, harmful, suboptimal)
└── Flag blocking factors
13. Context adjustment
├── Adjust effect estimates for jurisdiction characteristics
├── Widen confidence intervals for context uncertainty
├── Identify similar jurisdictions for comparison
└── Note implementation considerations
14. Priority scoring
├── Rank by |Gap| × Evidence Grade × Impact
├── Assign to priority tiers (Quick Win, Major Reform, etc.)
├── Generate enact/replace/repeal/maintain lists
└── Calculate total expected welfare gain
Phase 5: OUTPUT GENERATION
──────────────────────────
15. Recommendation dashboard
├── Enact list (new policies to adopt)
├── Replace list (existing policies to modify: current → optimal)
├── Repeal list (harmful policies to remove)
├── Maintain list (policies aligned with evidence)
└── Jurisdictional level and tracking guidance for each
16. Two-metric reporting
├── Income effect (pp/year)
├── Health effect (years)
└── Combined welfare score (for ranking)
17. Documentation
├── Generate jurisdiction-specific reports
├── Create methodology audit trail
├── Version control all recommendations
└── Publish to API/dashboard
Minimum Data Requirements Checklist
Before generating recommendations, verify:
Appendix C: Glossary
Brief definitions for quick reference. See referenced sections for full details.
Bradford Hill criteria: Strength, Consistency, Temporality, Gradient, Experiment, Plausibility, Coherence, Analogy, Specificity. See Bradford Hill Criteria Scoring Functions.
Corresponding Author: Mike P. Sinn, Institute for Accelerated Medicine ([email protected])
Data Availability: This specification describes a methodological framework. Policy databases referenced (V-Dem, Polity V, CPDS, World Bank WDI) are publicly available at URLs provided in Data Sources section. A complete replication package including data extraction scripts, analysis code, and recommendation generation algorithms will be deposited in a public repository upon system deployment.
Quantifying the gap between current global governance and theoretical maximum welfare, estimating a 31-53% efficiency score and $101 trillion in annual opportunity costs.
Thousands of jurisdictions have made different policy and budget choices over decades, creating a natural experiment. Optimocracy applies causal inference to this cross-jurisdictional time-series data to identify which policies predict above-average median income and healthy life years. It then publishes evidence-based recommendations for every major vote, tracks politician alignment, and funds aligned candidates via SuperPAC, making suboptimal policy politically expensive while preserving democratic structures.
Smokers lose at least one decade of life expectancy compared with those who have never smoked. The probability of surviving from 25 to 79 years of age was about twice as great in those who had never smoked as in current smokers (70 percent vs. 38 percent among women and 61 percent vs. 26 percent among men). Cessation before age 40 reduces the risk of death associated with continued smoking by about 90 percent. Adults who quit at ages 25-34, 35-44, or 45-54 gained about 10, 9, and 6 years of life respectively compared with those who continued to smoke.
4.
Blincoe, L. J., Miller, T. R., Zaloshnja, E. & Lawrence, B. A. The Economic and Societal Impact of Motor Vehicle Crashes, 2010 (Revised). https://rosap.ntl.bts.gov/view/dot/78697 (2015)
In 2010, there were 32,999 people killed, 3.9 million injured, and 24 million vehicles damaged in motor vehicle crashes in the United States. The economic costs totaled \(277 billion. When quality of life valuations are considered, the total value of societal harm from motor vehicle crashes was\)871 billion, representing 1.9 percent of GDP.
5.
Buchanan, J. M. & Tullock, G. The Calculus of Consent: Logical Foundations of Constitutional Democracy. (University of Michigan Press, Ann Arbor, 1962).
6.
Olson, M. The Logic of Collective Action: Public Goods and the Theory of Groups. (Harvard University Press, Cambridge, MA, 1965).
7.
Stigler, G. J. The theory of economic regulation. Stigler2, 3–21 (1971)
As a rule, regulation is acquired by the industry and is designed and operated primarily for its benefit.
8.
Becker, G. S. A theory of competition among pressure groups for political influence. Becker98, 371–400 (1983)
Political equilibrium depends on the efficiency of each group in producing pressure, the effect of additional pressure on their influence, the number of persons in different groups, and the deadweight cost of taxes and subsidies.
9.
Cartwright, N. & Hardie, J. Evidence-Based Policy: A Practical Guide to Doing It Better. (Cartwright, Oxford, 2012).
The key to evidence-based policy is understanding that evidence of effectiveness elsewhere is not evidence of effectiveness here without support for the claim that the causal mechanism will operate in the new setting.
10.
Haskins, R. & Margolis, G. Show Me the Evidence: Obama’s Fight for Rigor and Results in Social Policy. (Haskins, Washington, DC, 2009).
The federal government spends hundreds of billions of dollars annually on social programs with little rigorous evidence of effectiveness.
11.
Hahn, R. W. Regulatory reform: What do the government’s numbers tell us? in 208–253 (2000).
A review of 48 regulatory impact analyses finds substantial variation in methodology and quality, with many failing to provide adequate justification for regulatory choices.
The WWC reviews existing research on different programs, products, practices, and policies in education to provide educators with the information they need to make evidence-based decisions.
Heterogeneity in systematic reviews refers to variability among studies. I² describes the percentage of variability in effect estimates that is due to heterogeneity rather than sampling error.
14.
Petticrew, M. & Roberts, H. Systematic Reviews in the Social Sciences: A Practical Guide. (Petticrew, Malden, MA, 2006).
Systematic reviews can help policymakers by providing a rigorous and transparent method for synthesizing research evidence on the effectiveness of social interventions.
15.
Sunstein, C. R. The Cost-Benefit State: The Future of Regulatory Protection. (Sunstein, Chicago, 2002).
Cost-benefit analysis, properly understood, is not only a useful tool but also an indispensable safeguard against both excessive and insufficient regulation.
16.
Viscusi, W. K. Pricing Lives: Guideposts for a Safer Society. (Viscusi, Princeton, NJ, 2018).
The value of a statistical life provides a consistent metric for evaluating the benefits of risk reduction policies across domains.
The primary engine driving improvement has been a focus on the quality of empirical research designs. Additional sources: https://www.aeaweb.org/articles?id=10.1257/jep.24.2.3
.
18.
Imbens, G. W. & Rubin, D. B. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. (Imbens, Cambridge, 2015).
The potential outcomes framework provides a rigorous foundation for defining causal effects and understanding the assumptions required for their identification.
The synthetic control method provides a systematic way to choose comparison units in comparative case studies. A combination of comparison units often provides a better comparison for the unit affected by the policy intervention than any single comparison unit alone.
20.
Abadie, A., Diamond, A. & Hainmueller, J. Comparative politics and the synthetic control method. Abadie59, 495–510 (2015)
The synthetic control method provides a systematic way to construct comparison units in comparative case studies, making explicit the weights assigned to each unit.
Synthetic control methods have become one of the most widely used tools for evaluating the effects of policy interventions in comparative case studies.
When treatment timing varies across units, standard two-way fixed effects estimators can be severely biased. We propose alternative estimators that are robust to treatment effect heterogeneity.
Original paper establishing the 9 criteria for evaluating causal relationships in epidemiology Criteria: Strength, Consistency, Specificity, Temporality, Biological Gradient, Plausibility, Coherence, Experiment, Analogy Published in Proceedings of the Royal Society of Medicine Most influential framework for assessing causation from observational data Additional sources: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1898525/ | https://en.wikipedia.org/wiki/Bradford_Hill_criteria
Classic reference for effect size conventions in social science research. Defines small (d=0.2), medium (d=0.5), and large (d=0.8) effect sizes for standardized mean differences.
25.
Borenstein, M., Hedges, L. V., Higgins, J. P. T. & Rothstein, H. R. Introduction to Meta-Analysis. (John Wiley; Sons, 2009). doi:10.1002/9780470743386
Comprehensive guide to meta-analysis methods. Establishes that 10+ studies are needed for reliable heterogeneity estimation and publication bias assessment.
26.
Rothman, K. J., Greenland, S. & Lash, T. L. Modern Epidemiology. (Lippincott Williams; Wilkins, 2008).
Standard epidemiology textbook covering causal inference, dose-response relationships, and study design. Provides framework for assessing biological gradients in exposure-outcome relationships.
A common approach to evaluating robustness to omitted variable bias is to observe coefficient movements after inclusion of controls. This is informative only if selection on observables is informative about selection on unobservables. Additional sources: https://www.tandfonline.com/doi/abs/10.1080/07350015.2016.1227711
