econml | Carlos Mendez

Causal Machine Learning and the Resource Curse with Python EconML

Thu, 07 May 2026 00:00:00 +0000

Overview

Can natural resource wealth be both a blessing and a curse? And can local institutions determine which way it goes? In this tutorial, we use EconML’s CausalForestDML to estimate heterogeneous causal effects of mining and mineral prices on economic development — and test whether institutional quality moderates those effects differently for mining versus price shocks.

We use simulated data with known ground-truth parameters so we can verify that the method recovers the correct answers. The simulated dataset mirrors the structure of Hodler, Lechner & Raschky (2023), who studied 3,800 Sub-Saharan African districts using a Modified Causal Forest. This tutorial focuses on the DML methodology: how the Double Machine Learning framework separates nuisance estimation from causal effect estimation to produce valid, efficient heterogeneous treatment effect estimates.

For the economic narrative and a companion implementation in Stata 19, see Causal Machine Learning and the Resource Curse with Stata 19.

Learning objectives

By the end of this tutorial, you will be able to:

Understand the Double Machine Learning (DML) framework and the residualization argument that makes it work
Distinguish heterogeneity features (X) from nuisance controls (W) in CausalForestDML
Configure CausalForestDML for discrete multi-valued treatments with panel data
Estimate Average Treatment Effects (ATEs) and Group Average Treatment Effects (GATEs), and read the Bootstrap-of-Little-Bags standard errors EconML reports
Interpret GATE patterns to identify which variables moderate treatment effects
Use EconML-specific tools like SingleTreeCateInterpreter for data-driven subgroup discovery
Evaluate estimated effects against known ground-truth parameters and explain any remaining gap

Key concepts at a glance

The post leans on a small vocabulary repeatedly. The rest of the tutorial assumes you can move between these terms quickly. Each concept below has three parts. The definition is always visible. The example and analogy sit behind clickable cards: open them when you need them, leave them collapsed for a quick scan. If a later section mentions “honest splitting” or “Neyman orthogonality” and the term feels slippery, this is the section to re-read.

1. Potential outcomes $Y_i(t)$. The outcome unit $i$ would take under treatment value $t$. Each unit has one potential outcome per treatment level. We observe only one of them: the one matching the treatment actually received. The rest are counterfactual. They live in worlds we never see.

Example

Take district 47 in 2008. Four potential NTL outcomes exist for it: $Y_{47,2008}(0)$, $Y_{47,2008}(1)$, $Y_{47,2008}(2)$, and $Y_{47,2008}(3)$. They correspond to no mining, low prices, medium prices, and high prices. Only one is in the dataset. It is the one matching whatever treatment that district-year actually had. The other three are forever invisible.

Analogy

Every life decision is a fork in the road. You took one fork. The parallel-universe versions of yourself took the other forks. Their lives are real conceptual objects. You just cannot directly observe them. Causal inference reconstructs those parallel universes. It does so by looking at people who did take the other forks.

2. CATE — Conditional Average Treatment Effect, $\tau(\mathbf{x})$. The average treatment effect for units with covariate profile $\mathbf{x}$. The CATE is a function of $\mathbf{x}$, not a single number. Where the CATE bends with $\mathbf{x}$, the treatment helps some units more than others.

Example

Take a well-governed district profile in our data: exec_constraints = 6, quality_of_govt = 0.7, and so on. For that $\mathbf{x}$ the CATE is $\tau(\mathbf{x}) \approx 0.26$. Mining lifts log-NTL by about 0.26 for that profile. Now move to the weakest-institutions case: exec_constraints = 1. The same function gives only $\tau(\mathbf{x}) \approx 0.18$. The CATE is what makes this comparison possible.

Analogy

A drug’s “average effect” might be a 5-point reduction in blood pressure. But a doctor cares about a specific patient. Maybe a 65-year-old male with diabetes. The CATE is that personalized effect. It takes a patient profile in. It returns the expected effect for someone like them.

3. GATE — Group Average Treatment Effect. The CATE averaged over a pre-specified subgroup. The subgroup is defined by some variable $Z$. GATEs test targeted moderation hypotheses. A typical question: “does institutional quality moderate the effect of mining?”

Example

Sort districts by exec_constraints (1–6). Average the per-observation CATEs inside each level. At level 1 we get $\widehat{\mathrm{GATE}} \approx 0.18$. The number climbs to $\approx 0.26$ at level 6. That climb is the moderation pattern Finding 3 reports. It is exactly what the GATE plots in this post visualize.

Analogy

A nationwide marketing campaign might lift sales by 5% on average. Before scaling it up, the company asks a simple question: did it work better in cities than in rural towns? The GATE answers exactly that. It reports the campaign’s effect inside each store type. It surfaces heterogeneity that the headline ATE hides.

4. ATE — Average Treatment Effect. The CATE averaged over the entire sample, $E[\tau(\mathbf{X})]$. The headline policy number. It answers a single question: if we turned the treatment on for everyone, what average effect would we see?

Example

Take our 3,000 district-years. The estimated ATE for the 1-vs-0 contrast (mining at low prices vs. no mining) is $\widehat{\mathrm{ATE}} = 0.240$. On average, mining-at-low-prices raises log-NTL by 0.24. In unlogged NTL, that is about a 27% bump.

Analogy

“This drug lowers cholesterol by 12 points on average.” That is an ATE statement. A single number, suitable for a press release. It says nothing about whether the drug works better in some patients than others. That question belongs to GATEs and CATEs.

5. Nuisance functions $g_0, m_0$. These are two conditional means: $g_0(\mathbf{x}, \mathbf{w}) = E[Y \mid \mathbf{X}, \mathbf{W}]$ and $m_0(\mathbf{x}, \mathbf{w}) = E[T \mid \mathbf{X}, \mathbf{W}]$. We call them nuisance because we do not care about their values. We estimate them for one reason only. That reason is to strip out the part of $Y$ and $T$ that is predictable from $(\mathbf{X}, \mathbf{W})$. What remains is the variation that identifies the causal effect.

Example

$\hat g_0$ is a Gradient Boosting regressor. It predicts a district’s log-NTL from elevation, ruggedness, ethnic fractionalization, country, year, and so on. It ignores mining status. $\hat m_0$ is a Gradient Boosting classifier. It predicts the probability of each treatment level from the same covariates. Both predictions matter only as inputs to the residualization step.

Analogy

Astronomers photograph faint galaxies in two steps. First, they take a “dark frame” with the lens cap on. The dark frame records sensor noise. Then they subtract it from the real exposure. Nobody hangs the dark frame on their wall. It exists only to be subtracted. $g_0$ and $m_0$ are dark frames for confounding. Their job is to be subtracted out. That is what lets the real causal signal show through.

6. Cross-fitting (sometimes “sample-splitting” or “out-of-fold prediction”). Estimate the nuisance functions on one fold of the data. Apply them to a held-out fold. Rotate so that every observation is residualized using nuisance models that did not see it. Without this rotation, in-sample residuals come out systematically too small. That bias propagates straight into the second stage.

Example

Setting cv=5 in CausalForestDML splits the 3,000 observations into five folds of 600. The forest fits $\hat g_0$ and $\hat m_0$ on folds 1–4. It then residualizes fold 5 using those fitted models. The procedure rotates four more times. The end result: each district-year is residualized by nuisance models trained on a strictly disjoint sample.

Analogy

Suppose you give a class the same problems for practice and for the final exam. Students who memorized the practice will ace the final. The score reflects memorization, not learning. Hiding the final-exam questions until grading time fixes the problem. Cross-fitting does the same trick. It hides each observation from the very nuisance model that will eventually residualize it.

7. Honest splitting (a property of an honest causal forest). A causal tree uses one random subsample to choose its split structure: which variable, which threshold. It uses a separate random subsample to estimate the treatment-effect value in each leaf. The split-chooser and the leaf-estimator never share data. This separation is what licenses valid confidence intervals from the forest.

Example

Consider a single tree inside the forest. With honest=True, half of its bootstrap sample picks the splits. Maybe the choice is “split first on distance_capital, then on exec_constraints”. The other half computes the average CATE in each resulting leaf. Those leaf-level numbers are unbiased. The reason: the splits were chosen without seeing them.

Analogy

A jury that hears the evidence should not also write the verdict template. If the same people pick the conclusion language and hear the case, the verdict reflects their pre-baked preferences. It would not reflect the evidence alone. Splitting the two roles is a basic guard against motivated reasoning. Honesty does the same job inside one tree. Split-choosers and leaf-estimators are different “people”. The leaf values cannot be tailored to the splits that produced them.

8. Neyman orthogonality. A property of the DML estimating equation $\psi(W; \tau, \eta)$. Here $\eta = (g_0, m_0)$ collects the nuisance functions. The property is $\left.\partial_\eta E[\psi]\right|_{\eta=\eta_0} = 0$. In words: at the truth, the expected estimating equation is flat in the nuisance functions. Small errors in $\hat g_0$ and $\hat m_0$ enter the second-stage estimator only at second order.

Example

Suppose $\hat g_0$ misses the true $g_0$ by 10% on average. A naive plug-in two-stage procedure inherits roughly that 10% error in the causal estimate. With Neyman orthogonality, the picture changes. The same 10% nuisance error contributes only on the order of $(0.10)^2 = 0.01$ to the causal estimate. That is one percentage point — orders of magnitude less than the input. This is why a Gradient Boosting first stage works. It converges at a slower-than-parametric rate. Even so, the second-stage estimate of $\tau$ remains $\sqrt{n}$-consistent and asymptotically normal.

Analogy

Picture a self-righting boat. You can lean over the rail. You can slosh the cargo. You can even slip on the deck. The hull pulls itself upright every time. Stability is built into its geometry, not into never being disturbed. Neyman orthogonality is the hull design. It lets DML stay upright when the nuisance estimates wobble.

The DML Causal Forest

Potential outcomes and the CATE

Causal inference rests on the potential-outcomes framework (Rubin, 1974; Imbens & Rubin, 2015). For each unit $i$ and each treatment value $t$, we imagine an outcome $Y_i(t)$ that would be realized if $i$ received treatment $t$. The catch is the fundamental problem of causal inference: only the potential outcome corresponding to the treatment unit $i$ actually receives is observable. All other potential outcomes for that unit are counterfactual — they live in a world we never see. Causal inference is therefore an exercise in imputation: using the observed outcomes of comparable units to stand in for the missing counterfactuals.

The Conditional Average Treatment Effect (CATE) for a unit with covariates $\mathbf{x}$ is

$$\tau(\mathbf{x}) = E\{Y_i(1) - Y_i(0) \mid \mathbf{X}_i = \mathbf{x}\}.$$

In words: among units who look like $\mathbf{x}$, what is the average gap between the treated and untreated potential outcomes? When the function $\tau(\cdot)$ is constant across $\mathbf{x}$, every type of unit responds the same way and a single ATE summarizes everything. When $\tau(\cdot)$ bends with $\mathbf{x}$, we have treatment effect heterogeneity — mining might raise nighttime lights in well-governed districts and barely move them elsewhere. Estimating that bend, not just its average, is the whole point of a causal forest.

The partially linear model with heterogeneous effects

EconML’s CausalForestDML works inside the partially linear model of Robinson (1988), extended by Chernozhukov et al. (2018) to allow heterogeneous effects:

$$Y_i = \tau(\mathbf{X}_i)\, T_i + g_0(\mathbf{X}_i, \mathbf{W}_i) + \varepsilon_i, \qquad E[\varepsilon_i \mid \mathbf{X}_i, \mathbf{W}_i] = 0.$$

$$T_i = m_0(\mathbf{X}_i, \mathbf{W}_i) + v_i, \qquad E[v_i \mid \mathbf{X}_i, \mathbf{W}_i] = 0.$$

The outcome equation says that $Y_i$ depends on the treatment $T_i$ multiplied by a unit-specific effect $\tau(\mathbf{X}_i)$, plus an arbitrary, possibly nonlinear function $g_0$ of the controls, plus mean-zero noise. The “partially linear” name comes from $T$ entering linearly (multiplied by $\tau$) while $g_0$ is allowed to be any flexible function.

The treatment equation writes $T_i$ as the conditional-mean treatment $m_0(\mathbf{X}_i, \mathbf{W}_i)$ plus a residual $v_i$. For a continuous treatment, $m_0$ is a regression. For our four-level treatment, $m_0$ is a multi-class classifier — specifically, a GradientBoostingClassifier — and “$T - m_0$” is shorthand for the residual of treatment around its conditional probabilities.

The functions $g_0$ and $m_0$ are called nuisance functions because we do not care about their values. We estimate them only to remove the part of $Y$ and $T$ that is predictable from $(\mathbf{X}, \mathbf{W})$, leaving behind the variation that identifies the causal effect.

Why two stages? The residualization argument

Subtract $E[Y_i \mid \mathbf{X}, \mathbf{W}] = \tau(\mathbf{X}_i) \, m_0(\mathbf{X}_i, \mathbf{W}_i) + g_0(\mathbf{X}_i, \mathbf{W}_i)$ from the outcome equation. The $g_0$ terms cancel, and after a line of algebra. Define the residualized outcome and treatment as

$$\tilde Y_i = Y_i - E[Y_i \mid \mathbf{X}, \mathbf{W}], \qquad \tilde T_i = T_i - m_0(\mathbf{X}_i, \mathbf{W}_i).$$

Plugging these residuals into the partial linear model yields:

$$\tilde Y_i = \tau(\mathbf{X}_i) \cdot \tilde T_i + \varepsilon_i.$$

So if we (a) estimate $g_0$ and $m_0$ in a first stage with any flexible learner, (b) residualize both $Y$ and $T$, and (c) regress $\tilde Y$ on $\tilde T$ with covariate-dependent slope, that slope at point $\mathbf{x}$ recovers $\tau(\mathbf{x})$. This is exactly the Frisch–Waugh–Lovell logic — if you have not seen FWL before, the tutorial on the Frisch–Waugh–Lovell theorem walks through the linear case in detail.

The causal forest is the second-stage learner that estimates this covariate-dependent slope from $(\tilde T, \tilde Y, \mathbf{X})$, splitting on $\mathbf{X}$ to find regions where the local slope is approximately constant.

Neyman orthogonality: why first-stage errors barely matter

Think of residualization like noise-canceling headphones: the first stage removes the “background noise” of confounders from both the outcome and the treatment, so the causal forest only hears the “signal” of the treatment effect.

The formal version of that intuition is Neyman orthogonality. The DML estimating equation $\psi(W; \tau, \eta)$ — where $\eta = (g_0, m_0)$ collects the nuisance functions — satisfies

$$\left.\frac{\partial}{\partial \eta} E[\psi(W; \tau, \eta)] \right|_{\eta = \eta_0} = 0.$$

In words: at the truth, the expected estimating equation is flat in the nuisance functions. Small errors in $\hat g_0$ and $\hat m_0$ enter the second-stage estimator only through second-order terms. The practical consequence is striking: even if Gradient Boosting estimates $g_0$ and $m_0$ at the slow rate $O(n^{-1/4})$, much slower than the parametric $\sqrt{n}$ rate, the resulting estimate of $\tau$ is still $\sqrt{n}$-consistent and asymptotically normal (Chernozhukov et al., 2018, §2.2). A naive plug-in two-stage procedure — one that does not use the orthogonal moment — inherits the slower nuisance rate and loses valid inference.

Three levels of effects

The causal forest produces per-observation CATE estimates, which aggregate to three levels with different uses:

Level	Notation	What it measures	When to report
CATE	$\tau(\mathbf{x})$	Effect for a unit with covariates $\mathbf{x}$	Exploratory: feed into a decision tree or partial-dependence plot to see how effects vary.
GATE	$E[\tau(\mathbf{X}) \mid Z = z]$	Average CATE in a pre-specified subgroup defined by a variable $Z$	Theory-driven: testing whether a named covariate (e.g., institutional quality) moderates the effect.
ATE	$E[\tau(\mathbf{X})]$	Overall average across all units	Policy: the headline number for “what happens on average if we turn the treatment on?”

DML pipeline

flowchart LR
A["<b>Panel Data</b><br/>3,000 obs"]:::data
B["<b>First Stage</b><br/>GBM nuisance<br/>models"]:::first
C["<b>Residualize</b><br/>Y - E[Y | X,W]<br/>T - E[T | X,W]"]:::resid
D["<b>Causal Forest</b><br/>500 honest trees"]:::forest
E["<b>CATEs</b><br/>Per-observation<br/>effects"]:::cate
A --> B --> C --> D --> E
classDef data fill:#6a9bcc,stroke:#141413,color:#fff
classDef first fill:#d97757,stroke:#141413,color:#fff
classDef resid fill:#00d4c8,stroke:#141413,color:#141413
classDef forest fill:#141413,stroke:#d97757,color:#fff
classDef cate fill:#6a9bcc,stroke:#141413,color:#fff

Setup and configuration

We use CausalForestDML from EconML with Gradient Boosting nuisance models. The ground-truth parameters are defined inline so the tutorial is fully self-contained.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from econml.dml import CausalForestDML
from sklearn.ensemble import (GradientBoostingRegressor,
GradientBoostingClassifier)
# Ground-truth ATEs from the data-generating process
TRUE_ATES = {
'1-0': 0.250, # Mining effect
'2-0': 0.300, # Mining + medium price
'3-0': 0.550, # Mining + high price
'2-1': 0.050, # Medium price premium (small)
'3-1': 0.300, # High price premium (large)
'3-2': 0.250, # High vs medium step
}

Load the simulated data

The dataset simulates 300 districts across 8 countries observed over 10 years (2003–2012), following the structure of Hodler, Lechner & Raschky (2023). Treatment has four levels: no mining (0), mining at low prices (1), medium prices (2), and high prices (3).

DATA_URL = ("https://github.com/cmg777/starter-academic-v501"
"/raw/master/content/post/python_EconML/sim_resource_curse.csv")
df = pd.read_csv(DATA_URL)
print(f"Dataset: {len(df):,} observations")
print(f"Districts: {df['district_id'].nunique()}, "
f"Countries: {df['country_id'].nunique()}")

Dataset: 3,000 observations
Districts: 300, Countries: 8

The dataset contains 3,000 district-year observations with a heavily imbalanced treatment: 85% of observations are untreated (no mining), while each of the three mining groups comprises only 5% of the data. This imbalance makes causal inference challenging — the causal forest must learn from relatively few treated observations.

Descriptive statistics

Treatment distribution

labels = {0: 'No mining', 1: 'Low prices',
2: 'Med prices', 3: 'High prices'}
for t, n in df['treatment'].value_counts().sort_index().items():
print(f" {t} ({labels[t]}): {n:,} ({n/len(df):.1%})")

 0 (No mining): 2,550 (85.0%)
1 (Low prices): 150 (5.0%)
2 (Med prices): 150 (5.0%)
3 (High prices): 150 (5.0%)

Treatment distribution across the four groups. The 85/5/5/5 imbalance makes causal inference challenging.

The 85/5/5/5 split means the causal forest has 2,550 control observations but only 150 per treatment level. For within-mining comparisons (e.g., 3-1), only 300 observations contribute, making standard errors larger for price-effect estimates.

Outcomes by treatment group

for t in sorted(df['treatment'].unique()):
mask = df['treatment'] == t
m_ntl = df.loc[mask, 'ntl_log'].mean()
m_conf = df.loc[mask, 'conflict'].mean()
print(f" {t} ({labels[t]}): NTL={m_ntl:.3f} Conflict={m_conf:.1%}")

 0 (No mining): NTL=-1.137 Conflict=10.7%
1 (Low prices): NTL=-1.028 Conflict=18.0%
2 (Med prices): NTL=-0.930 Conflict=18.0%
3 (High prices): NTL=-0.615 Conflict=28.0%

The raw means show a clear gradient: higher treatment levels are associated with higher NTL and higher conflict rates. But these raw comparisons are biased because mining districts differ systematically from non-mining districts in geography, institutions, and economic development.

Naive comparison: why we need causal ML

for comp in ['1-0', '2-1', '3-1']:
a, b = int(comp[0]), int(comp[2])
naive = df.loc[df['treatment']==a, 'ntl_log'].mean() - \
df.loc[df['treatment']==b, 'ntl_log'].mean()
truth = TRUE_ATES[comp]
print(f" {comp}: Naive={naive:.3f} Truth={truth:.3f} Bias={naive-truth:+.3f}")

 1-0: Naive=0.109 Truth=0.250 Bias=-0.141
2-1: Naive=0.098 Truth=0.050 Bias=+0.048
3-1: Naive=0.413 Truth=0.300 Bias=+0.113

The naive 1-0 estimate of 0.109 is severely biased downward from the true effect of 0.250 — a 56% underestimate. This happens because mining districts tend to have worse geographic and institutional characteristics that independently reduce development. The DML Causal Forest removes this selection bias by residualizing both the outcome and the treatment against observed confounders before estimating the causal effect.

EconML estimation

Configuration

We separate covariates into two groups with distinct roles in the DML framework:

X features (10 variables): Enter the causal forest and can drive treatment effect heterogeneity. These include exec_constraints, quality_of_govt, gdp_pc, elevation, temperature, ruggedness, distance_capital, agri_suitability, population, and ethnic_frac.
W controls (2 variables): Used only in the first-stage nuisance models (country_id, year). These absorb country and time fixed effects but do not enter the causal forest.

X_COLS = ['exec_constraints', 'quality_of_govt', 'gdp_pc',
'elevation', 'temperature', 'ruggedness',
'distance_capital', 'agri_suitability', 'population',
'ethnic_frac']
W_COLS = ['country_id', 'year']

Fitting the model

Y = df['ntl_log'].values
T = df['treatment'].values
X = df[X_COLS].values
W = df[W_COLS].values
est_ntl = CausalForestDML(
model_y=GradientBoostingRegressor(n_estimators=200, max_depth=4,
random_state=42),
model_t=GradientBoostingClassifier(n_estimators=200, max_depth=4,
random_state=42),
discrete_treatment=True,
categories=[0, 1, 2, 3],
n_estimators=500,
min_samples_leaf=10,
honest=True, # Separate split/estimation samples
inference=True, # BLB confidence intervals
cv=5, # 5-fold cross-fitting
n_jobs=1,
random_state=42,
)
est_ntl.fit(Y, T, X=X, W=W, groups=df['district_id'].values)

 NTL: fitted in ~25s

Several configuration choices deserve explanation.

Honest trees (honest=True) split the data inside each tree into two halves. One half is used to choose the splits — which variable, which threshold — and the other half is used to estimate the leaf means. A standard regression tree uses the same observations for both jobs, which lets the tree pick splits that artificially separate noisy observations and then quote the resulting separation back as if it were signal. The “exam writer / exam taker” analogy: honesty stops the tree from setting questions it has already memorized the answers to. Operationally, honesty is what licenses asymptotically valid confidence intervals — without it, the leaf estimates are tighter than they should be and inference=True’s reported standard errors would be misleadingly small. Wager & Athey (2018) formalize the result and prove $\sqrt{n}$-asymptotic normality for honest causal forests.

Cross-fitting (cv=5) addresses a different overfitting risk. When the same data are used to estimate the nuisance functions $\hat g_0, \hat m_0$ and to apply them as residualizers, in-sample residuals are too small on average and bias the second stage. Cross-fitting splits the data into 5 folds, fits the nuisance models on 4 of them, applies the fitted models to the held-out fold, and rotates. Each observation is residualized using nuisance estimates that did not see it.

GroupKFold via groups=district_id. Our panel observes each district across multiple years. Plain $K$-fold would scatter rows from the same district across folds, so the nuisance models would peek at most of a district’s rows when predicting one held-out year — leakage that artificially shrinks first-stage residuals. Passing groups=df['district_id'].values to fit() triggers GroupKFold, which keeps every district inside one fold.

A common confusion: GroupKFold is not the same as clustered standard errors. It blocks within-district leakage in cross-fitting; it does not adjust the second-stage variance for within-district correlation in the residuals. The standard errors EconML reports are forest-level Bootstrap-of-Little-Bags SEs that treat observations as independent. With panel data, true clustered SEs would typically be larger. We flag this as a limitation again in the Discussion section.

Identification: the Conditional Independence Assumption

The causal forest leans on the Conditional Independence Assumption (CIA), also called unconfoundedness or selection on observables: after conditioning on the observed covariates $(X, W)$, treatment assignment is as good as random, in the sense that

$$\{Y_i(0), Y_i(1), Y_i(2), Y_i(3)\} \perp T_i \mid (\mathbf{X}_i, \mathbf{W}_i).$$

In plain English: once we know a district’s geography, institutions, demographics, country, and year, knowing whether mining is active there tells us nothing more about what its potential nighttime-lights outcomes would be. Because we built the simulated data ourselves, the CIA holds by construction — every confounder we created is in $(X, W)$.

In real data, the CIA is untestable and easy to violate. Two concrete violation channels for this application:

Mineral surveys. Mining companies often arrive in a district because a geological survey flagged the geology as promising. The same survey may also predict future infrastructure investment unrelated to mining. If those surveys are not in $(X, W)$, both treatment and the potential outcome are correlated with an unobserved confounder.
Political connections. Districts whose elites are aligned with the central government may both attract mining concessions and receive non-mining infrastructure (roads, electrification). An analyst without a measure of political alignment would mis-attribute the infrastructure effect to mining.

Hodler, Lechner & Raschky (2023) defend the CIA in their setting by including a rich set of geological, geographic, and institutional controls; the methodology in this tutorial is no stronger than that defense.

Average Treatment Effects

EconML’s ate_inference() returns the average causal effect for a chosen pair of treatment levels, together with a standard error and a confidence interval.

The standard error here is the SE of the forest-level ATE point estimate, not the SE of any one unit’s CATE. It comes from the Bootstrap of Little Bags (BLB), a sub-bootstrap procedure (Athey, Tibshirani & Wager, 2019, §4) tailored to forests. Rather than refit hundreds of full forests — which would cost $O(B \cdot \text{forest})$ — BLB partitions the existing forest’s trees into “bags”, computes bag-level estimates, and uses the variance across bags as an estimate of the sampling variance of the full-forest ATE. The trick exploits the conditional independence of trees grown on different sub-samples; it returns valid asymptotic confidence intervals at a fraction of the cost of the obvious resampling scheme. EconML enables BLB whenever you pass inference=True to the constructor.

We report 90% intervals (alpha=0.1) by default — the convention used in Athey, Tibshirani & Wager (2019) and Hodler, Lechner & Raschky (2023). The substantive conclusions are unchanged at 95%, but the wider intervals make the price-effect comparisons (which have low power because only 150 observations per treatment level contribute) look more uncertain than the asymmetric pattern actually warrants.

We compute all six pairwise treatment contrasts:

comparisons = [
('1-0', 0, 1), ('2-0', 0, 2), ('3-0', 0, 3),
('2-1', 1, 2), ('3-1', 1, 3), ('3-2', 2, 3),
]
for comp_label, t0, t1 in comparisons:
res = est_ntl.ate_inference(X, T0=t0, T1=t1)
lo, hi = res.conf_int_mean(alpha=0.1)
print(f" {comp_label}: ATE={res.mean_point:.4f} "
f"SE={res.stderr_mean:.4f} 90%CI=[{lo:.3f}, {hi:.3f}]")

 1-0: ATE=0.2398 SE=0.0701 90%CI=[0.124, 0.355]
2-0: ATE=0.2684 SE=0.0791 90%CI=[0.138, 0.399]
3-0: ATE=0.4598 SE=0.0811 90%CI=[0.326, 0.593]
2-1: ATE=0.0286 SE=0.1008 90%CI=[-0.137, 0.194]
3-1: ATE=0.2200 SE=0.1013 90%CI=[0.053, 0.387]
3-2: ATE=0.1914 SE=0.1093 90%CI=[0.012, 0.371]

Finding 1: Mining raises economic activity, after controlling for confounding. All three mining-vs-no-mining contrasts (1-0, 2-0, 3-0) are positive, with point estimates well separated from zero relative to their standard errors. The basic mining effect 1-0 is 0.240 (SE = 0.070, 90% CI = [0.124, 0.355]) — comfortably above zero and within sampling error of the ground-truth 0.250. The naive difference-in-means for the same contrast was 0.109; the DML forest has eliminated nearly all of that confounding bias. Because the outcome is log nighttime lights, an effect of 0.24 corresponds to roughly a 27% increase in unlogged NTL ($e^{0.24} - 1 \approx 0.27$).

Finding 2: The price gradient is non-linear. Comparing medium prices to low prices (2-1) returns an ATE of 0.029 with an SE of 0.101 — the 90% interval [-0.137, 0.194] easily contains zero. Medium prices, in this DGP, add nothing detectable beyond the basic mining effect. The high-vs-low contrast (3-1), in contrast, is 0.220 (SE = 0.101) and significant at the 5% level, with a 90% interval that excludes zero. The high-vs-medium step (3-2) is 0.191 and significant at 10%. The forest has recovered the qualitative shape of the true price-response curve — flat at low-to-medium prices, jumping at high prices — without being told to look for a non-linearity. This is the kind of finding causal ML buys you: shape discovery without functional-form pre-specification.

Treatment effect heterogeneity (GATEs)

Computing GATEs from per-observation CATEs

EconML returns per-observation CATEs through effect_inference(). To form a GATE we average those CATEs within a chosen subgroup, and to form a standard error we propagate the per-observation BLB standard errors. Doing this by hand is more illuminating than a one-line API call — it makes the relationship between CATE-level heterogeneity and group-level effects visible.

def compute_gate(est, df, z_var, t0, t1):
inf = est.effect_inference(X, T0=t0, T1=t1)
ite, ite_se = inf.point_estimate, inf.stderr
for z in sorted(df[z_var].unique()):
mask = df[z_var].values == z
gate = ite[mask].mean()
# Propagate BLB standard errors (see derivation below)
gate_se = np.sqrt(np.mean(ite_se[mask]**2) / mask.sum())

For a subgroup $g$ of size $n_g$, the GATE estimator is the simple average of the per-observation CATE estimates,

$$\widehat{\mathrm{GATE}}_g = \frac{1}{n_g} \sum_{i \in g} \widehat\tau(\mathbf{X}_i).$$

If we treat the $\widehat\tau(\mathbf{X}_i)$ as approximately uncorrelated within the group — a working assumption, since EconML’s BLB does not return their full covariance matrix — the variance of their average is

$$\mathrm{Var}\left(\widehat{\mathrm{GATE}}_g\right) \approx \frac{1}{n_g^2} \sum_{i \in g} \mathrm{Var}\left(\widehat\tau(\mathbf{X}_i)\right) = \frac{1}{n_g} \cdot \overline{\mathrm{SE}_i^2}.$$

Taking the square root gives the formula in the code: sqrt(mean(se_i^2) / n_g). The CIs we report are point $\pm 1.645 \cdot \widehat{\mathrm{SE}}$ for a 90% level. Two caveats are worth flagging up front: (i) the within-group independence assumption probably understates the SE in panel data where the same district appears multiple times in the same group, and (ii) this SE captures estimation uncertainty in the CATE function only, not sampling variability of the subgroup composition. As with the ATE, the headline qualitative pattern survives at 95% intervals.

GATEs by Executive Constraints

The mining effect (1-0) should vary with institutional quality, while the price effect (3-1) should be flat:

GATEs for the mining effect (1-0) by executive constraints. The upward slope shows that stronger institutions amplify the economic benefits of mining.

GATEs for the price effect (3-1) by executive constraints. The flat pattern confirms that institutions do not moderate price effects.

 1-0 (Mining vs No Mining):
Exec. Constr. GATE 90% CI N
----------------------------------------------------
1 0.175 [0.168, 0.182] 300
2 0.255 [0.249, 0.262] 330
3 0.240 [0.236, 0.244] 720
4 0.242 [0.238, 0.246] 780
5 0.243 [0.237, 0.250] 420
6 0.264 [0.259, 0.269] 450
Range: 0.089
3-1 (High vs Low Prices):
Exec. Constr. GATE 90% CI N
----------------------------------------------------
1 0.242 [0.232, 0.252] 300
2 0.197 [0.187, 0.206] 330
3 0.217 [0.211, 0.224] 720
4 0.227 [0.221, 0.233] 780
5 0.224 [0.216, 0.231] 420
6 0.211 [0.204, 0.219] 450
Range: 0.045

Finding 3: Institutions moderate the mining margin but not the price margin. The mining-effect GATEs (1-0) span a range of 0.089 across executive-constraint levels, climbing roughly monotonically from 0.175 at the weakest institutions to 0.264 at the strongest. Read substantively: weaker institutions cut the development gain from mining roughly in half. The price-effect GATEs (3-1) span only 0.045 and show no monotone pattern — a non-finding that is itself the finding. The GATE plot effectively flat-lines because the price step is, by construction, uniform across institutional environments in the DGP.

This asymmetry — institutions shaping the mining-vs-no-mining margin but not the price margin — is the structural prediction of the institutions-and-resources literature (Mehlum, Moene & Torvik, 2006) and the empirical pattern Hodler, Lechner & Raschky (2023) document for Sub-Saharan African districts. A causal forest does not assume the asymmetry; it discovers it. That is the distinguishing payoff of letting the slope $\tau(\mathbf{x})$ be a flexible function rather than fixing it parametrically (e.g., a single $\tau \times \mathrm{exec\_constraints}$ interaction term).

GATEs by Quality of Government

The same pattern appears when we use a continuous institutional measure:

GATEs for the mining effect (1-0) by quality of government. The positive relationship cross-validates the executive constraints finding.

GATEs for the price effect (3-1) by quality of government. The flat pattern is consistent across institutional measures.

The mining effect (1-0) shows a positive relationship with quality of government, while the price effect (3-1) remains approximately flat across the institutional quality distribution. This cross-validates Finding 3 using a different institutional measure.

Variable importance

EconML reports feature_importances_ for the causal forest — the normalized contribution of each $X$-variable to treatment-effect heterogeneity across all splits in all trees:

importances = est_ntl.feature_importances_

 distance_capital 0.171
ethnic_frac 0.142
ruggedness 0.135
population 0.126
agri_suitability 0.120
elevation 0.120
temperature 0.120
gdp_pc 0.034
quality_of_govt 0.018
exec_constraints 0.014

Feature importance for treatment effect heterogeneity. Geographic variables dominate splitting frequency, but the GATE plots show that institutional variables are the true moderators in the DGP.

This ranking looks paradoxical: the GATE plots above just demonstrated that exec_constraints is what bends the mining effect, yet exec_constraints is dead last by importance. The resolution is that feature importance and moderation are different objects.

A variable $X_j$ is a moderator of the treatment effect if changing it changes the effect:

$$\frac{\partial \tau(\mathbf{x})}{\partial x_j} \neq 0.$$

A variable’s forest importance, by contrast, is the variance-reduction-weighted frequency with which it is selected as a split variable. The two diverge in a predictable way:

Continuous variables (e.g., distance_capital, ethnic_frac) admit many candidate split thresholds and tend to be picked frequently for fine-grained slicing, even when each individual split contributes only a tiny amount to actual heterogeneity.
Coarse discrete variables like exec_constraints (6 levels) have at most 5 candidate splits. Even when one of those splits captures the dominant moderation pattern, the variable accumulates a smaller total importance than a continuous neighbor that splits 50 times.

Read importances as a screening signal — a “where might heterogeneity be hiding?” first pass. Confirm or reject moderation with a hypothesis-driven GATE, a partial-dependence plot of $\tau(\mathbf{x})$, or the CATE Interpreter described next. The GATE analysis above is what nails the institutional-moderation finding; the importance ranking is what would have made you suspicious enough to draw the GATE plot in the first place.

CATE Interpreter

EconML’s SingleTreeCateInterpreter fits a shallow decision tree to the estimated CATEs themselves — the tree’s outcome is the model’s prediction $\widehat\tau(\mathbf{X}_i)$, not the original $Y_i$. By splitting on $\mathbf{X}$, the tree finds the covariates and thresholds that best separate units with different treatment effects, returning a small set of subgroups summarized by their average $\widehat\tau$. It is a summary of the forest’s heterogeneity surface, not a re-estimation of treatment effects.

from econml.cate_interpreter import SingleTreeCateInterpreter
intrp = SingleTreeCateInterpreter(max_depth=2, min_samples_leaf=100)
intrp.interpret(est_ntl, X)
intrp.plot(feature_names=X_COLS)

Depth-2 decision tree summarizing CATE heterogeneity for the mining effect (1-0). Each leaf reports the mean estimated CATE for the subgroup defined by the splits above it.

Two design choices control how interpretable the output is. Tree depth trades off detail against communicability: depth 2 produces at most four leaves and a story you can tell out loud; depth 4 or more reveals interaction structure but rarely fits in a paper figure. Minimum leaf size (min_samples_leaf=100) prevents the tree from carving out tiny, noisy subgroups whose CATE estimates are statistically unreliable. We pull both into the named module constants CATE_TREE_DEPTH and CATE_TREE_MIN_LEAF in script.py so the choice is one place to change rather than scattered magic numbers.

The CATE Interpreter is a complement to, not a substitute for, the GATE analysis. GATEs are hypothesis-driven: you pre-specify the moderating variable (here, exec_constraints) and test how the effect varies across its values. The CATE Interpreter is exploratory: it asks “of all the covariates, which ones — at which thresholds — best separate high-effect from low-effect units?” Running both is good practice. If the tree’s top split corresponds to a pre-specified moderator, your theory is reinforced; if the tree finds a different split, you have learned something the theory did not predict and have a candidate for follow-up GATE plots.

Discussion

Limitations

No clustered standard errors. Clustered SEs allow the residual variance to differ across clusters (here, districts) and absorb arbitrary within-cluster correlation. EconML’s inference=True reports forest-level Bootstrap-of-Little-Bags SEs that treat observations as independent. With panel data — the same district appearing in multiple years — the BLB SEs are likely too small. We use GroupKFold by district to prevent first-stage data leakage, but that is a different problem from second-stage variance estimation. The companion Stata tutorial uses Stata 19’s cate command, which supports vce(cluster district_id) directly.
Contemporaneous outcomes. Hodler, Lechner & Raschky (2023) use treatment at time $t$ and outcome at $t+1$, which rules out reverse causality from outcome to treatment within the same year. Our simulated data uses contemporaneous treatment and outcomes; in real applications, lagging the outcome is cheap insurance.
Simplified covariate set. The real analysis uses 60+ covariates spanning geology, geography, demography, institutions, and pre-treatment outcomes; we use 12. The simulated DGP guarantees that the CIA holds because we control for every confounder we built in. Real-world identification is only as strong as the controls support, and “we used a causal forest” does not relax the CIA.

Assumptions

The CATE estimates rely on the Conditional Independence Assumption: treatment is independent of potential outcomes given $(X, W)$. The CIA is untestable from data alone — it asserts something about the unobserved potential outcomes. In observational work, the standard defense is a combination of (i) institutional knowledge of the treatment-assignment process, (ii) a rich, theory-motivated set of covariates, and (iii) sensitivity analyses (e.g., Rosenbaum bounds, $E$-values) that ask how strong an unobserved confounder would have to be to overturn the conclusion. None of these is a substitute for randomization. In the simulated data here, we know the CIA holds because we built it that way.

Summary and next steps

EconML’s CausalForestDML recovered all three ground-truth findings. The ATE for the basic mining effect (1-0 = 0.240) is within sampling error of the true value 0.250 and removes nearly all of the 0.141 confounding bias visible in the naive estimator. Price effects come out non-linear (2-1 = 0.029, n.s.; 3-1 = 0.220, significant at 5%; 3-2 = 0.191, significant at 10%) without any pre-specified non-linearity. GATE patterns reveal that institutions moderate the mining effect (range = 0.089 across executive-constraint levels) but not the price effect (range = 0.045, no monotone pattern).
The DML two-stage residualization argument is what makes the causal forest valid in observational settings. Substituting the treatment equation into the outcome equation reduces causal estimation to a regression of $\tilde Y$ on $\tilde T$, where the residualizers $\hat g_0$ and $\hat m_0$ can be any flexible learner. Neyman orthogonality means errors in the residualizers enter only at second order, so $\sqrt n$-consistent estimates of $\tau$ are recoverable even with $O(n^{-1/4})$ first-stage rates.
Feature importance is a screening tool, not a moderation test. Continuous variables accumulate importance because they offer many split points, even when they do not bend the treatment effect. The GATE plot of $\tau$ against the suspected moderator is the right tool for confirming moderation; importance is the right tool for identifying candidates worth plotting.
The CATE Interpreter is the exploratory dual of GATEs. A shallow decision tree on the predicted CATEs surfaces data-driven subgroups, complementing the hypothesis-driven GATE analysis. Use both: GATEs test theory, the interpreter audits theory.

For the economic story behind these findings and a parallel implementation using Stata 19’s built-in cate command, see the companion tutorial: Causal Machine Learning and the Resource Curse with Stata 19.

Exercises

Replace the nuisance models. Swap GradientBoostingRegressor with RandomForestRegressor(n_estimators=200). Do the ATE and GATE estimates change? Why or why not (think about Neyman orthogonality)?
Vary the number of trees. Try n_estimators=100 vs n_estimators=1000. How do the standard errors and GATE patterns change?
Test the GroupKFold assumption. Remove groups=df['district_id'].values from the fit() call. What happens to the confidence intervals?
Discretize quality of government. Create quartiles of quality_of_govt and compute GATEs on the quartiles instead of raw values. Do the patterns become clearer?
Explore the CATE interpreter depth. Increase max_depth from 2 to 4 in SingleTreeCateInterpreter. Do the additional splits reveal meaningful subgroups or just noise?

References

Causal Machine Learning for Policy Evaluation: From ATE to IATE to a Better Assignment Rule

Fri, 01 May 2026 00:00:00 +0000

Overview

A government runs a job-training programme for unemployed jobseekers and wants to know three things at once. Does the programme actually cause people to spend more months in employment over the next two and a half years? Does the effect depend on who the jobseeker is — for example, on how well they speak the local language? And if effects differ across people, can we use those differences to send training to the right jobseekers, rather than to everyone or to no one? These three questions correspond to three causal estimands — the ATE, the GATE, and the IATE — and answering them is the bread-and-butter of Causal Machine Learning (CML).

CML combines two ideas. From causal inference, it borrows the careful framing of treatment effects under unconfoundedness and the doubly-robust scoring functions that protect against modelling mistakes. From machine learning, it borrows flexible nuisance estimators — random forests, gradient-boosted trees, neural nets — that learn complicated outcome surfaces without forcing the analyst to specify them by hand. The result is a small toolbox — DoubleML for the average effect, doubly-robust averaging for subgroup effects, causal forests for individual effects — that turns observational data into actionable, personalised policy recommendations. This tutorial walks through the full toolbox on a synthetic Flemish-ALMP-style cohort of 5,000 jobseekers, modelled on the empirical case study in Cockx, Lechner & Bollens (2023) and the methodological roadmap in Lechner (2023). Because the data are synthetic, the true treatment effects are known — so every estimator can be benchmarked against the truth.

The CML roadmap

CML organises a treatment-effect study into a sequence of progressively finer questions. The diagram below shows the four-step roadmap that this tutorial follows: estimate the average effect, then break it down into group effects, then go all the way to individual effects, and finally turn those individual effects into a policy.

graph LR
A["<b>1. ATE</b><br/>Population<br/>average effect"] --> B["<b>2. GATE</b><br/>Effect by<br/>subgroup"]
B --> C["<b>3. IATE</b><br/>Effect for<br/>each individual"]
C --> D["<b>4. Policy</b><br/>Welfare-optimal<br/>assignment rule"]
style A fill:#6a9bcc,stroke:#141413,color:#fff
style B fill:#d97757,stroke:#141413,color:#fff
style C fill:#00d4c8,stroke:#141413,color:#fff
style D fill:#999999,stroke:#141413,color:#fff

The arrows are not just decorative. Each step builds on the previous one: a credible average effect is the floor on which any subgroup analysis stands, and credible group effects are the floor on which any individual analysis stands. Skipping the first step and jumping straight to a fancy heterogeneity model is the most common mistake in applied CML. We will resist that temptation by starting from the simplest possible baseline and only adding complexity when the data warrant it.

Learning objectives:

Distinguish the three CML estimands — ATE, GATE, IATE — and write each as a formal expectation.
Diagnose covariate overlap and explain why selection-on-observables matters in observational data.
Estimate the population-average effect with DoubleMLIRM, using random-forest nuisances and 5-fold cross-fitting.
Estimate group effects via doubly-robust pseudo-outcomes and individual effects via CausalForestDML.
Translate the individual-level effect estimates into a welfare-maximising training-assignment rule and benchmark it against treat-all and an oracle.

Key concepts at a glance

The post leans on a small vocabulary repeatedly. The rest of the tutorial assumes you can move between these terms quickly. Each concept below has three parts. The definition is always visible. The example and analogy sit behind clickable cards: open them when you need them, leave them collapsed for a quick scan. If a later section mentions “IATE” or “welfare-maximising rule” and the term feels slippery, this is the section to re-read.

1. Potential outcomes $Y_i(d)$. The outcome unit $i$ would have under treatment value $d \in \{0, 1\}$. Each unit has two potential outcomes. We observe only one. The other is counterfactual. It belongs to a world we never see.

Example

For unemployed worker 7421 with D = 1 (received training), we observe Y = 22 months employed. Their counterfactual $Y_{7421}(0)$ — the months they would have worked without training — is forever invisible. Causal inference reconstructs it from comparable untrained workers.

Analogy

Every life decision is a fork in the road. You took one fork. The parallel-universe versions of you took the other. Their lives are real conceptual objects you cannot directly observe.

2. ATE — Average Treatment Effect, $E[Y(1) - Y(0)]$. The mean causal effect across everyone in the population. Headline policy number. It answers a single question: if we trained everyone, what would the average bump in employment be?

Example

The naive ATE is 5.111 months. The DoubleML estimate is 5.520. The simulation’s ground truth is 5.628. DoubleML closes 92% of the bias the naive estimator carries. The true ATE is the target; DoubleML is the engine.

Analogy

“This drug lowers cholesterol by 12 points on average.” Single number, suitable for a press release. Says nothing about who responds best.

3. GATE — Group Average Treatment Effect, $E[Y(1) - Y(0) \mid Z = z]$. The CATE averaged over a pre-specified subgroup defined by $Z$. GATEs surface heterogeneity along axes you name in advance.

Example

Sort workers by dutch_prof (1=lowest, 4=highest). The GATEs are 7.47, 6.13, 4.50, 2.91 months. Workers with the weakest Dutch benefit most. The training compensates for a labour-market handicap.

Analogy

A nationwide marketing campaign lifts sales 5% on average. Before scaling up, you ask: did it work better in cities than in rural towns? GATE answers exactly that.

4. IATE — Individual Average Treatment Effect, $\tau(\mathbf{x})$. The treatment effect as a function of the full covariate vector. One per unit. Estimated by Causal Forest DML in this post. The IATE is the input to a personalized assignment rule.

Example

The Causal Forest produces 4,000 IATEs, one per worker. Mean \hat\tau = 5.456 months. Mean absolute error against truth = 0.40 months. The IATEs feed Step 5’s welfare rule.

Analogy

A drug’s “average effect” is a 5-point reduction in blood pressure. But a doctor cares about a specific patient — maybe a 65-year-old male with diabetes. The IATE is that personalized effect.

5. Propensity score and overlap $\pi(\mathbf{x})$. The probability of treatment given covariates. Overlap requires that $\pi(\mathbf{x})$ is bounded away from 0 and 1 for the kinds of units we want to compare. Without overlap there is no counterfactual to estimate from.

Example

Step 1 of the tutorial plots $\hat\pi$ for treated and untreated workers. Densities overlap across most of the support but thin out at the tails. The overlap diagnostic is the first check before any DR estimator runs.

Analogy

Casino’s odds for the next card. We never see the casino’s algorithm directly; we estimate it from many deals. Overlap is the rule that the deck must contain enough cards of every relevant kind.

6. Cross-fitting (K-fold sample-splitting). Split the data into $K$ folds. Train nuisances on $K-1$ folds; predict on the held-out fold; rotate. The DoubleML library uses 5 folds by default and rotates internally.

Example

DoubleMLIRM in Step 3 runs 5-fold cross-fitting on random-forest nuisances. We never invoke train/test splits ourselves; the library wraps the rotation. The orthogonal score is computed on out-of-fold residuals.

Analogy

Two-pass exam grading. One TA writes the rubric, a different TA applies it. The separation is what makes the grade defensible.

7. Causal forest. A random forest adapted for causal estimation. Built honestly: one subsample chooses splits, a different subsample estimates leaf values. Each leaf approximates a local CATE. Aggregating across trees gives the IATE function.

Example

Step 5 uses CausalForestDML from EconML with 1,000 honest trees. The IATE function it returns is what powers Step 6’s assignment rule. Variable importance flags dutch_prof and prior_emp_months as the strongest moderators.

Analogy

A panel of judges, each on a slightly different jury. Each judge votes a verdict for the case in front of them. Average the verdicts to get the panel’s call. Honesty ensures no judge writes the rubric they then enforce.

8. Welfare-maximising assignment rule. A policy that treats units with $\hat\tau_i > 0$ and skips those with $\hat\tau_i \le 0$. Maximises predicted welfare given the IATE estimates. Benchmarked against treat-all and an oracle rule.

Example

Step 6 evaluates three rules on a held-out sample. Treating everyone yields 5.520 months/person. The IATE rule yields 1.749 — much lower because most workers have positive but small effects. The oracle (using true $\tau$) yields a similar number, suggesting the IATE rule is near-optimal under the simulation’s structure.

Analogy

Giving training tickets only to people who would actually use them. Treat-all sends tickets to everyone. The IATE rule keeps tickets for the responders. The oracle is the rule a perfect-information planner would use.

Setup and imports

Before running anything, install the two CML libraries this tutorial depends on. doubleml provides the cross-fitted, orthogonal-score machinery for averages; econml provides the causal forest for individual effects.

pip install doubleml econml # https://docs.doubleml.org https://econml.azurewebsites.net

The next block imports the stack and fixes the random seed. Setting np.random.seed(RANDOM_SEED) is not dead code: DoubleML’s internal cross-fit splitter uses the legacy global numpy RNG, so removing this line causes the ATE to drift by O(1e-3) across runs.

import warnings
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from doubleml import DoubleMLData, DoubleMLIRM
from econml.dml import CausalForestDML
# Silence only the predictable noise from the third-party CML stack;
# real deprecation / convergence warnings still surface.
warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings("ignore", category=UserWarning)
RANDOM_SEED = 42
np.random.seed(RANDOM_SEED)
X_COLS = ["age", "edu_years", "prior_emp_months", "dutch_prof", "female", "migrant"]

The two CSVs read in the next section (cml_data.csv for the observed columns and cml_truth.csv for the hidden ground truth) ship with this post’s page bundle. If you’re following along outside the bundle, you can regenerate them by running script.py once — it produces both files plus all six figures.

Data: a synthetic ALMP cohort

The dataset is a synthetic Flemish-ALMP-style cohort of 5,000 jobseekers. Each row records six pre-treatment covariates ($X$) — age, years of education, months employed in the look-back window, Dutch proficiency on a 0–3 scale, sex, and migrant status — a binary treatment indicator $D$ (whether the jobseeker received training), and an outcome $Y$ measuring months employed during a 30-month follow-up window. Because the data are synthetic, a companion file (cml_truth.csv) stores the true individual treatment effect $\tau_i$ for every row, which lets us benchmark each estimator. The reader does not need to know how the data were generated; only that the truth is known.

df = pd.read_csv("cml_data.csv")
truth = pd.read_csv("cml_truth.csv")
print(f"Sample size : {len(df):,}")
print(f"Treatment share P(D=1) : {df['D'].mean():.3f}")
print(f"Mean outcome E[Y] : {df['Y'].mean():.2f} months employed (out of 30)")
print(df.describe().round(2))

Sample size : 5,000
Treatment share P(D=1) : 0.528
Mean outcome E[Y] : 22.68 months employed (out of 30)
age edu_years prior_emp_months dutch_prof female migrant D Y
count 5000.00 5000.00 5000.00 5000.00 5000.00 5000.00 5000.00 5000.00
mean 39.82 12.02 16.99 1.33 0.49 0.30 0.53 22.68
std 11.54 2.95 9.59 1.02 0.50 0.46 0.50 4.18
min 20.02 6.00 0.37 0.00 0.00 0.00 0.00 9.81
25% 29.78 10.01 9.49 0.00 0.00 0.00 0.00 19.73
50% 39.68 11.94 15.80 1.00 0.00 0.00 1.00 22.81
75% 49.95 14.01 23.33 2.00 1.00 1.00 1.00 25.79
max 59.99 20.00 54.75 3.00 1.00 1.00 1.00 30.00

The cohort is 5,000 jobseekers aged 20–60 (mean 39.8) with about 12 years of education and 17 months of prior employment in the look-back window. The treatment share of 52.8% is high relative to a real-world ALMP study, but it is calibrated so that propensity scores stay safely inside [0.21, 0.81] and so that overlap is preserved across all four Dutch-proficiency strata. The outcome — months employed in the 30-month window — has a mean of 22.68 and a standard deviation of 4.18, leaving plenty of room for a realistic 5-to-8-month treatment effect to be visible without bumping into the floor of zero or the ceiling of thirty.

The script also stores the true parameters in true_parameters.csv for later benchmarking. The true ATE is 5.628 months, and the true GATEs decline monotonically with Dutch proficiency: 7.634 (no Dutch), 6.123 (low), 4.612 (intermediate), 3.130 (native). In words, jobseekers who do not speak Dutch benefit roughly 2.4× more from training than those who already do — a pattern that mirrors the policy-relevant punchline of the Cockx, Lechner & Bollens (2023) study.

Estimands: ATE, GATE, and IATE

Before estimating anything, we have to be precise about what we are estimating. Causal Machine Learning targets three estimands of increasing granularity. Throughout the post, $Y(1)$ denotes the potential outcome under treatment and $Y(0)$ the potential outcome without it. Only one of these is observed for each person; the other is the counterfactual that estimation tries to recover.

The Average Treatment Effect (ATE) is the mean effect of training across the entire population:

$$\text{ATE} = E[Y(1) - Y(0)]$$

In words, this says: average the per-person treatment effect over everyone in the population. In code, this is the quantity DoubleMLIRM returns in dml_irm.coef[0] after a single call to .fit().

The Group Average Treatment Effect (GATE) restricts the average to a subgroup defined by a categorical variable $Z$:

$$\text{GATE}(z) = E[Y(1) - Y(0) \mid Z = z]$$

In words, this says: average the per-person effect only over people who share the value $Z = z$. We use $Z$ = dutch_prof, so $z \in \{0, 1, 2, 3\}$. In code, the GATE is computed by averaging the doubly-robust pseudo-outcome (defined later) within each value of df["dutch_prof"].

The Individual Average Treatment Effect (IATE) goes one level deeper, conditioning on the full covariate vector $X$:

$$\text{IATE}(x) = E[Y(1) - Y(0) \mid X = x]$$

In words, this says: at every covariate profile $x$, predict the effect of training for somebody with that profile. In code, the IATE is the per-row prediction returned by cf.effect(X_arr) after fitting CausalForestDML.

The framing of this post is observational — we assume unconfoundedness: conditional on $X$, treatment assignment is as good as random. The naive difference-in-means is therefore genuinely biased on these data, not just imprecise. CML methods earn their keep by addressing that confounding through flexible nuisance estimators and orthogonal scores.

Step 1 — Overlap diagnostic

Causal estimation under unconfoundedness only works if every covariate profile has a non-trivial chance of being treated and a non-trivial chance of being untreated. Otherwise the model is forced to extrapolate, and small modelling mistakes blow up. The standard diagnostic is to fit a propensity score $\hat{\pi}(X) = \widehat{P}(D = 1 \mid X)$ and check that the histograms of $\hat{\pi}$ for treated and untreated jobseekers overlap. We use a logistic regression here purely for visualisation — DoubleML and CausalForestDML will fit their own nuisance models later.

ps_lr = LogisticRegression(max_iter=1000, random_state=RANDOM_SEED).fit(df[X_COLS], df["D"])
ps_hat = ps_lr.predict_proba(df[X_COLS])[:, 1]
print(f"Propensity range : [{ps_hat.min():.3f}, {ps_hat.max():.3f}]")
print(f"P(D=1 | X) mean (treated) : {ps_hat[df['D']==1].mean():.3f}")
print(f"P(D=1 | X) mean (untreat.): {ps_hat[df['D']==0].mean():.3f}")
fig, ax = plt.subplots(figsize=(8.5, 5))
bins = np.linspace(0, 1, 31)
ax.hist(ps_hat[df["D"] == 0], bins=bins, alpha=0.65, color="#6a9bcc",
label="Untreated (D=0)", edgecolor="white")
ax.hist(ps_hat[df["D"] == 1], bins=bins, alpha=0.65, color="#d97757",
label="Treated (D=1)", edgecolor="white")
ax.set_xlabel(r"Estimated propensity score $\hat{\pi}(X)$")
ax.set_ylabel("Number of individuals")
ax.set_title("Covariate overlap: propensity-score distribution by treatment status")
ax.legend()
plt.savefig("cml_overlap.png", dpi=300, bbox_inches="tight")
plt.show()

Propensity range : [0.208, 0.810]
P(D=1 | X) mean (treated) : 0.551
P(D=1 | X) mean (untreat.): 0.502

Estimated propensities fall safely inside [0.21, 0.81], so neither the strict positivity assumption nor the conventional [0.05, 0.95] trimming bounds bind. The treated mean propensity (0.551) sits only 0.049 above the untreated mean (0.502) — a small but real gap that confirms the data are mildly confounded rather than randomised. That is exactly the regime where doubly-robust methods are designed to outperform a naive baseline: confounding is real, but not so severe that any sensible adjustment will close the gap. Now that overlap is established, we can move on to the simplest possible estimator and watch it fail.

Step 2 — Naive baseline: difference-in-means

The simplest estimator of an average treatment effect is the difference of two sample means: average $Y$ for the treated, average $Y$ for the untreated, subtract. Under unconfoundedness with random assignment this would be unbiased; under unconfoundedness with observational data it generally is not. We compute it here precisely so we can see the bias.

y_treated = df.loc[df["D"] == 1, "Y"].mean()
y_untreated = df.loc[df["D"] == 0, "Y"].mean()
naive_ate = y_treated - y_untreated
n1, n0 = int((df["D"] == 1).sum()), int((df["D"] == 0).sum())
s1, s0 = df.loc[df["D"] == 1, "Y"].var(ddof=1), df.loc[df["D"] == 0, "Y"].var(ddof=1)
naive_se = float(np.sqrt(s1 / n1 + s0 / n0))
print(f"True ATE : 5.628")
print(f"Naive estimate : {naive_ate:.3f} "
f"[95% CI {naive_ate - 1.96 * naive_se:.3f}, {naive_ate + 1.96 * naive_se:.3f}]")
print(f"Bias : {naive_ate - 5.628:+.3f} months")

True ATE : 5.628
Naive estimate : 5.111 [95% CI 4.926, 5.296]
Bias : -0.517 months

The naive difference-in-means delivers 5.111 months with a Welch-style 95% confidence interval of [4.93, 5.30]. Because we know the truth, we can see that the estimator is biased downward by 0.52 months — about 9.2% of the truth — and that its 95% CI fails to cover the true ATE of 5.628. Why? Because in the synthetic DGP, caseworkers steer low-Dutch-proficiency jobseekers (those with the largest treatment effects) into training, and those same jobseekers also have shorter prior employment and weaker employability. Their outcomes are pulled down by everything the covariates capture, and a simple comparison cannot disentangle the programme’s effect from the selection effect. This is a textbook illustration of confounding: “the programme seems to work less well than it really does” can be an artefact of who got selected into it.

Step 3 — ATE via Double Machine Learning

DoubleMLIRM implements the Interactive Regression Model of Chernozhukov et al. (2018): a cross-fitted, doubly-robust estimator of the ATE under unconfoundedness. Cross-fitting — splitting the data into folds and predicting each fold using nuisance models trained on the other folds — prevents the random forests from overfitting to their own training sample and contaminating the score. A useful analogy is grading homework: imagine assessing each student using a rubric calibrated on other students' papers, never their own — that way the rubric cannot have been tailored to inflate any individual grade. The doubly-robust score is orthogonal to small mistakes in either nuisance, which is what gives the estimator its $\sqrt{n}$ rate even when the nuisances are themselves slow-converging machine-learning fits.

The Interactive Regression Model uses two nuisance functions: an outcome regression $g(d, X) = E[Y \mid D = d, X]$ and a propensity score $m(X) = P(D = 1 \mid X)$. The doubly-robust ATE score, evaluated at observation $i$, is

$$\psi_i = g_1(X_i) - g_0(X_i) + \frac{D_i \, \bigl(Y_i - g_1(X_i)\bigr)}{m(X_i)} - \frac{(1 - D_i) \, \bigl(Y_i - g_0(X_i)\bigr)}{1 - m(X_i)}.$$

In words, this says: start from the pure outcome-regression contrast $g_1 - g_0$, and then add a residual correction that weighs each observation by the inverse of its propensity. The clever bit is that $E[\psi_i] = \text{ATE}$ as long as either $g$ or $m$ is correctly specified — that is the “double” in doubly robust. In code, $g_0(X_i)$ and $g_1(X_i)$ correspond to dml_irm.predictions["ml_g0"] and ["ml_g1"], $m(X_i)$ to ["ml_m"], $D_i$ to df["D"], and $Y_i$ to df["Y"].

We fit DoubleML with random-forest nuisances and 5-fold cross-fitting, with trimming_threshold=0.01 to discard the (tiny) extreme tails of the propensity.

dml_data = DoubleMLData(df, y_col="Y", d_cols="D", x_cols=X_COLS)
ml_g = RandomForestRegressor(n_estimators=200, max_features="sqrt",
min_samples_leaf=5, random_state=RANDOM_SEED, n_jobs=-1)
ml_m = RandomForestClassifier(n_estimators=200, max_features="sqrt",
min_samples_leaf=5, random_state=RANDOM_SEED, n_jobs=-1)
dml_irm = DoubleMLIRM(
dml_data, ml_g=ml_g, ml_m=ml_m,
n_folds=5, score="ATE", trimming_threshold=0.01,
)
dml_irm.fit(store_predictions=True)
ate_dml = float(dml_irm.coef[0])
se_dml = float(dml_irm.se[0])
ci = dml_irm.confint(level=0.95).iloc[0]
ci_low, ci_high = float(ci.iloc[0]), float(ci.iloc[1])
print(f"True ATE : 5.628")
print(f"DoubleML ATE : {ate_dml:.3f} [95% CI {ci_low:.3f}, {ci_high:.3f}]")
print(f"95% CI covers truth : {bool(ci_low <= 5.628 <= ci_high)}")
print(f"Bias : {ate_dml - 5.628:+.3f} months")

True ATE : 5.628
DoubleML ATE : 5.520 [95% CI 5.361, 5.680]
95% CI covers truth : True
Bias : -0.108 months

Once the random-forest nuisances absorb the dependence of both treatment assignment and the outcome on the covariates, the residual bias collapses from 0.517 to 0.108 months — about a 79% reduction — and the 95% CI [5.36, 5.68] now covers the true ATE. In substantive terms, the corrected estimate raises the implied programme effect from “about 5.1 extra months of employment” to “about 5.5 extra months” out of a 30-month window. The standard error also drops from 0.094 (naive) to 0.081, so DoubleML is not just less biased but also slightly more precise — the cross-fitted nuisance models soak up outcome variance that the naive estimator leaves in the residual.

Step 4 — GATE by Dutch proficiency

The ATE answers “what is the average effect across the population?” — but a policymaker thinking about who to train wants the next layer down: “does the effect depend on who you are?”. The cleanest way to extract subgroup effects from a DoubleML fit is to compute the doubly-robust pseudo-outcome $\psi_i$ for every individual, and then average it within each subgroup. This is the same $\psi_i$ as in the equation above; the trick is that $E[\psi_i \mid Z_i = z] = \text{GATE}(z)$, so a simple group-mean of the pseudo-outcomes is an unbiased estimator of the GATE.

preds = dml_irm.predictions
g0 = np.asarray(preds["ml_g0"]).squeeze()
g1 = np.asarray(preds["ml_g1"]).squeeze()
m = np.asarray(preds["ml_m"]).squeeze()
y_arr, d_arr = df["Y"].values, df["D"].values
psi = (g1 - g0
+ d_arr * (y_arr - g1) / m
- (1 - d_arr) * (y_arr - g0) / (1 - m))
rows = []
for z in [0, 1, 2, 3]:
mask = (df["dutch_prof"] == z).values
psi_z = psi[mask]
est = psi_z.mean()
se = psi_z.std(ddof=1) / np.sqrt(mask.sum())
rows.append({"dutch_prof": z, "n": int(mask.sum()),
"gate_estimate": est, "std_error": se,
"ci_low": est - 1.96 * se, "ci_high": est + 1.96 * se})
gate_df = pd.DataFrame(rows)
print(gate_df.to_string(index=False, float_format=lambda v: f"{v:7.3f}"))

 dutch_prof n gate_estimate std_error ci_low ci_high
0 1302 7.465 0.157 7.157 7.772
1 1469 6.127 0.140 5.852 6.402
2 1504 4.503 0.142 4.225 4.781
3 725 2.910 0.214 2.490 3.329

Averaging the cross-fitted doubly-robust pseudo-outcomes within each Dutch-proficiency stratum recovers the monotone decline almost exactly: 7.47 / 6.13 / 4.50 / 2.91 estimated against 7.63 / 6.12 / 4.61 / 3.13 truth. Every estimate is within 0.22 months of its target, all four 95% confidence intervals cover their respective truths, and the ratio of the lowest-proficiency to highest-proficiency effect (≈ 2.6× under the estimates, 2.4× under the truths) lines up with the policy punchline of Cockx, Lechner & Bollens (2023): training delivers the biggest payoff to those who are furthest from the local-language labour market. As expected, standard errors widen for the smallest stratum (n = 725, SE 0.214) and tighten where data are densest (n = 1,504, SE 0.142). With clean group effects in hand, the natural next step is to push down to individual effects.

Step 5 — IATE via Causal Forest DML

The GATE collapses every jobseeker in a Dutch-proficiency stratum into a single number. But two people with the same dutch_prof value can still differ in age, education, prior employment, and migrant status, and the training programme might help them very differently. The Individual Average Treatment Effect $\tau(x) = E[Y(1) - Y(0) \mid X = x]$ asks for a separate prediction at every covariate profile, and the CausalForestDML estimator from EconML — a Python implementation of the generalized random forest framework of Athey, Tibshirani & Wager (2019) — is one of the canonical ways to produce one. Think of a causal forest as a regular random forest, except the trees split on heterogeneity in the treatment effect rather than on heterogeneity in the outcome — every leaf becomes a small neighbourhood within which the IATE is locally constant, and the forest averages many such trees together.

cf = CausalForestDML(
model_y=RandomForestRegressor(n_estimators=200, min_samples_leaf=5,
random_state=RANDOM_SEED, n_jobs=-1),
model_t=RandomForestClassifier(n_estimators=200, min_samples_leaf=5,
random_state=RANDOM_SEED, n_jobs=-1),
discrete_treatment=True,
n_estimators=400, min_samples_leaf=15, max_samples=0.5,
random_state=RANDOM_SEED, n_jobs=-1,
)
X_arr = df[X_COLS].values
cf.fit(df["Y"].values, df["D"].values, X=X_arr)
iate_hat = np.asarray(cf.effect(X_arr)).ravel()
iate_low, iate_high = cf.effect_interval(X_arr, alpha=0.05)
mae = float(np.abs(iate_hat - truth["tau"].values).mean())
corr = float(np.corrcoef(iate_hat, truth["tau"].values)[0, 1])
print(f"True ATE : 5.628")
print(f"Mean of estimated IATEs : {iate_hat.mean():.3f}")
print(f"MAE(IATE, truth) : {mae:.3f}")
print(f"Corr(IATE, truth) : {corr:.3f}")

True ATE : 5.628
Mean of estimated IATEs : 5.456
MAE(IATE, truth) : 0.397
Corr(IATE, truth) : 0.956

The forest produces 5,000 individual-level effect estimates whose Pearson correlation with the true individual effects is 0.956 and whose mean absolute error is just 0.40 months. The mean of the estimated IATEs (5.456) is within 0.17 months of the true ATE (5.628) — so the forest is not only ranking individuals correctly (the policy-relevant property) but also broadly calibrated in level. The 0.4-month MAE is small relative to the 4.5-month spread of true effects across individuals, which means an assignment rule built on these estimates can hope to identify which jobseekers benefit most from training, not just whether the average effect is positive.

To check that the forest also recovers the GATE-style heterogeneity at the individual level, we look at the histogram of estimated IATEs split by Dutch proficiency.

The four IATE distributions slide leftwards as Dutch proficiency rises — exactly the pattern the GATE bar chart showed at the group level — and their union centres on the true ATE. The forest is internally consistent with the GATE estimates, and the visible spread within each colour shows that there is meaningful heterogeneity even among jobseekers who share the same dutch_prof value.

Step 6 — Method comparison

We now have three estimators of the ATE and one ground truth. A forest plot puts them side by side and lets the reader judge bias and CI coverage at a glance.

comp = pd.DataFrame({
"method": ["Naive (DiM)", "DoubleML (IRM)",
"CausalForestDML (mean of IATEs)", "Truth"],
"estimate": [naive_ate, ate_dml, iate_hat.mean(), 5.628],
"ci_low": [4.926, 5.361, iate_hat.mean() - 1.96 * iate_hat.std(ddof=1) / np.sqrt(len(iate_hat)), 5.628],
"ci_high": [5.296, 5.680, iate_hat.mean() + 1.96 * iate_hat.std(ddof=1) / np.sqrt(len(iate_hat)), 5.628],
})
comp["bias"] = comp["estimate"] - 5.628
print(comp.to_string(index=False, float_format=lambda v: f"{v:7.3f}"))

 method estimate ci_low ci_high bias
Naive (DiM) 5.111 4.926 5.296 -0.517
DoubleML (IRM) 5.520 5.361 5.680 -0.108
CausalForestDML (mean of IATEs) 5.456 5.416 5.497 -0.172
Truth 5.628 5.628 5.628 0.000

The forest plot tells the story in a single panel. The naive interval [4.93, 5.30] sits entirely below the true ATE — visually obvious confounding bias. DoubleML’s [5.36, 5.68] straddles the truth and is the only interval among the three that delivers correct coverage. The CausalForestDML mean-of-IATEs interval [5.42, 5.50] is the tightest of the three — it pools 5,000 individual estimates so the average is precisely pinned — but it is in fact slightly too narrow, and its upper bound of 5.50 sits 0.13 months below truth. The reason is methodological: this CI captures sampling uncertainty in the average of individual predictions, not in the population ATE itself, so it does not pick up the small downward calibration bias of the forest as a whole. The practical takeaway is to prefer DoubleML when the question is “what is the ATE?” and reserve CausalForestDML for ranking and heterogeneity.

Step 7 — A welfare-maximising assignment rule

The whole reason to estimate individual treatment effects, rather than stop at the average, is that they enable personalised policy. Suppose training has a fixed cost equivalent to four months of employment per jobseeker. The welfare-optimal assignment rule is then trivial in principle: train person $i$ if and only if the true effect $\tau_i$ exceeds the cost. We don’t know the truth in practice, so the obvious surrogate is to plug in the IATE estimate $\hat{\tau}_i$ from the causal forest.

We benchmark four rules: treat no one, treat everyone, treat where $\hat{\tau}_i > 4$ (the IATE rule), and an oracle that has access to the true $\tau_i$. Welfare under any rule is computed as

$$W(\text{rule}) = E\bigl[\,\text{rule}(X) \cdot (\tau(X) - c)\,\bigr],$$

where $c = 4$ months is the cost of training. In words, for every person the rule treats, we add their true treatment effect minus the cost; the welfare of a rule is the average of those net contributions across the cohort.

COST = 4.0
assign_treat_none = np.zeros(len(df), dtype=int)
assign_treat_all = np.ones(len(df), dtype=int)
assign_iate_rule = (iate_hat > COST).astype(int)
assign_oracle = (truth["tau"].values > COST).astype(int)
def welfare(rule, tau_true, cost):
return float((rule * (tau_true - cost)).mean())
policy = pd.DataFrame({
"rule": ["Treat none", "Treat all",
"IATE rule (treat where iate_hat > cost)",
"Oracle (treat where true tau > cost)"],
"share_treated": [assign_treat_none.mean(), assign_treat_all.mean(),
assign_iate_rule.mean(), assign_oracle.mean()],
"avg_welfare": [welfare(assign_treat_none, truth["tau"].values, COST),
welfare(assign_treat_all, truth["tau"].values, COST),
welfare(assign_iate_rule, truth["tau"].values, COST),
welfare(assign_oracle, truth["tau"].values, COST)],
})
print(policy.to_string(index=False, float_format=lambda v: f"{v:7.3f}"))

 rule share_treated avg_welfare
Treat none 0.000 0.000
Treat all 1.000 1.628
IATE rule (treat where iate_hat > cost) 0.839 1.749
Oracle (treat where true tau > cost) 0.838 1.758

Once we have credible per-person effect estimates, the welfare comparison is striking. Holding training back from everyone yields zero net welfare. Treating everyone yields 1.63 months of net welfare per person — the ATE of 5.63 minus the cost of 4.0. Switching to a targeted rule that trains only individuals with estimated IATE above the 4-month cost threshold treats 83.9% of the cohort — almost identical to the 83.8% the oracle would treat — and lifts welfare to 1.749 months per person, recovering 99.5% of the oracle’s 1.758-month welfare and beating treat-all by 7.4%. The IATE rule’s small remaining gap (just 0.009 months per person) reflects the 0.4-month MAE in the individual estimates: the rule occasionally treats a person it shouldn’t and skips a person it should, but those errors net out to a tiny welfare loss because the misranked individuals are concentrated near the cost cutoff where the welfare slope is shallow.

Discussion

We started with three questions. Does training cause more months of employment? Yes — DoubleML estimates the ATE at 5.520 months [5.36, 5.68], and that 95% CI covers the true 5.628; the simpler naive comparison would have understated the effect by about half a month and produced a CI that misses the truth entirely. Does the effect depend on who the jobseeker is? Strongly yes — the GATE declines monotonically from 7.47 months for jobseekers with no Dutch to 2.91 months for native speakers, a 2.6× ratio that is a real policy signal, not noise. Can we use those differences to assign training better? Also yes — feeding the CausalForestDML’s IATE estimates into a simple “treat where $\hat{\tau}_i > c$” rule (with $c$ the per-jobseeker cost of training) captures 99.5% of the welfare an oracle would achieve and improves on treating everyone by 7.4%.

The methodological discipline behind these answers is what separates CML from a “throw a random forest at it” approach. DoubleML’s cross-fitting and orthogonal scoring give the ATE estimator a $\sqrt{n}$ rate even with slow-converging machine-learning nuisances; the doubly-robust pseudo-outcome lets us reuse those nuisances for an internally consistent GATE without re-fitting; and the causal forest produces individual-level estimates that respect the same identification logic. A practitioner thinking about a real ALMP would now have a defensible answer to the question that matters most: not just “should we run this programme?” but “for whom?”.

The case study also surfaces a subtle but important caveat about which tool to use for which question. The CausalForestDML mean-of-IATEs has the tightest 95% CI of any estimator in the comparison, but that interval is for the average of individual predictions, not for the population ATE. Its upper bound (5.50) does not cover the truth (5.628), and treating it as a competitor to the DoubleML interval would be a methodological mistake. DoubleML for the ATE; causal forest for ranking and heterogeneity — that is the operational division of labour the literature recommends and that this case study demonstrates concretely.

Limitations and next steps

The result is encouraging but rests on assumptions that are worth flagging carefully:

Synthetic data with easy overlap. Estimated propensities are bounded inside [0.21, 0.81] by construction, so neither the DoubleML trimming_threshold = 0.01 nor the doubly-robust pseudo-outcome’s division by $m$ and $1 - m$ is stressed on these data. In a real ALMP cohort, propensities can drift toward 0 or 1, the doubly-robust score becomes sensitive to small denominators, and trimming choices matter much more than they appear to here.
Unconfoundedness. Every causal claim assumes selection-on-observables: conditional on the six covariates, treatment assignment is as good as random. The synthetic DGP satisfies this by construction; in a real application this is the strong identifying assumption that justifies DoubleML and CausalForestDML over a naive comparison.
Treatment share. The cohort has 52.8% treated, which is higher than typical real-world ALMP studies. The synthetic DGP is calibrated to keep overlap comfortable in every stratum, so readers should not over-interpret the magnitude of effects.
Forest CI is not a substitute for the DoubleML CI. The CausalForestDML mean-of-IATEs interval misses the truth even though the forest is well-calibrated overall. Use it for heterogeneity, not for ATE inference.
Cost is fixed and known. The welfare comparison takes the four-month cost as given. In practice the cost of an ALMP intervention is itself uncertain and could vary across jobseekers (administrative cost, opportunity cost, displacement effects), and the optimal assignment rule should propagate that uncertainty.

Next steps to strengthen and extend the analysis:

Replace the single Dutch-proficiency-based GATE with policy trees (Athey & Wager, 2021), which learn the assignment rule directly from data rather than relying on a hand-picked stratification variable.
Compare CausalForestDML against the Modified Causal Forest (mcf) package used in Cockx, Lechner & Bollens (2023), which targets exactly this setting.
Stress-test overlap by drifting the propensity-score distribution toward 0 or 1 and re-running the full pipeline; observe how trimming choices and DR-score variance change.
Extend to multi-valued treatments (e.g., several training programmes) and use DoubleMLAPO to estimate the average potential outcome for each arm.
Run the doubly-robust pipeline on a real ALMP dataset with weaker overlap and check whether the policy-relevant punchline (lower Dutch → larger benefit) survives outside the synthetic DGP.

Takeaways

Naive difference-in-means is biased on observational data — visibly so. It estimates 5.111 months [4.93, 5.30] against a true ATE of 5.628, a 0.52-month downward bias whose 95% CI fails to cover the truth.
DoubleML closes 79% of the bias gap and delivers correct coverage. The IRM estimate of 5.520 [5.36, 5.68] both covers the true 5.628 and tightens the standard error from 0.094 (naive) to 0.081.
Effect heterogeneity by Dutch proficiency is real and policy-relevant. Estimated GATEs of 7.47 / 6.13 / 4.50 / 2.91 across levels 0–3 line up against truths 7.63 / 6.12 / 4.61 / 3.13, with all four 95% CIs covering their target.
CausalForestDML recovers the individual effect surface with 0.956 correlation and 0.40-month MAE — small relative to the 4.5-month spread of true effects across individuals.
A simple IATE-based assignment rule recovers 99.5% of oracle welfare (1.749 vs 1.758 months per person) and beats treat-all by 7.4% — the central practical reason to estimate individual effects in the first place.
CausalForestDML’s CI for the average of IATEs is not a substitute for DoubleML’s CI for the ATE. The forest interval [5.42, 5.50] misses truth despite the forest being well-calibrated overall — a methodological subtlety worth remembering.
Easy overlap in this synthetic DGP is a feature of the case study, not a property of CML. Real-world ALMP applications will encounter tighter propensity bounds, and trimming will matter much more than it appears to here.
Next step. Replace the hand-picked Dutch-proficiency stratification with a learned policy tree to maximise welfare directly; compare CausalForestDML to the mcf package on a real ALMP cohort.

Exercises

Change the cost. Re-run Step 7 with COST = 2.0 and COST = 6.0 months. How does the IATE rule’s share-treated change? At what cost does the rule converge to “treat all” or “treat none”, and does the welfare gap to the oracle widen or shrink?
Swap the nuisance learner. Re-fit DoubleMLIRM with LassoCV for ml_g and LogisticRegressionCV for ml_m. Does the ATE estimate change meaningfully? Does the 95% CI still cover the truth, and is the standard error smaller or larger than with random forests?
Stress-test heterogeneity. Compute the IATE separately for $X$ profiles that differ only in migrant (holding the other five covariates at their median values). Does the CausalForestDML predict a clear migrant effect, and is it consistent with the GATE pattern by Dutch proficiency?

References

Acknowledgements

AI tools (Claude Code, Gemini, NotebookLM) were used to make the contents of this post more accessible to students. Nevertheless, the content in this post may still have errors. Caution is needed when applying the contents of this post to true research projects.