Inference on predictions

[hhsievertsen]

Hi Team,

thanks for great tools, great documentation and great responses here. I have a setting where I am predicting CATES on a different sample where no one is treated to study how we would expect the treatment effect to be in that population. So basically I am doing

library(grf)
# Generate data
n <- 100
p <- 10
X1 <- matrix(rnorm(n * p), n, p) # original sample
X2 <- matrix(rnorm(n * p), n, p) # new sample where I don't observe W
W <- rbinom(n, 1, 0.5)
Y <- pmax(X1[, 1], 0) * W + X1[, 2] + pmin(X1[, 3], 0) + rnorm(n)
# Get forest
c.forest <- causal_forest(X1, Y, W, num.trees = 500)
# Predict on new sample
c.pred <- predict(c.forest, X2, estimate.variance = TRUE)
# Average CATE
mean=mean(c.pred$predictions)

Now, Ideally I would like to exploit the estimate SEs on the CATEs to test hypotheses on the out of sample predictions. I.e. whether the average CATE !=0. My first naive approach is to get the se for the mean using se=mean(c.pred$variance.estimates), assuming covariances are zero. Is that totally silly and what should I rather do, if anything?

Cheers, Hans

[erikcs]

Hi @hhsievertsen, I'll instead suggest an answer to a slightly modified question that you could ideally use instead: "given CATEs estimated on a training set X.train, do they do a good job predicting treatment effects on a test set X.test? If so, an estimate of RATE on X.test should be positive and significant". An intro to "RATE" is here

Note you need (X,Y,W) for all n=n.train+n.test for this approach, you could split your data accordingly (CF gives you an estimate of the ATE defined by some population X. If you have two draws X1 and X2 from this population, then both ATEs are estimates of the ATE defined by X, which is why you perhaps want to rephrase your question).

(Tagging @scottfleming in case you have any follow-up questions, as I will be gone for a few days)

[hhsievertsen]

Hi @erikcs, thanks for the quick reply. The RATE approach seems really useful!

The challenge is that I don't observe W in the second sample. Imagine that we have an RCT in two regions. One found an effect of W the other not.

I now get access to the microdata in the former and I use that to get the CF. On the second sample I also get the microdata, but not the information on who was treated. I then predicted the CATEs on the second sample using the forest fit on the former sample. And I find that the average CATE is much closer to zero in the second sample. In line with a hypothesis that the different conclusions is due to treatment effect heterogeneity. However, I would like to get some confidence intervals around the average predicted CATE in the second sample. How could I do that? I might have overlooked something in the RATE vignette, but I don't see how can apply that.

Cheers Hans

[swager]

The problem you're describing, @hhsievertsen, is a problem that's recently received some attention in the literature; see, e.g.,

Dahabreh, Issa J., Sarah E. Robertson, Jon A. Steingrimsson, Elizabeth A. Stuart, and Miguel A. Hernan. "Extending inferences from a randomized trial to a new target population." Statistics in medicine 39, no. 14 (2020): 1999-2014. Nie, Xinkun, Guido Imbens, and Stefan Wager. "Covariate Balancing Sensitivity Analysis for Extrapolating Randomized Trials across Locations." arXiv preprint arXiv:2112.04723 (2021).

Unfortunately, we don't currently have the methods described in these papers implemented in GRF. You could, however, use GRF to estimate the nuisance components for the doubly robust approach described in Dahabreh & al (and this is functionality we may add to GRF in the future).

[hhsievertsen]

Hi @swager thank you for the references. That looks very useful. I'll keep an eye on GRF updates and meanwhile think about alternative strategies. Cheers.

[erikcs]

Just adding these code references for future reference, the estimator looks like an AIPW-style thing we could add to GRF

https://github.com/serobertson/ExtendingInferences Extending inferences from a randomized trial to a new target population.pdf Appendix.pdf

[erikcs]

Hi @hhsievertsen, we've added a simple vignette describing a function you could copy/paste to estimate ATEs + SEs on a new test set here: https://grf-labs.github.io/grf/articles/ate_transport.html

[hhsievertsen]

Excellent. Just what I was after. Thanks a lot.

[hhsievertsen]

Hi @erikcs

one question. I am trying to get the intuition of the reweighting with the inverse odds term. If I am interested in the ATE for a subsample of my target sample, would you then restrict to this sub sample first and estimate the phat for being in the subsample of the target sample, or would you estimate the phats on the full sample first and then restrict to the subsample after estimating the phats?

Many thanks again! Hans

[erikcs]

Hi @hhsievertsen, fitting the p-model on the full target sample first sounds like the way to go the get best p-hat estimates, then you can subset afterwards for the subgroup you want

[hhsievertsen]

Hi @erikcs,

sorry to continue this closed topic. One final question, following up on my last question (I promise!). If I do that, n_OBS should still be for the full target sample, and not just n for the subsample within the target sample, or? Otherwise there is an asymmetry between p and n in the last term making it very large. Hope that makes sense Thanks! H

[erikcs]

Sorry, I think I misread your first post, if by subsample you mean you want the ATE for X.test.subset = X.test[subset, ], then you can just use the vignette example as average_treatment_effect_test(forest, X.test.subset)

[hhsievertsen]

Thanks @erikcs!