Last updated on January 19, 2017
A few weeks ago, one of my doctoral advisees wrote to me asking me how she could test for a specific mechanism behind the causal effect she is trying to estimate in her job-market paper.
Letting [math]y[/math] be her outcome of interest, [math]D[/math] her treatment of interest, [math]x[/math] be a vector of control variables, and [math]\epsilon[/math] be an error term with mean zero, my student was estimating
(1) [math]y = \alpha_{0} + \beta_{0}{x} + \gamma_{0}{D}+\epsilon[/math],
in which she was interested in [math]\gamma[/math], or the causal impact of [math]D[/math] on [math]y[/math].
But more importantly for the purposes of this post, she was also interested in whether [math]M[/math] is a mechanism through which [math]D[/math] causes [math]y[/math]. I suggested the usual thing I often see done, which is to estimate
(1′) [math]y = \alpha_{1} + \beta_{1}{x} + \phi_{1}{M} + \gamma_{1}{D}+\nu[/math],
in which case if [math]\gamma[/math] dropped out of significance and [math]\phi[/math] was significant (and had the “right”) sign, then she could say that [math]M[/math] was a mechanism through which [math]D[/math] cause [math]y[/math]. I also suggested maybe conducting a Davidson-MacKinnon J-test for non-nested hypotheses to assess the robustness of her mechanism finding.
As it turns out, the standard method of testing for mechanisms show in in equation (1′) above is wrong. A few days after my student asked me how to test for her posited mechanism, my colleague Jason Kerwin (if you don’t already know of his work, you should) commented in a different context at our Friday Trade and Development seminar that this was a mistaken way to do it, and talked about a recent paper by Acharya et al. (2016) in the American Political Science Review in which the authors:
- Show that the method laid out in equation 1 can lead to biased estimates,
- Develop a method to test for whether [math]M[/math] is a mechanism through which [math]D[/math] causes [math]y[/math] which relies on determining which controls are pre-treatment, and which controls are post-treatment (including presumed mechanism [math]M[/math]), and
- Provide R and Stata codes to use their method.
As if that weren’t important enough, Acharya et al.’s method allows doing one thing that social scientists often dream of doing, viz. ruling out all other possible mechanisms, beyond any reasonable (i.e., statistical) doubt.
That is, their method not only tells you whether [math]M[/math] is a mechanism through which [math]D[/math] causes [math]y[/math], it can also tell you whether there is any significant amount of statistical variation left in the causal relationship flowing from [math]D[/math] through [math]y[/math] after [math]M[/math] is accounted for. In my experience, this really is the most powerful part of Acharya et al.’s contribution.
Acharya et al.’s paper should be read in its entirety, but briefly, their method consists in the following for the example above:
- Split the vector of controls [math]x[/math] into pre- and post-treatment controls, respecitvely [math]x_{0}[/math] and [math]x_{1}[/math]. Let [math]x = (x_{0},x_{1})[/math].
- Estimate equation (1′) above.
- Compute [math]\tilde{Y} = Y – \hat{\phi}{M}[/math].
- Estimate [math]\tilde{Y} = \delta + \theta{X}_{0} + \kappa{D}+\xi[/math].
The coefficient of interest is still [math]D[/math], but it now tells us what is the effect of the treatment variable after removing the effect of the mechanism. This means that if one fails to reject the null hypothesis that [math]\kappa = 0[/math], this means that the treatment itself has no effect on the dependent variable once the mechanism is taken into account.*
(The setup in steps 2 to 4 above relies on generated regressors, the whole procedure needs to be bootstrapped, which is why Acharya et al. provide R and Stata codes; there are two typos in their Stata code; if you’d like to use it, email me see below.)
As soon as I realized what Acharya et al.’s method could do, I had to try it out. In a paper I am working on with coauthors, we look at whether participation in contract farming provides partial insurance to growers by way of reducing income variability. Our preliminary findings suggest that yes, participation in contract farming does reduce income variability, and thus provide growers with a means of partially insuring against income fluctuations.
One of the theoretical predictions in our paper is that this happens via the presence in the data of fixed-price contracts–that is, contracts which guarantee a given price per unit. Using Acharya et al.’s method, I find that fixed-price contracts appear to be the only mechanism by which participation in contract farming provides partial insurance. More on this paper in a few months, when it is ready to circulate.
UPDATE: After receiving several emails asking me for it, here is the fixed up Stata code. Keeping the above notation, the following code (i) removes the semicolon on the last line, which prevented the code from running in my version of Stata, and (ii) multiplies [math]\hat{\phi}[/math] by [math]M[/math] when computing [math]\tilde{Y}[/math]:
gen y_tilde=. capture program drop deboot program define deboot, rclass #delimit ; regress y M D x_0 x_1; replace y_tilde = y - _b[M]*M; regress y_tilde D x_0; return scalar deffect = _b[D]; #delimit cr end bootstrap deffect = r(deffect), reps(500) seed(123456789): deboot
___
* There is no way a blog post can render justice to a paper as rich and interesting as Acharya et al.’s, and their argument is more complicated. Again, the paper should be read in full to fully understand the assumptions behind and the limitations of their method.