Category: Uncategorized

The annual meetings of the Agricultural and Applied Economics Association begin tomorrow night in Chicago, so before I fly to Midway, I thought it would be a good time to talk about the agricultural and applied economics profession as distinct from the economics profession, but also in terms of how it closely resembles economics.

To do so, I thought I should feature a recent Twitter thread by Beatrice Cherrier, a historian of economic thought at the University of Caen in Normandie who is currently looking at the history of agricultural and applied economics. I’ll post Beatrice’s tweets below, interspersed with some of my own comments.

I received the following email from Indiana University’s Dan Sacks in response to my post yesterday:

Dear Marc,

I enjoy reading your blog and find it informative and stimulating. However your recent post on “achieving statistical significance with covariates” may mislead some readers. The basic issue is that it seems fine to me if the precision of your coefficient is sensitive to the inclusion of pre-determined covariates, as long as the expected value is not. That is, in such cases it seems fine to emphasize the precisely estimated result.

Here are more details. You note that in the model

Y = a + bX + cD + e,

the estimated coefficient c on D might or might be statistically significant, depending on what is included in the control vector X. The usual concern in the applied literature—which of course I share completely—is that if we don’t condition on a sufficient set of confounders, then c is estimated with bias. We all want to avoid bias. Bias is about expected values, though, not statistical significance, and it is not obvious to me that we should be worried about models in which including covariates changes the statistical significance (but not the expected value) of the results. Including pre-determined regressors which are uncorrelated with D but (conditionally) correlated with Y will generally reduce var(e), reducing the standard error of c and possibly leading to statistical significance. The fact that our results are only significant if we control for some set of X’s does not necessarily mean that there is bias – only that we might be underpowered without enough controls.

Here’s a hypothetical example. You run an RCT looking at the effect of an unconditional cash transfer on happiness. You randomly assign different people to get money or not. This is an expensive intervention so you don’t have a huge sample. You estimate that D = 0.1 (se = 0.06) without controlling for anything, but when you control for a vector of characteristics measured at baseline, you estimate D = 0.09 (se= 0.04). In this case, I think we would all agree that it’s fine to control for age. [There is a different issue which is about data mining for statistical significance, but I think that’s not the point you’re raising, either.]

I used to be a real purist about this, particularly as a grad student. “If your experiment/IV is valid,” I would ask, “then why do you need to include controls?” But the answer is that controls help with statistical precision.

Dan

Those of us who do applied work for a living will have at some point noticed that, depending on which variables we include in X on the right-hand side (RHS) of an equation like

(1) y = a + bX + cD + e,

the coefficient c on the treatment variable D might go from significant to insignificant or vice versa.

That this is true is the very reason why it is common practice in applied work to present several specifications of equation (1) in the same table, ranging from the most parsimonious (i.e., a regression of y on D alone) to slightly less parsimonious (i.e., a regression of y on D and ever increasing subsets of X) to the least parsimonious (i.e., a regression of y on D and all the controls in X). It is also the rationale behind the method put forth by Altonji et al. (2005) to assess the robustness of a finding.

I came across an interesting new working paper by Lenz and Sahn by way of Dave Giles’ blog, titled “Achieving Statistical Significance with Covariates,” in which the authors conduct an interesting meta-analysis of articles published in the American Journal of Political Science which reveals that in almost 40% of the observational studies analyzed, researchers obtain statistical significance of c by tinkering with the covariates included (or not, as it were) in X.

Here is the abstract of Lenz and Sahn’s paper: