I had to address the topic of significance–broadly defined–at some point in this series of posts. With that said, this post is not about the seeming arbitrariness of the conventional levels of significance, i.e., of the 10, 5, and 1 percent levels of significance. First, because like Ellickson (1994), I believe that social norms emerge and evolve to minimize transactions costs, and the conventional levels of significance do just that by minimizing the amount of explanation producers of empirical findings have to provide for their findings and by minimizing the amount of thinking the consumers of empirical findings have to do in order to figure out just how (statistically) significant those findings are. In other words, the conventional levels of significance provide something for everyone to hang their hat on, and they are easy to explain to the general public.
And second, because as Noah Smith noted last summer, “[d]issing p-values in 2015 is a little like dissing macroeconomics in 2011–something that gives you a free pass to sound smart in certain circles … But like all hipster fads, I expect this one to fade.”
Rather, this post is about statistical vs. economic significance. Every so often, you run into a paper in which the authors have a good story, a good identification strategy, and robust, statistically significant findings, but in which there is little to no discussion of the findings’ economic significance.
What is economic significance? For the purpose of this discussion, let’s define statistical significance in its usual sense–Is the null hypothesis that the coefficient of interest is statistically different from zero rejected at the 90, 95, or 99 percent levels of confidence?Similarly, let’s define economic significance as how much something matters in the real world–Put simply: Is the treatment effect big or small?
So when I say that every so often you run into a paper where economic significance gets the short shrift, what I mean is that it is not uncommon for applied researchers (especially younger ones, or those of a more structural bent) to get so caught up in other things that when they have a statistically significant finding, they start writing and ignore the economic significance of their finding.
(I was that person once: In my first year as an assistant professor, I remember a senior colleague asking me what the marginal effect of my finding was, and I responded that I didn’t care about that given that I had statistical significance. More broadly, almost all of the posts in this series are motivated by a mistake I made at some point, and which I would like to help younger researchers avoid.)
So I guess I don’t have much more of a point than “Make sure you discuss the economic significance of your findings on top of their statistical significance.” Basic econometrics courses are of no help here, as they tend to be too generic to get into economic significance beyond broad recommendations. Applied courses tend to be a lot better; suppose you have a good identification strategy to estimate the effect of a policy in which some consumers receive a lump-sum transfer on those consumer’s marginal propensity to consume (MPC), which you find is significant at the 99 percent level of confidence. What if that estimate tells you that the effect of that lump-sum transfer is to change MPC by 0.02 percent? Chances are the policy isn’t really worth it. But if your estimate says that change is equal to 20 percent, the story changes.
Keeping statistical and economic significance in mind, there are four possible cases:
- A finding is statistically significant and economically significant. This is the ideal case, and the one that makes your job easiest when it comes to convincing readers that you have a publishable finding.
- A finding is statistically significant but economically insignificant. For me, this is second-best. You may be tempted to gloss over economic significance in such cases because you worry about the consequences of being honest about reporting an economically insignificant finding, but I think the consequences of trying to hide this are much worse than the consequences of being upfront about it. Besides, there is something to learn from such cases: Some things just don’t work, or they don’t work as well as previously thought.
- A finding is statistically insignificant and economically significant. This is very likely to happen when you have too small a sample size and you don’t have much statistical power. For such cases, I recommend taking a look at this old, under-appreciated Econometrica article by Don Andrews titled “Power in Econometric Applications.”
- A finding is statistically insignificant and economically insignificant. This is the most difficult case to work with. In order to publish such null findings, you have to work very, very hard to show that you are demonstrating evidence of absence of an effect rather than dealing with absence of evidence (i.e., low statistical power). I have managed to publish one such finding once, but it took (i) my contradicting widespread conventional wisdom, (ii) several robustness checks, and (iii) a sympathetic editor for this to happen.