Last updated on November 30, 2014
!["Bro, this elixir will guarantee that my predicted values are not outside of [0,1]."](https://marcfbellemare.com/wordpress/wp-content/uploads/2014/11/Longhi.jpg)
I frequently run into “determinants” papers. Typically, a student in the second-year qualifying research paper seminar I teach in our PhD program will submit a paper proposal that is a “determinants” paper. More frequently, however, I see “determinants” papers when new manuscripts are assigned to me for handling at Food Policy.
“What’s a ‘determinants’ paper?,” you ask? A “determinants” paper is a paper in which the authors do not specifically try to answer a question of the form “Does X cause Y?”
Rather, authors of “determinants” papers typically regress some outcome of interest (e.g., whether one works or not) on a number of covariates (e.g., age, gender, education, race, etc.), look at what’s significant, and then they make up stories about why those covariates that are significant have specific signs.
The problem is that “determinants” paper are hardly social science. Any reasonably smart undergraduate can load a data set into Stata, run a linear regression (though typically, “determinants” paper writers are absolutely obsessed with probit and logit, for some reason), and make stuff up about why the partial correlations she has estimated look the way they do. That’s not how social science works. A quick refresher on the scientific method:
- Formulate a hypothesis: “An increase in X causes an increase in Y.”
- Make a prediction: “If X increases, so will Y.”
- Testing: Collect data on X and on Y, and use the right methods to identify the putative causal relationship flowing from X to Y.
- Analysis: Answer the research question. If the theory has been falsified on the basis of a credible research design, offer ways in which the theory can be adapted to account for the result. Otherwise, advocate for replication.
The issue with “determinants” papers is that they put the ox before the cart, i.e., the author decided to have a bit of fun playing with data, found some interesting partial correlations, and then retro-fitted a story to fit the facts. And when you read “determinants” papers, they typically feel cheap. It probably goes without saying that, in economics journals at least, “determinants” papers are much less likely to get published than papers that follow the scientific method.
This isn’t to say that “determinants” paper don’t have their uses. For example, a “determinants” paper can help uncover who is the most likely to take up some kind of optional policy intervention. But one has to be exceedingly clear about the limitations of that type of work, and a “determinants” paper is highly unlikely to land you a publication at a top field or general journal.
I am working on a paper that looks at what drives people to practice urban agriculture with the objective of informing policies that aim to encourage people to practice urban agriculture, for example, but I don’t expect this to be my best publication ever. Rather, I just want to guide policy making, given that many seem to want to shove urban agriculture down the throats of people who are probably the least likely to want to practice it.
Nice post Marc. I generally agree about “determinants” papers. In my mind, the key to your post is the insincere way much of this research is presented, i.e. lying about the research process.
I do see a few minor roles for this type of research. Your drivers of an optional policy intervention “determinants” paper example makes sense to me. Another possible area for this research is pre-hypothesis planning. Having lower tier papers document previously found partial correlations might help future researchers with hypothesis forming. But the space for this type of paper is quite small, and definitely at the lower end of the research spectrum.
Here’s hoping less of these papers are submitted to Food Policy in the coming days.
Sounds like a kitchen sink regression.
Marc, isn’t it more about the identification problems rather than sequencing?
Steps 1 and 2 are always present, though implicitly. People have informal mental models that generate hypotheses. So, intuition makes first steps, then Stata steps in and shows correlation. (These variables didn’t get into regressions accidentally, right?)
The problem is that when “the determinants of x” aren’t really the determinants of x. But this happens even if you follow the scientific method. In contrast, if your intuition got the structural model right and the properly done estimation confirmed it—you have determinants useful for policy making.
PS: Glad to find your blog, Marc.
Thanks for reading and for your comment, Anton. Glad you enjoy the blog.
Yes, people (almost) always have a theory of change in their mind, which drives the exclusion/inclusion of certain covariates or controls. And I guess that yes, what bothers me about the determinants approach is that its almost always completely unidentified. I think your point about policy making is similar to what I was saying about external validity and policy targeting.
It is also possible to use the word determinants in a title of a paper while also following the scientific method. And it is also possible to not use the word “determinants” when in fact not even following the scientific method. The process is hard to pin down unless one is identifying a model from a theoretical specified model or framework, from which many variables are often ignored or assumed. Many discoveries have come out of the so called kitchen sink methods or when working outside the frontiers of set hypothesis. Now will you be biased regarding what paper to read or analyze based on this creterion? We are getting into data mining methods where you go beyond set hypothesis and look for hypothesis.
Right. My post had nothing to do with paper titles, and everything to do with the method (or lack thereof) followed.
As for my biases, as an editor, I am mostly biased against flawed analyses.