“The Determinants of…”: How Not to Do Social Science

Last updated on November 30, 2014

A nice post by Frances Woolley titled “Economists Aren’t in the Prediction Business–and That’s a Good Thing” over at Worthwhile Canadian Initiative reminded me of how much I dislike “determinants” papers, and so I thought I should write a short post about the topic.

I frequently run into “determinants” papers. Typically, a student in the second-year qualifying research paper seminar I teach in our PhD program will submit a paper proposal that is a “determinants” paper. More frequently, however, I see “determinants” papers when new manuscripts are assigned to me for handling at Food Policy.

“What’s a ‘determinants’ paper?,” you ask? A “determinants” paper is a paper in which the authors do not specifically try to answer a question of the form “Does X cause Y?”

Rather, authors of “determinants” papers typically regress some outcome of interest (e.g., whether one works or not) on a number of covariates (e.g., age, gender, education, race, etc.), look at what’s significant, and then they make up stories about why those covariates that are significant have specific signs.

The problem is that “determinants” paper are hardly social science. Any reasonably smart undergraduate can load a data set into Stata, run a linear regression (though typically, “determinants” paper writers are absolutely obsessed with probit and logit, for some reason), and make stuff up about why the partial correlations she has estimated look the way they do. That’s not how social science works. A quick refresher on the scientific method:

1. Formulate a hypothesis: “An increase in X causes an increase in Y.”
2. Make a prediction: “If X increases, so will Y.”
3. Testing: Collect data on X and on Y, and use the right methods to identify the putative causal relationship flowing from X to Y.
4. Analysis: Answer the research question. If the theory has been falsified on the basis of a credible research design, offer ways in which the theory can be adapted to account for the result. Otherwise, advocate for replication.

The issue with “determinants” papers is that they put the ox before the cart, i.e., the author decided to have a bit of fun playing with data, found some interesting partial correlations, and then retro-fitted a story to fit the facts. And when you read “determinants” papers, they typically feel cheap. It probably goes without saying that, in economics journals at least, “determinants” papers are much less likely to get published than papers that follow the scientific method.

This isn’t to say that “determinants” paper don’t have their uses. For example, a “determinants” paper can help uncover who is the most likely to take up some kind of optional policy intervention. But one has to be exceedingly clear about the limitations of that type of work, and a “determinants” paper is highly unlikely to land you a publication at a top field or general journal.

I am working on a paper that looks at what drives people to practice urban agriculture with the objective of informing policies that aim to encourage people to practice urban agriculture, for example, but I don’t expect this to be my best publication ever. Rather, I just want to guide policy making, given that many seem to want to shove urban agriculture down the throats of people who are probably the least likely to want to practice it.