Econometrics

Development / Econometrics / Economics / Methods / Policy / Politics — 2 Comments
7
May 12

Implementation Bias in Randomized Controlled Trials

From a new paper (link opens a .pdf file) by Oxford’s Tessa Bold and her coauthors:

The recent wave of randomized trials in development economics has provoked criticisms regarding external validity and the neglect of political economy. We investigate these concerns in a randomized trial designed to assess the prospects for scaling-up a contract teacher intervention in Kenya, previously shown to raise test scores for primary students in Western Kenya and various locations in India. The intervention was implemented in parallel in all eight Kenyan provinces by a nongovernmental organization (NGO) and the Kenyan government. Institutional differences had large e ffects on contract teacher performance. We find a signifi cant, positive effect of 0.19 standard deviations on math and English scores in schools randomly assigned to NGO implementation, and zero effect in schools receiving contract teachers from the Ministry of Education. We discuss political economy factors underlying this disparity, and suggest the need for future work on scaling up proven interventions to work within public sector institutions.

Bold et al.’s finding points to an important problem with the findings of many randomized controlled trials (RCTs): No matter how careful one is in ensuring that subjects are randomly assigned to the treatment and control groups, almost all RCTs rely on only one implementing partner. Continue reading →

Econometrics / Economics / Impact Evaluation / Methods / Policy / Politics / Social Sciences — 6 Comments
24
Apr 12

Identifying Causal Relationships vs. Ruling Out All Other Possible Causes

Portrait of Artistotle (Source: Wikimedia Commons.)

I was in Washington last month to discuss my work on food prices, in which I look at whether food prices cause social unrest, at an event whose goal was to discuss the link between climate change and conflict.

As many readers of this blog know, disentangling causal relationships from mere correlations is the goal of modern science, social or otherwise, and though it is easy to test whether two variables x and y are correlated, it is much more difficult to determine whether x causes y.

So while it is easy to test whether increases in the level of food prices are correlated with episodes of social unrest, it is much more difficult to determine whether food prices cause social unrest.

In my work, I try to do so by conditioning food prices on natural disasters. To make a long story short, if you believe that natural disasters only affect social unrest through food prices, this ensures that if there is a relationship between food prices and social unrest, that relationship is cleaned out of whatever variation which is not purely due to the relationship flowing from food prices to social unrest. In other words, this ensures that the estimated relationship between the two variables is causal. This technique is known as instrumental variables estimation.

Identifying Causal Relationships vs. Ruling Out All Other Causes

As with almost any other discussion of a social-scientific issue nowadays, the issue of causality came up during one of the discussions we had at that event in Washington. It was at that point that someone implied that it did not make sense to talk of causality by bringing up the following analogy: Continue reading →

Agriculture / Econometrics / Economics / Impact Evaluation / Methods — Comments Off
19
Mar 12

Slides of My Keynote Lecture at Last Weekend’s “Economics and Management of Risk in Agriculture and Natural Resources” Conference

I was trained as an agricultural and applied economist, so I have spent a lot of time doing research on risk as it relates to agriculture and development (see here and here for published articles).

Because of this, I have been involved with the annual Economics and Management of Risk in Agriculture and Natural Resources conference for the past few years.

I first presented at that conference in 2009, and since I had then volunteered to organize the conference, I was in charge of the conference program in 2010 and of logistics in 2011.

This year, I was asked to give the keynote lecture, in which I chose to discuss what the “credibility revolution” that took place in economics over the past ten years or so — which has lead to economists to adopting stricter standards of evidence and of statistical identification — means for agricultural and applied economics as a field.

In case you have an interest in this topic, I am making the slides of my keynote lecture are available. I think the content of those slides is especially relevant for current graduate students of agricultural and applied economics.

The Economics and Management of Risk in Agriculture and Natural Resources conference is usually held somewhere on the Gulf Coast. This year, it was held in Pensacola, FL. I took the picture on top of this post while walking along the beach early Saturday morning.

Econometrics / Impact Evaluation / Methods / Social Sciences — Comments Off
16
Feb 12

Randomization and Inference

Experiments have become an increasingly common tool for political science researchers over the last decade, particularly laboratory experiments performed on small convenience samples. We argue that the standard normal theory statistical paradigm used in political science fails to meet the needs of these experimenters and outline an alternative approach to statistical inference based on randomization of the treatment. The randomization inference approach not only provides direct estimation of the experimenter’s quantity of interest — the certainty of the causal inference about the observed units — but also helps to deal with other challenges of small samples. We offer an introduction to the logic of randomization inference, a brief overview of its technical details, and guidance for political science experimenters about making analytic choices within the randomization inference framework. Finally, we reanalyze data from two political science experiments using randomization tests to illustrate the inferential differences that choosing a randomization inference approach can make.

That’s the abstract of a forthcoming American Journal of Political Science article by Luke Keele, Corrine McConnaughy, and Ismail White.

That being said, I really can’t wait for summer to arrive so I can finally get through my “Documents to Read” folder.

Commentary / Econometrics / Impact Evaluation / Methods / Social Sciences — 2 Comments
9
Feb 12

On the (Mis)Use of Regression Analysis: Country Music and Suicide

This article assesses the link between country music and metropolitan suicide rates. Country music is hypothesized to nurture a suicidal mood through its concerns with problems common in the suicidal population, such as marital discord, alcohol abuse, and alienation from work. The results of a multiple regression analysis of 49 metropolitan areas show that the greater the airtime devoted to country music, the greater the white suicide rate. The effect is independent of divorce, southernness, poverty, and gun availability. The existence of a country music subculture is thought to reinforce the link between country music and suicide. Our model explains 51 percent of the variance in urban white suicide rates.

That’s the abstract of an article published in Social Forces – a top-10 journal in sociology — in 1992.

Before my snark gets me into trouble: Yes, I do realize that the article was published in 1992, back when most social science researchers only had a flimsy grasp of identification and causality. I also realize it would be foolish to impose on the authors of the above-referenced article the same standards of identification we impose upon ourselves today.

Yet, I cannot help but think that someone with a lesser of understanding of causality than the average reader of this blog is bound to eventually stumble upon the abstract, think “Hey, that totally makes sense!,” and run with it.

I’m sure there are also examples of such findings in other disciplines. If you know of any, please share.

(HT: Friend and former student Norma Padron, who is doing her PhD at Yale and has just launched a nice health economics blog.)

← Older Entries