Two interesting articles were published within a few days of one another last week on the topic of experimental methods in political science.
The first article is by Jasjeet S. Sekhon and Rocio Titiunik in the American Political Science Review, and it discusses the uses and misuses of natural experiments:
Natural experiments help to overcome some of the obstacles researchers face when making causal inferences in the social sciences. However, even when natural interventions are randomly assigned, some of the treatment–control comparisons made available by natural experiments may not be valid. We offer a framework for clarifying the issues involved, which are subtle and often overlooked. We illustrate our framework by examining four different natural experiments used in the literature. In each case, random assignment of the intervention is not sufficient to provide an unbiased estimate of the causal effect. Additional assumptions are required that are problematic. For some examples, we propose alternative research designs that avoid these conceptual difficulties.
In other words, many of the natural experiments found in the literature do not allow identifying causal effects, and the authors do a good job of providing examples of four published natural experiments whose findings they question. The findings in Sekhon and Titiunik’s article apply to some regression discontinuity designs as well.