When we write a dynamic model in economics, we typically use the subscript t to denote a given time period, and we usually say that t = 1, 2, …, T, where T denotes the last time period considered by our model. Likewise, we usually use T to denote the number of time periods considered in a longitudinal data set.
With that in mind, the World Bank’s David McKenzie argues for more T in the experiments conducted by development economists in a forthcoming article in the Journal of Development Economics:
The vast majority of randomized experiments in economics rely on a single baseline and single follow-up survey. If multiple follow-ups are conducted, the reason is typically to examine the trajectory of impact effects, so that in effect only one follow-up round is being used to estimate each treatment effect of interest. While such a design is suitable for study of highly autocorrelated and relatively precisely measured outcomes in the health and education domains, this article makes the case that it is unlikely to be optimal for measuring noisy and relatively less autocorrelated outcomes such as business profits, household incomes and expenditures, and episodic health outcomes. Taking multiple measurements of such outcomes at relatively short intervals allows one to average out noise, increasing power. When the outcomes have low autocorrelation and budget is limited, it can make sense to do no baseline at all. Moreover, I show how for such outcomes, more power can be achieved with multiple follow-ups than allocating the same total sample size over a single follow-up and baseline. I also highlight the large gains in power from ANCOVA analysis rather than difference-in-differences analysis when autocorrelations are low and a baseline is taken. This article discusses the issues involved in multiple measurements, and makes recommendations for the design of experiments and related non-experimental impact evaluations.
This brings to mind what one of my friends who works in the microfinance industry had told me the last time we argued about the effects of microfinance on poverty: “It can take a long time to get out of poverty even in the best of scenarios, so evaluating the impact of microfinance after just one or two years tends to shortchange microfinance.”
Moreover, this makes me less worried about not having had the luxury of conducting a baseline survey for the randomized controlled trial I am conducting with Michael Carter and Catherine Guirkinger on the impacts of crop insurance on the welfare of cotton producers in southern Mali. Thankfully, we will be doing at least two rounds of follow-up survey in order to study the dynamic effects of our intervention, and we are working on finding funding for a third round.
UPDATE: In the time between the moment I wrote this post on Sunday morning and the moment it was published, David offered his own blog post on his paper on the World Bank’s Development Impact blog.
Hipstermetrics
At first we were convinced that 100 percent of the variance in bike market size could be explained by the population density of a city. If you live an a densely populated area like San Francisco, bicycling is an efficient way to get around the city. If you live in Los Angeles, getting on a bicycle can’t really get you anywhere. To our surprise, population density has a nearly zero correlation with our bicycle index. If anything, it very weakly suggests the more densely populated the city, the less prevalence of biking.
That’s from a post titled “The Fixie Bike Index,” on the Priceonomics Blog.
If, like me, you are nowhere near hip enough to know what a fixie is, let me spare you a Google search: a fixie is a fixed-gear bicycle, which is apparently a much-coveted item among hipsters. That’s right: grown men and women enjoy riding around town on a bike like the one you and I used to ride when we were 8 years old.
That being said, let’s go back to the image above. Note how the above-referenced post explains how “population density has a nearly zero correlation with our bicycle index,” which, “[i]f anything, (…) very weakly suggests the more densely populated the city, the less prevalence of biking.”
I guess someone missed the lecture on how sensitive the mean is to outliers back in college. A quick look at the scatter plot and regression line above indicate that the latter is driven by the point on the far right.
Remove that point, and it looks like there might be a positive relationship between a city’s bike index and the density of its population. Trim all four outliers, and it’s really not obvious what is going on.
Surely there’s a bookshop in Williamsburg that has a used copy of Kennedy’s Guide to Econometrics for sale?
(HT: @mungowitz‘s snark, which is not to be confused with Echidna’s Arf.)