Last updated on November 3, 2013
Though there was a time where critics of development economics could get away with throwing around terms like “neoliberal” and “Washington consensus” around in order to be heard by policy makers, it seems that nowadays, the views of development economists largely prevail in development policy. Part of that is most likely due to the overwhelming focus of development economists on answering narrower but answerable questions. That is, on questions like “Do deworming drugs improve educational outcomes?” rather than on questions like “Do structural adjustment programs foster economic growth?”
The focus on smaller questions has led to impact evaluation activities that are much more credible than they used to be. Whereas in the 1980s and 1990s one could get away with comparing outcomes pre- and post-intervention, today any impact evaluation worth its salt has to have a credible research design, i.e., one that allows credibly estimating the causal impacts of a given intervention.
So in the last few years, “impact evaluation” has become quite the buzzword, and everyone — from the greenest of students in Masters programs in development to the development NGOs, and from the big development agencies like USAID to philanthropies like the Gates Foundation — is obsessed with impact evaluation.
That’s a good thing, at least on the face of it: If we know what works, we can better target development interventions, and so development policy can more effectively lift people out of poverty.
Not in my Backyard?
But does everyone really want to be evaluated? I’ve long suspected that, for many actors in development policy, but specifically for NGOs, the answer is “No.” Indeed, many people work with NGOs because they are true believers in the mission of the NGO they work for. Oh, sure: they’ll talk about impact evaluation because the donors want to hear about it. But do they really want to be evaluated? On the one hand, there are true believers. On the other hand, there are those who think “Well, what if an impact evaluation finds no impact? In my heart of hearts I know what we do is right.”
Recently, I found the answer to my question, at least for an important NGO that has been claiming much (perhaps too much) success in an area in which I am doing research. Here is an email exchange I had with their director in charge of “impact evaluation.” You be the judge–I’ve made the exchange anonymous on their end, and ellipses denote parts where I have removed things that aren’t necessary to understand the exchange:
From: b***@NGO.org
To: marc.bellemare@gmail.com
Subject: Impact Evaluation Partnership ProposalHi Marc,
Thank you for your interest in [NGO]. Your research sounds very interesting.
I am leaving to New York tomorrow morning and I will not be back to the office before Tuesday.
I am sorry I won’t be able to respond to your request until the end of next week.
…
Best,
B***
This was promising, and I was pretty happy they responded to my email, so I wrote back almost immediately:
From: marc.bellemare@gmail.com
To: b***@[NGO].org
Subject: Impact Evaluation Partnership ProposalHi B***,
Thanks for the quick response. I will have time to chat next week after you return … Is there a time when you would be free to chat on Wednesday or Thursday? I leave for Canada on Friday.…Thanks,MArc
This was the response I got, verbatim:
From: b***@[NGO].org
To: marc.bellemare@gmail.com
Subject: Impact Evaluation Partnership ProposalHi Marc,
I am really sorry but I am simply too overwhelmed with the work of the department right now to take on new commitments. [NGO] is in the middle of many changes and staff are pressed on all fronts. We don’t have the bandwidth to work with external research partners at this time. We haven’t done that in the past but we are hoping to be able to in the future.
…
Best,
B**
Thinking that I hadn’t been very clear in my initial email, I responded as follows:
From: marc.bellemare@gmail.com
To: b***@[NGO].org
Subject: Impact Evaluation Partnership Proposal
B***,I think you misunderstood what I had in mind. I am not asking you to do any work, and I am not asking [NGO] to conduct any impact evaluation either. We would be coming in with our own money to evaluate the impact of [NGO]’s program and of alternative policies in an effort to understand what works.
All that we would need would be a thorough understanding of what it is that [NGO] does a part of its [major policy intervention], so that we can design similar [interventions] as one of a few (policy) treatments we wish to look at in order to compare impacts.
My understanding is that [NGO] wants to maximize its impact … As such, we both are after the same objective, and this project would require zero funds committed from you guys. We’re only asking for a bit of your time in sharing your procedures if this gets funded.
Marc
I never got a response.