Impact Evaluation: Not in my Backyard?

Though there was a time where critics of development economics could get away with throwing around terms like “neoliberal” and “Washington consensus” around in order to be heard by policy makers, it seems that nowadays, the views of development economists largely prevail in development policy. Part of that is most likely due to the overwhelming focus of development economists on answering narrower but answerable questions. That is, on questions like “Do deworming drugs improve educational outcomes?” rather than on questions like “Do structural adjustment programs foster economic growth?”

The focus on smaller questions has led to impact evaluation activities that are much more credible than they used to be. Whereas in the 1980s and 1990s one could get away with comparing outcomes pre- and post-intervention, today any impact evaluation worth its salt has to have a credible research design, i.e., one that allows credibly estimating the causal impacts of a given intervention.

So in the last few years, “impact evaluation” has become quite the buzzword, and everyone — from the greenest of students in Masters programs in development to the development NGOs, and from the big development agencies like USAID to philanthropies like the Gates Foundation — is obsessed with impact evaluation.

That’s a good thing, at least on the face of it: If we know what works, we can better target development interventions, and so development policy can more effectively lift people out of poverty.

Not in my Backyard?

But does everyone really want to be evaluated? I’ve long suspected that, for many actors in development policy, but specifically for NGOs, the answer is “No.” Indeed, many people work with NGOs because they are true believers in the mission of the NGO they work for. Oh, sure: they’ll talk about impact evaluation because the donors want to hear about it. But do they really want to be evaluated? On the one hand, there are true believers. On the other hand, there are those who think “Well, what if an impact evaluation finds no impact? In my heart of hearts I know what we do is right.”

Recently, I found the answer to my question, at least for an important NGO that has been claiming much (perhaps too much) success in an area in which I am doing research. Here is an email exchange I had with their director in charge of “impact evaluation.” You be the judge–I’ve made the exchange anonymous on their end, and ellipses denote parts where I have removed things that aren’t necessary to understand the exchange:

From: b***@NGO.org
To: marc.bellemare@gmail.com
Subject: Impact Evaluation Partnership Proposal

Hi Marc,

Thank you for your interest in [NGO]. Your research sounds very interesting.

I am leaving to New York tomorrow morning and I will not be back to the office before Tuesday.

I am sorry I won’t be able to respond to your request until the end of next week.

Best,

B***

This was promising, and I was pretty happy they responded to my email, so I wrote back almost immediately:

From: marc.bellemare@gmail.com
To: b***@[NGO].org
Subject: Impact Evaluation Partnership Proposal

Hi B***,

Thanks for the quick response. I will have time to chat next week after you return … Is there a time when you would be free to chat on Wednesday or Thursday? I leave for Canada on Friday.
Thanks,
MArc

This was the response I got, verbatim:

From: b***@[NGO].org
To: marc.bellemare@gmail.com
Subject: Impact Evaluation Partnership Proposal

Hi Marc,

I am really sorry but I am simply too overwhelmed with the work of the department right now to take on new commitments. [NGO] is in the middle of many changes and staff are pressed on all fronts. We don’t have the bandwidth to work with external research partners at this time. We haven’t done that in the past but we are hoping to be able to in the future.

Best,

B**

Thinking that I hadn’t been very clear in my initial email, I responded as follows:

From: marc.bellemare@gmail.com
To: b***@[NGO].org
Subject: Impact Evaluation Partnership Proposal
B***,

I think you misunderstood what I had in mind. I am not asking you to do any work, and I am not asking [NGO] to conduct any impact evaluation either. We would be coming in with our own money to evaluate the impact of [NGO]‘s program and of alternative policies in an effort to understand what works.

All that we would need would be a thorough understanding of what it is that [NGO] does a part of its [major policy intervention], so that we can design similar [interventions] as one of a few (policy) treatments we wish to look at in order to compare impacts.

My understanding is that [NGO] wants to maximize its impact … As such, we both are after the same objective, and this project would require zero funds committed from you guys. We’re only asking for a bit of your time in sharing your procedures if this gets funded.

Marc

I never got a response.

No related content found.

6 comments

  1. Mary

    I was thinking about this in the past after several discussions with people on the impact of projects by NGOs. One person in a discussion was aware of some supposed research projects that merely ended, pulled, so it’s impossible to evaluate. I was thinking about it again when I heard the podcast for “I was just trying to help” earlier this year when they talked about not wanting to look at the data: http://www.thisamericanlife.org/radio-archives/episode/503/i-was-just-trying-to-help (Money for nothing and your cows for free).

    It reminds me of the unpublished clinical trials issues. I understand that sometimes the data isn’t good. And I accept that some projects will fail. But evaporating the project, or not looking at data at all, is just as bad when NGOs do it as when pharma does it. There should be an “AllTrials” for all NGO projects too.

  2. Dave Algoso

    Marc, I agree with your general point. No one really wants to be evaluated if they already believe in what they’re doing, and already face accountability pressures on other fronts. Why open yourself to a new one? There’s also a difference between evaluation for accountability versus evaluation for learning (the “MeE” of Lant Pritchett et al) but the more it sounds like the former, the less someone will want to engage in the latter.

    But that said, I think your example in no way proves or even supports the point. I have to jump to the defense of the anonymous evaluation director (even though you asked me to be the judge — not that anyone’s ever had to ask). I think you’re being ungenerous to them. I’d be curious what your initial description of the research looked like, because even from the steps you describe in your final response, it sounds like a couple of meetings that someone with a packed schedule has little reason to take. “All that we would need is a thorough understanding” still sounds like a lot of work for them. And “my understanding is that [NGO] wants to maximize its impact” even sounds a bit challenging. I gotta be honest, if I got those emails from someone I didn’t know and had never worked with before, I’d be hesitant to meet with you. Sorry to be so blunt.

  3. Marc F. Bellemare

    Dave, thanks for your thoughts, and no need to apologize for being blunt; I was looking for honest feedback. What I wanted was for the organization to share it’s written materials on or protocols for what they were doing, so we could reproduce it ourselves, full stop. At worst, I think it would’ve involved sending me a ton of pdf documents that I would have read. Or, I could’ve accompanied one of their people to the field at some point–again, at no cost to them.

    Here’s the initial email I sent the email address I initially found for the NGO:

    “I am an assistant professor in the Department of Applied Economics at the University of Minnesota, and I am writing to ask whether we could chat in the next five to 10 days.

    I am currently writing a grant proposal for the [donor] in which I propose to look at [topic]. Specifically, I am interested in doing two things:

    1. Year 1: [Something entirely on my own unrelated to the work of those guys, except that it's on the same topic.]

    2. Years 2 and 3: Conduct an impact evaluation of [policies], including [NGO]‘s policy intervention.

    I am contacting you because I believe #2 above could help [NGO] fine-tune its approach and answer questions [NGO] decision makers might have, and because I would like to partner with [NGO] if at all possible. In that sense, both [NGO] and our research team would benefit from such a partnership.”

    As you can see, I wasn’t asking for much, only a conversation and some reading material.

  4. Marc F. Bellemare

    P.S.: Maybe my “my understanding is that [NGO] wants to maximize its impact” does sound challenging, but whenever I’m served the “I’m too busy” line, my BS detector goes off. Everyone is busy, but anyone can make time for what matters to them.

  5. ben

    I’m sympathetic to the NGO guy. Put yourself in their shoes. =
    1. The evaluator is going to tell you whether you program is a success or failure using a metholodgy you don’t understand in a process completely out of your control
    2. The methodology is billed as “scientific” and this is intriguing, many people put great stock in this methodology. However, it does have some smart critics who seem to make good points.
    3. You’re aware that evaluations using this methodology sometimes find no impact, not because the project has failed but because of data problems, timing, etc. You’re not sure how you would try to explain this if it happened in your case.
    4.You’ve seen practitioners of the methodlogy in a room together, tearing apart each other’s results, making you wonder how strong these results are really going to be and who’s going to come along and tear apart the results of your evaluation.
    5. You’ve heard stories of academics completely losing interest in evaluating a project and leaving it to ruin as soon as it becomes clear that the results won’t make it into a hgihly ranked journal

    Would you really want your project evaluated? I wouldn’t. In fact, I think it’s more suprising when someone wants to have their project evaluated than when they don’t.

  6. MJ

    I’m with Dave, but for a simpler reason. No freebies are truly free. Volunteers always need management support to be of any use, and get anything out of their internship. Your offer is akin to a tied donation (only to be used for impact evaluation by UoM).

    Now maybe you know this NGO’s project area intimately, but as an NGO practitioner I have gotten more than a bit wary of these kind of offers. Inevitably the researchers need all kinds of logistical support etc. Even if you guys do not, and even if that was totally clear in your email, a busy person (as B*** claims to be) may have misread the email. On the other hand if you turned up with an offer of, say, 30% of the core costs of the project being evaluated, plus a free evaluation (by rigorous, unsentimental academics without a particular axe to grind), then I would hope/expect they would have been much more enthusiastic.

    That is not to say that B*** could not have reacted better (although your blog post might not be the best incentive in that regard), but I sympathise with their plight. People always get offended when NGOs refuse free stuff, and yet, even though your offer was a million miles from SWEDOW, the mark of a good NGO is to turn down what you don’t need just as much as it is to welcome robust, independent impact evaluation.

    Maybe they’ve already got another university partnership doing just that, or at least part of that? But the parts not already covered are difficult to explain etc, and B*** is struggling to find the time to elucidate them?