The impact of evaluation
Alanna Sheikh started a bit of a debate last week on the limitations of impact evaluations. She cites Andrew Natsios (a former USAID administrator)
USAID has begun to favor health programs over democracy strengthening or governance programs because health programs can be more easily measured for impact. Rule of law efforts, on the other hand, are vital to development but hard to measure and therefore get less funding.
Lots of things are vital for development, but something being vital doesn’t mean that aid funding is necessarily an effective way of supplying it. Not only that, but something being difficult to measure does not make it impossible. And sure enough, JPAL and IPA have conducted a number of evaluations of governance projects, such as working with the police in Rajasthan, on peace education and ex-combatant reintegration projects in Liberia, and evaluating anti-corruption strategies in Indonesia.
Randomised impact evaluations give the strongest evidence available on a project’s effectiveness. If USAID is beginning to favor projects with evidence of impact that is a good thing. The challenge for governance and rule of law advocates is to prove their impact.
Dennis Whittle of Globalgiving.org adds another limitation:
Formal evaluations, including the gold standard of randomized controlled trials, are not scalable. We simply do not have the time and resources to do centralized, in-depth evaluations of everything.
This argument is like not bothering with lifeboats if they can’t fit everyone in. Evaluations are crucial if we are going to learn whether or not we are wasting our money. And who knows, we might not be able to evaluate every single project, but if we keep coming up with compelling theories of change and keep replicating our findings in different settings, we could certainly try to evaluate every single intervention.
Print this page
Share
Comments
Measuring impact
Our institute offers a workshop to teach organizations how to measure the impact of their BoP ventures. We help ventures choose the best methodology for their organization, from randomization to quasi-experimental and beyond depending on their resources and needs. Measuring impact isn't about implementing the gold standard evaluation, but rather ensuring you collect data in rigorous way to ensure you end with robust data that can be used to assess and enhance the venture. One of the largest assessments we've conducted thus far was not randomized, but rather utilized a quasi-experimental design of those who chose and didn't choose to buy from the venture which made the most sense for them to understand their model and resources available.
If you are interested in learning more about how to measure your impact join us at our next workshop October 26-28th in Ann Arbor, MI, you can register at the following location: http://bop2009.org/ia.aspx
thanks Dennis, and we're
thanks Dennis, and we're certainly a fan of the GlobalGiving.org decentralised approach as well!
Don't throw out the lifeboats...
Hi Lee,
I must not have communicated my message very well. I am not in favor of throwing out the lifeboats - sometimes they are the right answer, just like RCTs! But, like RCTs, they are not the full answer, for both cost and other reasons. There are many other factors that go into making a sea voyage safe for the passengers.
For a broader discussion of this (with inputs from people who have thought a lot about it), see:
http://denniswhittle.blogspot.com/2010/08/scoring-points-or-making-progr...
And keep up the good work - I am a fan of IPA and the people I know there.
Dennis
You are of course right that
You are of course right that evaluation should not detract for serving people.
It SHOULD though detract from governments and NGOs thinking they are serving people when in fact they are not, or could be serving them much more effectively.
Research by IPA and others has shown how some interventions, which seem like great ideas on paper, simply don't have much impact in practice. Without evaluation we could easily continue to plough aid into useless interventions, when that money could be much better spent serving people with interventions which are PROVEN to work.
Proportional Expectations Needed for Evaluation
I've been working in monitoring & evaluation (M&E) for over ten years now and the one mantra I would continually share with implementing partners and colleagues alike is..."M&E should never detract from the work at hand, which is serving people." During this time in the development sector as a whole, I've also seen an increasing desperation to “know” what is inherently beyond logic and induction, especially to us as outsiders.
I agree entirely with Whittle's concern about the "gold standard," which is especially troubling when one is talking about grassroots-up initiatives. Imposing expectations to "try to evaluate every single intervention" on people who are in the process of organizing at the local level is most certainly a drain on their time and scarce resources. And what so many people on the ground have told me again and again is that abstract metrics don’t help them understand their relationship to improving the well-being of the people they serve. As members of the community, they read trends through what’s happening on the ground, rather than using any theory. It's time for us to recognize that one can monitor through data, but also through dialogue.
M&E implemented solely for the purpose of accountability fails to result in improved programming and, in some cases, undermines the effectiveness of the very interventions it is trying to measure. Let's always consider what is the appropriate cost and complexity needed for evaluation (especially given the size and scope of the program) and aim for proportional expectations so we ensure evaluation is a tool for learning, not policing.
"Those who work selfishly for results are miserable." ~Sri Krishna, Bhagavad Gita, The Song of God
Post new comment