From the IPA Blog

RCTs: The Backlash

May 31/11 | From the blog
by Lee Crawfurd

The media attention surrounding the release of More Than Good Intentions and Poor Economics has led to a few criticisms of RCTs cropping up around the interwebs:

Arvind Subramanian worries about the macroeconomic effects of aid, something not addressed by RCTs.

If Chris Blattman was picking research like stocks, he would be shorting evaluations.

Martin Ravallion asks - Are we really assessing development impact? (and David McKenzie and Berk Özler respond).

Marc Bellemare thinks that the most valuable criticism of RCTs is in the questions that they cannot answer.

Edward Carr, Prof of Geography, argues for the need for greater integration with qualitative work.  Part One and Part Two

David Week asks for the evidence that RCTs have been effective in driving development.

Jennifer Lentfer talks about the need to listen to what the poor want. (which Dean Karlan and Jacob Appel discuss in Foreign Policy)

And finally, there is a spoof post on the ever-entertaining "Stuff Expat Aid Workers Like."

Whats that, you want a handy 5-bullet-point definitive rebuttal list? Sure thing;

1. RCTs can't answer every question. Nobody ever said that. 

2. Having said that, there are probably more questions that they can answer than you originally thought. 

3. Of the set of questions that they can answer, they are probably the best way of answering them. This shouldn't be too controversial a claim. 

4. Billions of dollars, and probably the vast majority of aid spending, still goes on projects that have not been rigorously evaluated. So we aren't quite yet in the position to be worried about over-kill. 

5. Yes we should be careful about extrapolation and generalizing. But this also applies to other sources of evidence, and is not an argument against relying on evidence to the fullest extent possible, or generating as much new evidence as is cost-effective.

 

Update:

Tom Murphy just sent me remarks on the "evaluation fad" by Jonathan C. Lewis at the Microfinance USA Conference

Dennis Whittle says we need to "make the beneficiary king". 

Sasha Dichter laments extrapolation from too few studies.

Ian Thorpe suggests that RCTs are going through a familiar "hype cycle."

Do add any other links we may have missed here.

Comments

ONE, Governance and RCTs - beyond vaccines?

Here is my review of Karlan/Appel and Banerjee/Duflo.

http://one.org/blog/2011/06/13/bucks-bangs-and-governance-two-must-read-books-on-aid/

Personally, while I very much agree with Lee that not enough is known about what works (in what context) in terms of development interventions, I remain somewhat sceptical about the potential of RCTs to provide explanatory answers to complex questions. That might of course mean that we should be asking simpler questions, but there's more to development than vaccines and de-worming.

For an organisation like ONE - where I lead on governance - the books provide very useful food for thought. I look forward to seeing how J-PAL and IPA's work on governance develops.

Oh, and here's another

http://patrickmoriarty.org/2011/05/31/it%E2%80%99s-the-political-economy-stu/

Good line (not mine): "did we really need to link de-worming to education outcomes in an RCT to know that worms are a) bad for us and b) cheap and easy to treat?"

One more

I think this one should also be inlcuded in this list, not directly related but definetly a useful read and comments. http://blogs.worldbank.org/impactevaluations/a-rant-on-the-external-vali...

Post new comment

The content of this field is kept private and will not be shown publicly.
Type the characters you see in this picture. (verify using audio)
Type the characters you see in the picture above; if you can't read them, submit the form and a new image will be generated. Not case sensitive.
*Required Field
Empowered by:
Copyright © 2011 Innovations for Poverty Action. All rights reserved.