From the IPA Blog

Cochrane’s Incomplete and Misleading Summary of the Evidence on Deworming

Jul 20/12 | From the blog
 
Summary: The Cochrane Collaboration’s recent summary of the evidence on treating school-age children for soil-transmitted intestinal worms (or STH) is incomplete and misleading. While we do not comment on the evidence of the health and cognitive outcomes reviewed, we continue to find that the educational benefits alone justify mass school-based deworming. We strongly endorse the WHO and Copenhagen Consensus’s recommendation to mass treat children for STH.  
 

The Cochrane Collaboration recently updated their review of the evidence on treating school-age children for soil-transmitted helminths (intestinal worms which infect nearly a quarter of the world’s population). While many of the methodological flaws in the 2007 review have been addressed, the new review is still incomplete and provides a misleading summary of the evidence on mass deworming. Deworm the World, Innovations for Poverty Action (IPA), the Center for Effective Global Action (CEGA) and the Abdul Latif Jameel Poverty Action Lab (J-PAL) all encourage countries with moderate to high worm loads to mass treat for STH because it is one of the most cost-effective ways of increasing school attendance. These results remain unchanged and more recent studies (see below) have only strengthened the case. We have not yet carefully reviewed Cochrane’s summary of the evidence on height, weight, anemia, and cognitive outcomes (although others have commented), but the educational benefits alone are sufficient to justify mass school-based deworming.
 
Problems with the Cochrane review: While claiming to cover published and unpublished studies the review excludes three important randomized and quasi-randomized studies on STH which all show positive results. Bleakley (2004) shows that an early 20th century campaign to eradicate hookworm in the US south improved school attendance, literacy, and income in adulthood. Ozier (unpublished) shows that children under one year of age (and therefore too young to be treated) at the time of mass deworming in their communities in Kenya had significantly improved cognitive outcomes due to spillover effects.  Baird et al (unpublished) shows strong long term labor market outcomes of children treated for STH.
 
Miguel and Kremer (2004) is the only study quoted in the main results section of the Cochrane review with outcomes on student attendance (the other quoted studies are not clustered and therefore appropriately ignored). Yet the review authors down-weight the Miguel and Kremer results on student attendance for a number of reasons. For example, that there is “high attrition” in hemoglobin (Hb) data, even though there is zero attrition for the actual student attendance results!  (The reason for the smaller Hb sample is that it was only collected for a random subsample, information that is readily available from the authors).
 
The Cochrane authors also criticize Miguel and Kremer (2004) for a lack of baseline (pre-treatment) data on school attendance. This is a strange point for at least four reasons. First, given the experimental design there is no systematic reason to expect differences in school attendance between the treatment and control groups. Second, Miguel and Kremer (2004) show that there is baseline balance along a wide range of other characteristics in the treatment and control groups (in Table 1), many of which correlate strongly with school attendance (i.e., family socioeconomic status). But most importantly, there is in fact baseline “balance” between Group 2 and Group 3 in the high-quality school attendance data in 1998, when both were still “control” before Group 2 was phased into deworming (see Table 8 of the Miguel and Kremer 2004 paper for this data). The Cochrane authors apparently missed this in their read of the paper. Finally, the Cochrane authors strangely “slice” up the data into impact in the first year after deworming versus the second year, and then across different “comparisons” of the treatment groups. Measuring impacts in small subsamples of the data inevitably reduces the precision of the estimate, making an effect that is strongly significant in the full sample look insignificant in small slices.
 
Missing the point about mass deworming. In their conclusion, the review’s authors state that “Screening school children for intestinal helminths and then treating those infected probably has some value…[but] there is insufficient evidence to recommend deworming drugs in targeted community programmes.” This argument overlooks the fact that if a program is effective for individuals with worms, it will still affect the same individuals (presumably with the same effect) if they are reached through a mass treatment campaign. The only reasons to prefer a screening approach is if deworming drugs had negative effects on uninfected children (they do not), or if the costs of treating uninfected children in a mass campaign were greater than the costs of individually testing children to determine whether they required treatment (in fact it is much cheaper to mass treat than to diagnose and treat).
 
School attendance is an important goal in its own right but it is also a step towards longer term outcomes. Preliminary results from Baird et al suggest that mass treatment for STH leads to longer hours worked and higher earnings 10 years after treatment—results remarkably similar to those found in the US South. We therefore strongly endorse the WHO and Copenhagen Consensus’s recommendation to mass treat children for STH.
 
Annie Duflo, Executive Director, IPA
Alissa Fishbane, Managing Director, Deworm the World Initiative, IPA
Rachel Glennerster, Executive Director, J-PAL Global
Michael Kremer, Gates Professor of Developing Societies, Department of Economics, Harvard
Temina Madon, Executive Director, CEGA
Edward Miguel, Professor of Economics, UC Berkeley

Comments

Deworming as one of the most effective education interventions?

 

Given the new evidence: does this mean that deworming is or is not one of the most cost effective education interventions? 
Even if, as he Cochrane reply says, deworming does not have a significant effect on school attendance the important question how does it compare to other education interventions (which are also so often unsuccessful)? 
The Copenhagen Consensus make a good argument for demand side interventions that encourage people to stay in school, like deworming. 

Any analysis of this would be really interesting for anyone who wanted to decrease absenteeism.

 

Some other thoughts upon reading the above article:
- Reading the blog by the IPA, they clearly makes an error of assuming that just because some studies show the positive effect of dewoming is good evidence for these positive effects. They do not discuss the statistical relevance of these positive studies when compared to other studies.
- However the Cochrane reply does not address the following the issue that it may be the case that sometimes deworming improves schooling and other times it does not, so an analysis of all the evidence may show insignificant evidence that deworming increases schooling, yet this could be the wrong conclusion to reach.
- Givewell seem stronger worded against deworming than the above post from the IPA. They also do not care much talk about education effects but concentrate on long term health benefits
- I have found this to be one of the most interesting and discussions on these topics I have read in a while.
- My own view is that the power of education interventions are overestimated. For example the 20% increased wages in later life stat the IPA quote is highly misleading, and likely mainly due to people who have gone through education having a competitive advantage (see: http://www.givingwhatwecan.org/where-to-give/charity-evaluation/education )

Cochrane’s independent appraisal of reliable evidence

 

We thank the authors for their response to the updated Cochrane review. We note their concerns about our analysis and we respond below. However, statistical intricacies related to one outcome from one trial need first to be set in context.  As Cochrane authors our motivation lies solely in providing an independent appraisal of the available evidence about the effects of health treatments, policies and public health strategies. The current deworming strategy, as promoted by the WHO and multiple advocacy groups, is widely claimed to be the ‘best spend’ to improve school health, school performance, increase productivity and reduce poverty.  Our review systematically identifies, appraises and summarizes the available evidence from randomized controlled trials to support or refute these claims. We seek evidence of these desired impacts on child mortality and school performance; and evidence on important outcomes required to achieve these desired impacts, including nutritional indicators, haemoglobin, cognition and school attendance.

Our review shows that the evidence on desired impact (child mortality and school performance) is absent. The evidence on the important outcomes is not strong, being largely confined to evidence of improved weight gain in three trials conducted over 15 years ago. Two of these trials were carried out in a specific setting in Kenya by the same team (Stephenson et al) in an area with very high worm loads - probably, we would surmise, far higher than in most settings where programmes are promoted today. The third trial that showed important weight gain was from urban India. Subsequently conducted trials - some of them very big - have largely either failed to demonstrate benefits, or have not been published. The latter includes a trial of one million children completed in 2006. Importantly, this trial, for which no data are currently available, was carried out at the same site in India as the earlier study showing an effect on weight gain.

To recap the findings of the Cochrane review: apart from a mixed effect on weight, with three older studies showing effects, our analysis of the global evidence base shows there is fairly good evidence of no impact on haemoglobin; no reliable evidence on cognition or school performance; and the evidence for school attendance is limited.

We turn now to the criticisms in the blog:

  • The comment that some trials are missing. Our analysis was limited to randomized controlled trials comparing mass administration with placebo or no treatment. The three additional studies cited in the blog do not meet these criteria and so were appropriately excluded.
  • The quality of the evidence for school attendance was low. This was reached using standard GRADE criteria, including a high risk of selection bias (due to inadequate methods of generating the random sequence and concealing treatment allocation, and the lack of baseline data), and the statistical uncertainty given that the result did not reach standard levels of statistical significance (95%).
  • In relation to baseline values for school attendance in the Kenya study, the reporting of relevant baseline data is standard practice for randomized trials, and a requirement by the CONSORT Group.  Small differences in baseline values can cause spurious conclusions with small numbers of units of randomization if only end values are presented rather than change values. It remains possible that the differences in end values reported are the result of modest baseline differences.
  • The blog comments that we split data across years and hence reduce power.  We did indeed present data before one year and after one year to allow us to use as much data as possible in these time periods, since some trials present data at multiple follow up periods, and to help detect any externalities of the intervention in trials with longer follow-up. A combined analysis across the two time periods does not substantively change the results or conclusions.
  • Children known to be infected with worms are usually treated - this is what happens in clinical practice or in screening programmes. We personally would prefer that children were not infected with worms as they are unpleasant. But the policy under debate here is about the effects of mass deworming programmes and whether these demonstrate an effect at the population level. Our review collates the best available evidence on this, as we have summarized above.

Evidence-informed health policy, that we have been part of over the last 15 years, means we need to consider the global evidence base, not make policies based on individual trials or studies. Our review is complete in this regard, apart from any studies that remain unpublished and outside the public domain.  The review raises questions about the generalizability of the results on school attendance from this single trial by Miguel and Kremer, given the lack of nutritional and cognitive effects in our systematic review of all reliable studies.  Inevitably this also leads to questions about the plausibility that worms are a major and widespread cause of absenteeism at school, and whether mass deworming programmes have the kind of impact on school performance and development that is claimed by the advocates.

David Taylor Robinson, Sarah Donegan, David Sinclair, Paul Garner (30th July 2012)

Reviewing the reviews

I assume you are aware of Givewell's remarks on the Cochrane review (essentially to the contrary of IPA's take), posted here:

http://blog.givewell.org/2012/07/13/new-cochrane-review-of-the-effective...

Post new comment

The content of this field is kept private and will not be shown publicly.
Type the characters you see in this picture. (verify using audio)
Type the characters you see in the picture above; if you can't read them, submit the form and a new image will be generated. Not case sensitive.
*Required Field
Empowered by:
Copyright 2014 Innovations for Poverty Action. All rights reserved.