Agriculture / East Africa / Impact Assessment / Report / Research

Millennium Villages Project: Success? Failure? Unknown?–The controversy continues

Young Boys Posing - Malawi

Young boys in Malawi on a break from class pose for the camera at The United Nations Millennium Villages Project (photo on Flickr by whl.travel).

The Economist has started an interesting new blog, ‘Feast and famine: Demography and development’. On this blog, the magazine’s correspondents report on and analyse matters relating to demography and development, including food production, public health and other factors that determine the wealth and poverty of nations.

The current blog post argues that the jury is still out on whether the Millennium Villages Project has lifted half a million people in 14 villages out of poverty.

‘For something designed to improve lives in some of the poorest parts of the world, the Millennium Villages Project certainly stirs up a lot of bad blood. The project, the brain child of Jeffrey Sachs of Columbia University in New York, takes 14 “villages” (mostly small areas) with around 500,000 people, and scales up aid to them in the hope of springing the poverty trap in which they are caught. Late in 2011, there was a flurry of accusation and rebuttal at the time of the first independent evaluation of one of the villages, Sauri in Kenya, which challenged some of the claims made on behalf of the villages. The Economist reviewed the dispute here and Mr Sachs criticised our account. Now debate has erupted again, producing yet another round of criticism online, as well as duelling editorials in two leading British scientific journals . . . .

‘This time, the Millennium Villages project, responding to criticism from people at the World Bank and elsewhere that there were no “control villages” with which to compare the 14 in the project, has published a detailed account of how nine millennium villages and nine comparisons sites have fared over the past three years, judged by 18 indicators, ranging from child mortality and maternal health to measles immunisation and the use of anti-malaria bednets. The study appears in the Lancet, a British journal. . . .

‘Gabriel Demombynes . . . argues that the Lancet study overstates the annual fall in child mortality by using what he thinks are misleading periods for calculation. He argues that the fall in child mortality should be calculated over a slightly longer period, so the annual fall works out at 5.9%, not 7.8%, as in the Lancet. And he uses figures from his own study for comparison. These show that the countries where the millennium villages are experienced average annual falls in child mortality of 6.5%. In other words, on his calculations, the fall in child mortality in the villages was slightly less than the average for the region as a whole, instead of much greater.

‘The Economist concluded its previous article by saying that the evidence does not yet support the claim that the millennium villages project is making a decisive impact. That still seems about right.’ (Read the whole article: Jeffrey Sachs and the millennium villages: Millennium bugs, 14 May 2012,

Here are excerpts from some previous posts:

‘For the past couple of years, nutrition has become the most important lens for looking at poverty reduction. This article examines some of the reasons for that. It argues that the focus of the 1960s and 1970s on growing more staple foods (with aid to offset shortfalls) came unstuck with the Ethiopian famine of 1984. Attention then switched to targeting economic growth and the income of the poorest, but this was found wanting when the commodity-price spikes of 2007-08 and 2010-11 came along. So nutrition came to be seen as a more rounded way of judging whether the lives of the poor are really getting better. A report by the IMF and World Bank casts new light on why nutrition matters. It points out that countries around the world have done a terrible job of improving nutrition. . . .’ Development: Why nutrition matters, 24 Apr 2012

Regarding ‘People and the Planet’, a new report from Britain’s Royal Society, the blog says, ‘. . . In general, the report is weak on the trade-offs between economic growth and pollution. It is extremely desirable that the poorest people in the world should become less poor. But it is practically unavoidable that as they do so, pollution will increase. The question is by how much. At the moment, the average African produces less than one tonne of CO2 equivalent each year; the average American produces more than ten times as much. A report by Britain’s finest scientific minds explaining how the poorest could rise towards American standards of living without also rising towards American standards of pollution would have been extremely valuable. Alas, this is not that report. Population and growth: But on the whole it stinks, 30 Apr 2012

One thought on “Millennium Villages Project: Success? Failure? Unknown?–The controversy continues

  1. Those interested in the Millennium Villages Project and the controversy about how it is going about defining its success should read the following article in Retraction Watch Blog about the project’s correction of a heavily criticized article it recently published (8 May 2012) in the Lancet.

    Responding to the criticism, Jeffrey Sachs, director of the Millennium Villages Project, says the team’s oversight will be overhauled: ‘The criticisms from Demombynes and Clemens regarding this timing issue have been on point and helpful. While the progress in the Millennium Villages has been notable, an accurate comparison with local, regional, and national trends in comparable periods is essential. In response to the valuable criticisms, and more generally in order to strengthen the project, I am leading an overhaul of the research organization of the project, including the creation of an independent expert group chaired by Prof. Robert Black (Chair of International Health at Hopkins) to scrutinize, assess, and help to improve the data collection, processing, and analysis. Dr. Paul Pronyk has left the project, and I will co-chair a new faculty research committee with Dr. Cheryl Palm. This new faculty research committee will be the counterpart of the independent expert group.’

    Find the whole (interesting) article, and 3 comments, in Retraction Watch–
    Millennium Villages Project forced to correct Lancet paper on foreign aid as leader leaves team—here: http://retractionwatch.wordpress.com/2012/05/31/millennium-villages-project-forced-to-correct-lancet-paper-on-foreign-aid-as-leader-leaves-team/

    More excerpts from the Retraction Watch Blog article:
    ‘A senior member of a high-profile foreign aid research team has left the project on the heels of a Lancet correction of a heavily criticized paper the team published earlier this month.

    ‘Paul Pronyk, who until last week was director of monitoring and evaluation at Columbia University’s Center for Global Health and Economic Development, which runs the Millennium Villages Project, wrote a letter to the Lancet acknowledging errors in the paper, “The effect of an integrated multisector model for achieving the Millennium Development Goals and improving child survival in rural sub-Saharan Africa: a non-randomised controlled assessment,” originally published May 8. That admission came after Jesse Bump, Michael Clemens, Gabriel Demombynes, and Lawrence Haddad wrote a letter criticizing the work, which was published this week accompanied by corrections to the paper. . . .

    ‘This is not the first time the group’s work has been questioned. Two of the authors of the Lancet critique also wrote a letter to the American Journal of Clinical Nutrition to criticize a paper published last year in that journal. The issues were similar, but what really caught our eye in the letter was what the authors said was a lack of transparency on the part of the Millennium authors:

    ‘[T]he article’s calculations—arrived at through multiple layers of reweighting and matching—cannot be independently checked or replicated by other researchers because the data are strictly internal to the project. Authors of the study informed us that the data will be unavailable to outside researchers until several years in the future, even though results based on those data have already been published in this and other peer-reviewed journals.

    ‘The group responded to the criticisms by doing another analysis of their findings, but not issuing a correction. . . .

    Written by ivanoransky
    May 31, 2012 at 1:20 pm
    3 Responses

    ‘It’s difficult to judge solely from this report, but it seems Sachs has manned up and confronted the problems. His lack of weasel-words and clear statement acknowledging the poor analysis, and even better his (unintentional) lack of oversight, are most welcome. Not being a public health scientist, I can’t judge about the data privacy issue, I hope others can.’
    D G Rossiter
    June 1, 2012 at 2:04 am

    Reply
    ‘Not unusual. If you have a large dataset and you plan more mining on it, you keep it for yourself.’
    Jon Beckmann
    June 1, 2012 at 6:07 am

    Reply
    ‘Perhaps not unusual, but certainly borderline amoral, as they are receiving donations from large numbers of corporations and charities, including UN organizations which are receiving tax payers’ money through member state contributions. http://www.millenniumvillages.org/uploads/partners/MP-Partners-List-for-Website—15Aug2011.pdf . With a setup like this it could very well be argued that the data has to be made publicly available.

    ‘I personally like the approach of large consortia like ENCODE which make their data available for public use pretty much as soon as it has been generated and pre-processed, and ask to not perform large scale data analysis. This allows the community to use the data (and publish results generated from it) in context of narrowly defined research questions, and gives the consortia time to prepare their “big” papers.’
    Sebastian
    June 1, 2012 at 9:48 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s