90 % des RCTs réalisés en Inde sont conçus par des chercheurs non indiens ou indiens vivant hors du pays.

© IRD - Marianne Donnat

Randomized trials in development economics: revolution or illusion?

Summary

I like

0

Subject

In October 2019, the Economics equivalent of the Nobel Prizeofficially the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel was awarded to Esther Duflo, Abijit Banerjee and Miguel Kremer from the Poverty Action Lab at the Massachusetts Institute of Technology, for their work adapting the methods of randomized controlled trials used in medical and clinical contexts for projects in the area of development. The aim is to assess the impact of a project, policy or program by comparing it with a “control” group. Here, three experts give us their perspective on randomized controlled trials, or RCTs, applied to the field of development policy: Jonathan Morduch, development economist at New York University who has worked on several of these RCTs, Gulzar Natarajan, economist at the Global Innovation Fund and current head of the Finance Department of Andhra Pradesh in India, and François Roubaud, economist, statistician at IRD, and co-editor of “Randomized Control Trials in the Field of Development - A Critical Perspective” (Oxford University Press), to which many specialists in this area contributed, including Jonathan Morduch and two winners of the Nobel Prize for Economics, Angus Deaton and James Heckman.

The speakers

  • Jonathan J. Morduch

    © NYU Robert F. Wagner Graduate School of Public Service

    '' Complementary approaches, such as ethnography or qualitative evaluation, are often needed to properly interpret the results of RCTs ''

    Jonathan Morduch

    Development economist at New York University

  • Portrait de Gulzar Natarajan

    Gulzar Natarajan

    © AIDF - Indian bureaucracy

    '' It is very naive to imagine that new policies can be designed based on the results of these trials ''

    Gulzar Natarajan

    Economist at the Global Innovation Fund, and current head of the Finance Department of Andhra Pradesh in India

  • François Roubaud

    © IRD - Amélie Debray

    '' Results too singular to be transposed to a larger community ''

    François Roubaud

    Economist, statistician at IRD, and co-editor of "Randomized Control Trials in the Field of Development - A Critical Perspective"

The answer of Jonathan Morduch

Development economists use RCTs in two different ways: first, to measure the impacts of a project (implemented in the field by funding bodies, NGOs, etc.) and secondly, to run experiments in real-life contexts (by artificially manipulating a parameter to thereby deduce “laws”).  I worked with this approach for the first time in 2007, in order to assess the impact of a project aimed at “ultra-poor” people in southern India. A local NGO had provided households with a small amount of money, a training course and social support. This experiment showed us that the people who had these assets improved their situation. But we also established that the control group did better too, thanks to a rise in wages on the labor market. In the end, both groups made similar progress and therefore the overall impact of the project was zero. This result was a great surprise to the NGO - and to us as well. I came to the conclusion that the power of RCTs does not solely reside in the fact that this tool allows us to establish whether something works – or not – it also provides an experimental structure to try something different in a systematic way. In this case, to run another experiment with another asset transfer programme. This shift towards experimentation constitutes a genuine change in approach, making it possible to create variations in an environment, and offering new insight into microfinance contracts and financial behavior. Of course, the tool is not always appropriate: it is highly limited for understanding macroeconomics or systemic corruption, for example. Furthermore, critics are concerned that “randomistas”, or advocates of RCTs, do not pay enough attention to context. My own experience has shown me that complementary approaches, such as ethnography or qualitative evaluation, are often required to interpret RCT results correctly. What might be yet more important is that the question of whether a project is successful or not is not necessarily the most interesting one, in the end. RCTs, along with the experimental mindset that they require, can be tools that make it possible to ask different kinds of questions: “How do people make decisions? Can a change in design make a difference? Who wins and who loses, and why?” Debates around RCTs are often too focused on how they determine if something works, rather than the power of these methods to spark new questions. 

 

The answer of Gulzar Natarajan

RCTs in economics arrived in India in the early 2000s, in experiments aiming to assess the effectiveness of microfinance to eliminate poverty. After years of observing the development of these approaches in my country, I believe that they are seen by their advocates as far more powerful that they actually are. It is highly naive to imagine that new policies can be created based on the results of these trials, rather than on knowledge built up over decades of prior work! 
However, RCTs can be suited to certain areas: comparative trials have established that children that receive worming tablets are less sick, spend more time at school and learn better than others. But no matter how cheap and effective this medication is, it can never make up for the lack of schools and textbooks! Effectively, RCTs display key limitations when it comes to creating actual development policy. Sometimes, they can even have a negative impact. Such was the case for “Aadhaar”, the individual ID number system adopted by India, which was endorsed by RCTs and supposed to help direct welfare benefits more effectively toward the most poverty-stricken. However, due to problems regarding access, Internet connection and technological limitations, the system meant that the most disadvantaged populations found themselves denied access to food benefits, welfare payments and other social services. 
Another problem comes from the fact that over 90% of RCTs performed in India are designed by non-Indian or expat Indian researchers and financed mainly by foreign funding bodies. This means that practically none of these experiments are guided by priorities raised or needs felt by Indian decision-makers. Beyond the size and diversity of the Indian context, RCT advocates and funding bodies are also attracted by the lack of regulation surrounding experiments in this country. However, as these are often small, fringe experiments, their conclusions rarely draw the attention of the public beyond the scientific community, and do not affect the creation of public policy in India, a country that is not reliant to any great extant on development aid. In smaller countries, however, where public and private international funding bodies have a greater influence, RCTs could have a much bigger impact and shape new policies. From an ethical point of view, that is very worrying! 

 

The answer of François Roubaud

In theory, RCTs have an undeniable advantage when it comes to assessing the impact of a project, policy or program. But their use in practice has revealed certain limitations, particularly the difficulties of respecting the sampling protocol. This means it is no longer randomized, or is focused on groups that are too specific: so how can these overly specific results be used and applied on the scale of a larger community? One of the aims for studies carried out in the field of development is to understand the processes shaping the societies in question. However, while the RCT method measures the impact of something (e.g. personalized support for unemployed people to find a job), it does not explain the factors at play (more information and self-esteem, reduced pressure, etc.), thereby limiting the possibilities for transposing these results to other contexts. Even the impact evaluation itself is subject to criticism: rather than measuring the impact of a microcredit, for example, RCTs actually test how a population takes up the microcredit according to various variables, which then enables them to “sell” or distribute a product or service better, with the trial operator imagining that it has a positive social impact. More broadly, RCTs are only able to evaluate micro-interventions, but cannot address major fiscal, commercial or sector-specific policies affecting economic structures. This means that, on the one hand, only a very small proportion of public policies are likely to be evaluated by this method, and, on the other, RCTs are based on a very narrow conception of development. This also raises the question of compliance with ethical obligations, which is important for all research: by manipulating the environment in question (people’s lives) and taking an interest in particularly vulnerable communities, is the need for protection of these groups really respected? When, for an RCT, microcredits are awarded to people who are considered insolvent without considering the risks of excess debt, or water is cut off for renters in a slum to see if it spurs the owners to pay the water bill, it would appear that faith in the progress of science and the promise of advances in the future is overriding any ethical concerns. The Covid-19 pandemic has also shone a harsh light on the limitations on this method. Although RCTs have been recognized by the Nobel Prize for their key contribution to reducing poverty, they have not been able to provide concrete solutions to fighting the catastrophic impact of this crisis, which represents the most brutal hit to poverty ever experienced. Here, we can see the vast distance that separates RCTs used in the field of development from the clinical trials used in medicine, to test vaccines for example. However, things are shifting among RCT advocates, who are now proposing ethical charters that recognize the need for different methods. I am hopeful that the work edited by Florent Bédécarrats, Isabelle Guérin and myself has played a role in these adjustments, but the question of whether they are real or merely superficial remains unanswered for the moment. 


  • CONTACTS

Jonathan Morduch, Wagner Graduate School of Public Service, New York University

Gulzar Natarajan, Global Innovation Fund

François Roubaud, LEDA - DIAL (IRD/CNRS/Université Paris-Daupnine)

 

  • PUBLICATION

Randomized Control Trials in the Field of Development: A Critical Perspective, Florent Bédécarrats, Isabelle Guérin & François Roubaud, Oxford University Press, 2020
DOI:10.1093/oso/9780198865360.001.0001

 

  • JOURNALIST

Alice Bomboy

 


Ler o artigo em portuguêsExperimentos Aleatórios Controlados na Área de Desenvolvimento: revolução ou miragem?