The Rise of the Social Experiment in Policymaking

Part I of a series of blogposts on Evidence-based Policymaking in the Public Sector

Evidence-based policymaking has long existed as a governance practice[i] but has gained special relevance over the past few decades. In tandem with ICT growth and increasingly freer flows of information, evidence generation, collection, and evaluation have grown more sophisticated as well. Very simply, evidence matters, in that it helps governments and policymakers make more effective decisions, defend their positions, and even earn (or lose) public trust.

Today, public policy research methods consist of practices that draw extensively from a range of disciplines — economics, sociology, anthropology, psychology, and even medicine, among other things — demonstrating the interdisciplinarity of the sciences. Its hybrid methods have allowed practitioners to examine at varying degrees of depth and breadth the entanglements between human behaviour, social structures, markets, and political systems to model and demystify the world we live in.

Of such methods, the social “experiment” — which borrows both practices as well as parlance from the world of clinical trials — has been particularly enthusiastically adopted as a tool for evidence generation and validation. Indeed, experimental practices, such as the randomised control trial (RCT), have several advantages over other non-experimental forms of policy evaluation, most significantly by addressing statistical “bias” and thus providing for more robust conclusions to be drawn from investigations of policy interventions in the real world[ii]. However, issues of quantitative rigour aside, the experimental method also lends additional tangible social value to the art and science of policy design and evaluation.

For one, it does this by allowing for a greater contextualisation of local problems simply by placing researchers in closer contact with stakeholders on the ground. Indeed, RCTs have contributed immensely to our understanding of differential impacts on, receptiveness to, and even unintended externalities associated with implementing social welfare interventions on different target demographics. This proximity, in turn, allows researchers to more closely understand causal mechanisms underlying observed trends, marking another shift in the landscape of policy design: from more quantitative, hard data-based evaluations to a broader and more holistic mixed-methods approach that incorporates valuable qualitative elements in hypothesising about and investigating social phenomena.

Experiments in the policy space were also typically, especially at the beginning, spearheaded by academics, research groups, and smaller organisations such as NGOs that were often not as constrained by political agenda or electoral promises as governments, and thus could design and roll out experiments with relatively more freedom, and generate a body of evidence that was soon recognised for its utility. These initial successes allowed RCTs to rapidly gain credence as a respectable tool for driving evidence-based policymaking. Eventually, interventions — especially those on the larger-scale — began to be implemented in collaboration with local governments as this facilitated on-ground implementation, especially when it involved securing buy-in from public sector stakeholders. Today, this is common practice and examples of it are scattered across public policy literature, ranging from the continent-spanning Deworm the World initiative[iii] to efforts such as the United Kingdom government’s Behavioural Insights Team and What Works Centres. Scores of similar initiatives across the world have shown that working with local governments or partners also serves to significantly enrich experiments with context as these partners are likely to be more familiar with local problems and thus can offer perspectives that researchers may have missed.

Further, the replicability of experimental studies that is inherent to its design — and indeed is one of the core tenets of the scientific method — encourages the development of iterative investigations that generate over time a rich body of evidence on the effects of different governance and policy decisions. For instance, practitioner knowledge on poverty alleviation owes an invaluable debt to the series of (ongoing) studies on cash transfers to pull populations out of the “poverty trap”[iv] across the Global South; these findings, which are the result of dozens of related, sequential experiments, could simply never have been uncovered by a single government or research group. Because of these studies, today, we know that conditional cash transfers generate significantly different outcomes for poor families based on who the recipient is and what their geographical context is. For instance, a two-year RCT in Burkina Faso showed that cash transfers to male heads of families lead to better child outcomes in lean years and more household investments such as housing[v], while in countries such as North Macedonia transfers to mothers lead to better outcomes such as more nutritious food being made available children[vi], or that cash transfers complemented with behavioural interventions such as providing information on finance or nutrition (e.g., in Nepal[vii] and Nigeria[viii]) tended to yield more lasting returns to investment, and that on average, such interventions can lead to education, nutrition and job market returns that can extend for decades and potentially even generations.

Another example of social experiments yielding valuable contextual information in the same field are RCTs on extending access to micro-credit institutions and instruments for the poor as a way of helping them escape the poverty trap. Here, a study of residents in rural Bangladesh led to an unexpected finding: the introduction of these institutions actually worsened outcomes for the ultra-poor by weakening existing social bonds which they would earlier utilise to smooth consumption during lean periods[ix]. This is an example of an externality that would not have been visible without experimental research, and these findings have encouraged the development of a new strand of investigations on the relationship between community networks, decision making and poverty alleviation interventions.

We could indeed be living in the golden era of the use of experimental studies in the social policy space, which have proliferated swiftly over the past three decades. Research efforts in this space have not only paid dividends in terms of concrete benefits to some of the world’s poorest and most vulnerable but are also beginning to be recognised globally and sought after as a practice in good governance. In 2019, Abhijit Banerjee, Esther Duflo, and Michael Kremer Prize in economics for using experimental methods in alleviating global poverty.

Today, more and more governments, funders, and national and multilateral organisations have been utilising experimental methods, which allow them to direct resources into efforts that yield demonstrated results, generate evidence on the efficacy of their programmes, as well as iterate upon and improve their interventions. Several governments across the Global North and South now have either their own policy evaluation teams or routinely engage consultants to drive their policy formulation efforts. There are even several kinds of relatively new funding efforts for public welfare, such as social impact bonds and private-sector impact investing, that have come to be tied to the measurement of outcomes.

Closer home, the Indian School of Business (ISB) has been an active player in conducting experimental studies in the social policy and governance sphere. While the school continues, through its various Research Centres and Initiatives (RCIs) and Research Initiatives, to work closely on issues of national importance, it has over the past few years begun to engage more closely with state and central governments to develop and conduct rigorous experimental studies in areas such as education[x], entrepreneurship[xi], [xii] and capacity building for civil servants. Watch this space for subsequent posts on how we evaluate public welfare schemes and civil service capacity building efforts across India, and how these help shape policy design and delivery in the country.

[i] Different authors identify different points in time as the beginning of evidence-based policymaking, beginning with ancient Greece where “Aristotle put forward the notion that different kinds of knowledge should inform rulemaking”. For more historical context, see Parkhurst, J. (2017). The politics of evidence: from evidence-based policy to the good governance of evidence (p. 182). Taylor & Francis.

[ii] For more details, please see Sherman, L. W. (2003). Misleading evidence and evidence-led policy: Making social science more experimental. The Annals of the American Academy of Political and Social Science, 589(1), 6-19.

[iii] https://www.evidenceaction.org/dewormtheworld/

[iv] “Poverty traps” are recognised in economic literature as self-reinforcing mechanisms due to which current poverty is a direct cause of future poverty. For more details, see Kraay, A., & McKenzie, D. (2014). Do poverty traps exist? Assessing the evidence. Journal of Economic Perspectives, 28(3), 127-48.

[v] Akresh, R., De Walque, D., & Kazianga, H. (2016). Evidence from a randomised evaluation of the household welfare impacts of conditional and unconditional cash transfers given to mothers or fathers. World Bank Policy Research Working Paper, (7730).

[vi] Armand, A., Attanasio, O., Carneiro, P., & Lechene, V. (2020). The effect of gender-targeted conditional cash transfers on household expenditures: Evidence from a randomised experiment. The Economic Journal, 130(631), 1875-1897.

[vii] Levere, M., Acharya, G., & Bharadwaj, P. (2016). The role of information and cash transfers on early childhood development: evidence from Nepal (No. w22640). National Bureau of Economic Research.

[viii] Carneiro, P., Kraftman, L., Mason, G., Moore, L., Rasul, I., & Scott, M. (2021). The impacts of a multifaceted prenatal intervention on human capital accumulation in early life. American Economic Review, 111(8), 2506-49.

[ix] Banerjee, S. B., & Jackson, L. (2017). Microfinance and the business of poverty reduction: Critical perspectives from rural Bangladesh. Human relations, 70(1), 63-91.

[x] Impacts of Technology on Teaching and Learning Efficacy in Telangana Schools

[xi] Impact Assessment of Common Service Centres 2.0 Scheme

[xii] Impact Assessment of Udyam Abhilasha (UA): Entrepreneurship Awareness Campaign

Image Credit: Frédéric Barriol