Part I of a series of blogposts on Evidence-based Policymaking in the Public Sector

Evidence-based policymaking has long existed as a governance practice[i] but has gained special relevance over the past few decades. In tandem with ICT growth and increasingly freer flows of information, evidence generation, collection, and evaluation have grown more sophisticated as well. Very simply, evidence matters, in that it helps governments and policymakers make more effective decisions, defend their positions, and even earn (or lose) public trust.

Today, public policy research methods consist of practices that draw extensively from a range of disciplines — economics, sociology, anthropology, psychology, and even medicine, among other things — demonstrating the interdisciplinarity of the sciences. Its hybrid methods have allowed practitioners to examine at varying degrees of depth and breadth the entanglements between human behaviour, social structures, markets, and political systems to model and demystify the world we live in.

Of such methods, the social “experiment” — which borrows both practices as well as parlance from the world of clinical trials — has been particularly enthusiastically adopted as a tool for evidence generation and validation. Indeed, experimental practices, such as the randomised control trial (RCT), have several advantages over other non-experimental forms of policy evaluation, most significantly by addressing statistical “bias” and thus providing for more robust conclusions to be drawn from investigations of policy interventions in the real world[ii]. However, issues of quantitative rigour aside, the experimental method also lends additional tangible social value to the art and science of policy design and evaluation.

For one, it does this by allowing for a greater contextualisation of local problems simply by placing researchers in closer contact with stakeholders on the ground. Indeed, RCTs have contributed immensely to our understanding of differential impacts on, receptiveness to, and even unintended externalities associated with implementing social welfare interventions on different target demographics. This proximity, in turn, allows researchers to more closely understand causal mechanisms underlying observed trends, marking another shift in the landscape of policy design: from more quantitative, hard data-based evaluations to a broader and more holistic mixed-methods approach that incorporates valuable qualitative elements in hypothesising about and investigating social phenomena.

Experiments in the policy space were also typically, especially at the beginning, spearheaded by academics, research groups, and smaller organisations such as NGOs that were often not as constrained by political agenda or electoral promises as governments, and thus could design and roll out experiments with relatively more freedom, and generate a body of evidence that was soon recognised for its utility. These initial successes allowed RCTs to rapidly gain credence as a respectable tool for driving evidence-based policymaking. Eventually, interventions — especially those on the larger-scale — began to be implemented in collaboration with local governments as this facilitated on-ground implementation, especially when it involved securing buy-in from public sector stakeholders. Today, this is common practice and examples of it are scattered across public policy literature, ranging from the continent-spanning Deworm the World initiative[iii] to efforts such as the United Kingdom government’s Behavioural Insights Team and What Works Centres. Scores of similar initiatives across the world have shown that working with local governments or partners also serves to significantly enrich experiments with context as these partners are likely to be more familiar with local problems and thus can offer perspectives that researchers may have missed.

Further, the replicability of experimental studies that is inherent to its design — and indeed is one of the core tenets of the scientific method — encourages the development of iterative investigations that generate over time a rich body of evidence on the effects of different governance and policy decisions. For instance, practitioner knowledge on poverty alleviation owes an invaluable debt to the series of (ongoing) studies on cash transfers to pull populations out of the “poverty trap”[iv] across the Global South; these findings, which are the result of dozens of related, sequential experiments, could simply never have been uncovered by a single government or research group. Because of these studies, today, we know that conditional cash transfers generate significantly different outcomes for poor families based on who the recipient is and what their geographical context is. For instance, a two-year RCT in Burkina Faso showed that cash transfers to male heads of families lead to better child outcomes in lean years and more household investments such as housing[v], while in countries such as North Macedonia transfers to mothers lead to better outcomes such as more nutritious food being made available children[vi], or that cash transfers complemented with behavioural interventions such as providing information on finance or nutrition (e.g., in Nepal[vii] and Nigeria[viii]) tended to yield more lasting returns to investment, and that on average, such interventions can lead to education, nutrition and job market returns that can extend for decades and potentially even generations.

Another example of social experiments yielding valuable contextual information in the same field are RCTs on extending access to micro-credit institutions and instruments for the poor as a way of helping them escape the poverty trap. Here, a study of residents in rural Bangladesh led to an unexpected finding: the introduction of these institutions actually worsened outcomes for the ultra-poor by weakening existing social bonds which they would earlier utilise to smooth consumption during lean periods[ix]. This is an example of an externality that would not have been visible without experimental research, and these findings have encouraged the development of a new strand of investigations on the relationship between community networks, decision making and poverty alleviation interventions.

We could indeed be living in the golden era of the use of experimental studies in the social policy space, which have proliferated swiftly over the past three decades. Research efforts in this space have not only paid dividends in terms of concrete benefits to some of the world’s poorest and most vulnerable but are also beginning to be recognised globally and sought after as a practice in good governance. In 2019, Abhijit Banerjee, Esther Duflo, and Michael Kremer Prize in economics for using experimental methods in alleviating global poverty.

Today, more and more governments, funders, and national and multilateral organisations have been utilising experimental methods, which allow them to direct resources into efforts that yield demonstrated results, generate evidence on the efficacy of their programmes, as well as iterate upon and improve their interventions. Several governments across the Global North and South now have either their own policy evaluation teams or routinely engage consultants to drive their policy formulation efforts. There are even several kinds of relatively new funding efforts for public welfare, such as social impact bonds and private-sector impact investing, that have come to be tied to the measurement of outcomes.

Closer home, the Indian School of Business (ISB) has been an active player in conducting experimental studies in the social policy and governance sphere. While the school continues, through its various Research Centres and Initiatives (RCIs) and Research Initiatives, to work closely on issues of national importance, it has over the past few years begun to engage more closely with state and central governments to develop and conduct rigorous experimental studies in areas such as education[x], entrepreneurship[xi], [xii] and capacity building for civil servants. Watch this space for subsequent posts on how we evaluate public welfare schemes and civil service capacity building efforts across India, and how these help shape policy design and delivery in the country.