Inside the machine

Adam Coutts highlights worrying fault lines in the relationship between academics and government

How can academic research better influence government policy?

Each year UK Research and Innovation (UKRI), the national funding agency for scientific research, directs billions of pounds towards university research. Yet the relationship between academic researchers and government policy practitioners is increasingly one of mutual frustration. The formal structures for cooperation are weak and the practical barriers substantial.

Academics rarely spend extended periods inside government. Only around 110 UKRI policy fellows have been placed since the scheme began in 2021, against a UK academic workforce of over 220,000.¹ Effectiveness is even harder to establish. A systematic review identified 1,923 research-policy engagement initiatives worldwide but found that only 6% had any publicly available evaluation of their impact.² We have built plenty of infrastructure for connecting research to policy. But do we know whether any of it works?

This article draws on a four-year research secondment to the Work and Health Unit at the Department for Work and Pensions (DWP), where I helped design and evaluate one of the UK’s largest social policy randomised controlled trials, and on conversations with serving and former officials, politicians and colleagues across the academic and policy worlds. My argument is simple. The fix is not more evidence, and it is not another scheme for exchanging it. It is to put independent researchers inside government on sustained terms, working with officials on the same interventions and testing what actually works, rewarding the people who do that work and building the standing capacity that lets institutional memory survive between projects.

The e-lephant in the room is the role of artificial intelligence. Government is pouring money into it, and if knowledge leaves when a researcher leaves, why not store it in a system that never resigns? Maybe civil servants do not need evidence reviews and what works centres anymore when Claude can do it in three hours? It is a fair question, and I will come back to it. The short answer is that an LLM model can hold evidence but not judgment and knowledge of local conditions and networks. Does an LLM know what young Gary trying to get a job is doing in Barnsley and his struggles? No. Judgment and empathy are what government is short of.

What I found inside 

I came to this trial from the outside. I was an early career researcher at Cambridge. I had come across JOBS II, a group work programme for unemployed people, through my own research and PhD work ten years earlier.³ Once I saw that DWP was running it, I started contacting people there to find a way in. It took me over a year to reach the official who invited me to help design the trial; three other DWP staff had already told me it was not possible. In fact, one even said the research was not needed given they were already achieving good outcomes on existing employability programmes. This is not unusual. 

Academic researchers rarely know who to approach or how independent research fits into the policy process. Analysts inside government are not findable online, and their roles are opaque to outsiders.

In the four years I was inside the department, six secretaries of state came and went. The trial outlasted all of them.

How decisions really get made 

The public story is familiar: objective officials supply the evidence, ministers take the political decisions. Former senior officials describe something different. They are not simply passing findings to ministers but constructing an interpretation and coherent story of what is realistic at that time. They draw on political intelligence, operational knowledge, institutional memory and whatever evidence they can find. What reaches the minister is not the academic evidence published in a journal but an official’s judgement of what it means politically and what may work for the media, for the public and on-the-ground at that time.

A study of more than a hundred junior officials found that, rather than following any template or method, they relied on what the researchers called “improvised expertise”. This involves collating bits of knowledge from various sources and reacting to private office demands and media coverage.⁴ Senior officials I spoke to recognised the pattern: decision-making was ad hoc, different teams working in fundamentally different ways. This is not an argument against official judgement. It is an argument for giving officials better material and information with which to judge and assess. The real question is not whether policymakers have read the relevant research, but whether the frameworks they use to interpret it are any good. As David Halpern has observed, the more familiar a problem feels, the more certain officials become that they understand it, even when that certainty rests on very little.⁵ An embedded researcher earns their place here: not by supplying more evidence but by improving the judgements that decide how it gets used.

At DWP the problem was never a shortage of evidence but the lack of a shared way to connect employment support to mental health outcomes. No amount of extra research would have supplied that; sustained presence inside the institution did, working with analysts to reframe the question in terms they could act on. The trial happened only because of deliberate design: external funding for an embedded researcher, the cross-departmental Work and Health Unit and the compliance systems and people that kept the randomised design intact.

Whether evidence shapes a decision also depends on timing, political climate and who is in the room. At DWP the employment teams emphasised the employment results and the psychologists the wellbeing ones, from a single trial that produced both. An embedded researcher sees this and can push for a fuller reading; someone publishing from outside cannot.

Resistance and trust

The cultural resistance runs deep on the academic side too. When I began the secondment, several colleagues who work on welfare told me I had gone over to the dark side. The remark is revealing and disappointing. It reflects a view, common in parts of the social sciences, that working with government on employment policy is not a way to improve people’s lives but a form of complicity with a system that harms them. The incentive structures reinforce the distance. I have been told more than once, applying for tenured academic posts in the UK, that what is required is five first-authored journal articles. Four years inside government designing and evaluating a major trial counted for little.

Trust sits underneath all this. Officials need to know that an academic collaborator will deliver sober, rigorous analysis rather than use the platform to make a political argument. Too many, in officials’ experience, treat engagement as a chance to denounce the whole framework as a product of austerity or neoliberalism rather than to improve a programme. This is not a caricature. It is a real barrier, and it pushes officials towards a handful of trusted individuals rather than opening engagement more widely.

A lost tradition

This weakness was not always the norm. Between the late 1990s and the early 2010s government commissioned large-scale social policy evaluations that brought academics into sustained, structured contact with delivery on a scale not seen since. The New Deal for Communities (1998 to 2010) was a £2 billion programme in 39 of England’s most deprived neighbourhoods; its national evaluation, led by Sheffield Hallam University, ran for nearly a decade across a consortium of fourteen research organisations.⁶ The National Evaluation of Sure Start (2001 to 2012), led by Professor Edward Melhuish at Birkbeck, ran five interlocking research modules over eleven years.⁷ The Employment Retention and Advancement demonstration (2003 to 2007) was, at the time, the largest randomised controlled trial of a social policy ever conducted in this country, explicitly to build UK capacity for experimental evaluation.⁸

These programmes shared features largely absent today: long time horizons, serious investment in evaluation, multidisciplinary teams with sustained access to administrative data and a genuine appetite for learning. They were also an unmatched training ground. Much of that has been lost: the programmes ended, the teams dispersed, and under austerity the appetite for large social experiments faded.

Where we are now

Where evaluations of current engagement schemes exist, the findings are sobering. A recent evaluation of a £3.9 million programme to build academic capacity for policy engagement found it improved individual skills but had no effect on policy outcomes, and that the most experienced benefited most, widening the inequalities in who gets to engage.⁹ Personal networks remain the system rather than a workaround: when policymakers need academic expertise they turn to informal contacts or commission think tanks, often bypassing universities altogether. The shift shows up in who government pays to do its research. A 2018 parliamentary answer listed at least nine universities among DWP’s recent research contractors; today the department’s live portfolio of around 35 projects is led entirely by commercial research firms, with not a single university as lead.¹⁰

None of this means nothing is being done. The What Works Network, set up in 2013 and now spanning thirteen centres, has built real demand for evaluation; the Behavioural Insights Team made fast, practical trials normal; and since 2021 the Evaluation Task Force, a joint Treasury and Cabinet Office unit, has trained thousands of civil servants and built a public registry of evaluations.¹¹ These are real gains, and the Treasury deserves credit. But they fall short of the scale the problem demands, and are aimed mostly at the Government’s own analytical capacity, not the harder task this article is about: getting independent researchers inside departments and keeping them there.

There is an asymmetry at the centre, too. As David Halpern has pointed out, the Treasury demands rigorous evidence from departments seeking funding but is far less exacting about its own fiscal choices.

A crowded market

Academics now work in a crowded market. Think tanks, management consultancies and commercial research firms all compete to shape policy, and on speed, flexibility and ease of contracting (the things procurement values most), they will always win. Ethics review, data-access negotiations and publication timelines make academic work look slow next to a consultancy that can start on Monday and has access to the procurement channels of Crown Commercial Services. That friction is real, and it explains much of the shift in the DWP figures.

What can academics help with

The first thing is methodological rigour, uncovering what is really going on in trials or departments through ethnographic, embedded fieldwork. Think tanks and consultancies can describe what is happening and recommend what to do, but no consultancy was going to design and supervise a multi-year randomised controlled trial inside DWP over four years. The second is independence that survives the contract: peer-reviewed research is verifiable and adds to a public record that outlasts any political cycle, whereas a consultancy report belongs to the client and is often buried in an archive. The third is capacity that lasts. An embedded academic can train analysts, share evidence and networks, introduce methods and build institutional memory; a think tank or consultancy delivers a report and moves on.

The cost of getting this wrong is not only intellectual. Government spent around £1.4 billion on management consultancy in 2022 to 2023.¹² A six-month academic buyout costs a university roughly £40,000 to £50,000. Not a like-for-like comparison, but a gap of that order is hard to ignore.

What both sides need to hear 

What needs to change

The quick wins cost almost nothing and could be delivered within six months at departmental level: publish organisation charts and staff directories so outside researchers can find the right people; create template data-access agreements with standard rules on publishing rights; and have analytical teams publish short, readable case studies of their own work.

The medium-term changes need cross-departmental coordination and modest funding, and the place to start already exists. The UKRI policy fellowships, run by the Economic and Social Research Council, have since 2021 placed researchers inside departments and What Works centres for up to 18 months, and come close to the model this article argues for. The task is to scale and stabilise them, extend them across social policy, economics and public health, and act on the lessons from their evaluations: clear partnership agreements, more than one point of contact to survive staff churn and proper peer support.¹³ UKRI should also attach funding to the Areas of Research Interest departments already publish, and add an Evaluation Fellows stream modelled on the United States Office of Evaluation Sciences,16 with researchers embedded to design and run the trials that test whether policies work.¹⁴

The structural changes need sustained political will. The schemes that work should be put on a permanent footing and funded as such, so capacity and relationships are not rebuilt from scratch each spending round. And the role of a chief social science adviser, removed in 2010 and only partly revived as the National What Works Adviser between 2013 and 2022, should be restored on a permanent footing, the more easily because the Evaluation Task Force already runs the What Works Network. 

The case for hybrid careers

It is hard to take seriously a critique of government from someone who has never been inside it, yet in the university public policy schools it is the norm. It is equally hard for an official who has only ever worked in Whitehall to see how a policy actually lands on-the-ground in our communities. The response is not better communication across the divide but more people who have stood on both sides of it.

The trial I helped run held together because someone could keep the randomised design and the daily reality of a jobcentre in view at the same time and hold the two when they pulled apart. I came to that work from a university rather than from inside government, which is the point: it took someone willing to cross the boundary. The analysts I worked with kept and used the evidence, knowledge and networks after I left. None of that survives where academics comment from outside, officials commission from inside, and the two meet only at the report stage.

What AI cannot do

I promised to come back to artificial intelligence, and this is the place. If the worry is that knowledge leaves when a researcher does, the instinct now is to capture it in a model and have done. AI is good at that narrow task. It can store what a trial found, summarise a literature and keep a record that never retires or changes teams. But evidence was never the scarce resource. Judgement is. Knowing which finding fits the case in front of you, which caveat matters, when a result is too fragile to act on and when it is solid enough to defend to a minister, none of that can be generated by a model or handed over in a document. It is built by doing the work, and it lives in people.

The deeper problem

Departments now work with academics the way they hire the big consultancies: a brief is written, a contract let, a deliverable received. Officials have become customers. They commission and procure with great skill but produce less of the thinking and own less of the design of policy than they once did. So when an academic arrives to collaborate rather than deliver, the people across the table are set up to buy a product, not to build knowledge alongside someone. Embedding researchers is a good option, because you cannot relearn how to produce by procuring more cleverly. You do it by keeping people in the building who still do the work.

The country and the Civil Service are stretched, and short of exactly this: people who can do serious research and understand how a department works from the inside. Rebuilding that group is one of the cheapest and most useful things government could do. It costs a fraction of what government spends on consultants, and unlike a consultancy report it leaves skills, memory and judgement behind. The point of all this is to get things done and make people’s lives better, in society and in government.

I am grateful to Matt Leach (Building Centre and the Built Environment Trust); Jennifer Dixon and Dame Carol Black, who supported the research idea and funding; and James A. Riccio (Manpower Demonstration Research Corporation) for their comments, suggestions and insights.

Adam Coutts is a Senior Research Fellow at the Centre for Business Research, University of Cambridge.