The new evidence-based policy: public participation between ‘hard evidence’ and democracy in practice

Evidence & Policy • vol xx • no xx • 1–19 • © Policy Press 2019 Print ISSN 1744-2648 • Online ISSN 1744-2656 • https://doi.org/10.1332/174426419X15704985880872 Accepted for publication 07 October 2019 • First published online 25 November 2019 This article is distributed under the terms of the Creative Commons Attribution 4.0 license (http://creativecommons.org/licenses/by/4.0/) which permits adaptation, alteration, reproduction and distribution without further permission provided the original work is attributed. The derivative works do not need to be licensed on the same terms.

• What counts as evidence is still deeply contested in British policy making. • To be seen to be producing valid evidence for policy is one source of greater legitimacy in policy circles for increasingly widespread practices of deliberative public participation. • To be seen to be producing valid evidence for policy does not always result in these processes being taken more seriously. • There is a need for better understanding of the complex multi-scalar politics of discussions about evidence-based policy in the UK.
To cite this article: Pallett, H. (2019) The new evidence-based policy: public participation between 'hard evidence' and democracy in practice, Evidence & Policy, vol xx no xx, 1-19, DOI: 10.1332/174426419X15704985880872

Introduction
The announcement of civil service reform in the UK Government -encompassing a smaller and reshaped workforce of civil servants, increased focus on policy implementation and Government accountability, and the introduction of new skills to the civil service -coincided with the start of the 2012-2015 contract of the Sciencewise programme. Sciencewise is a Government-funded body which aimed to promote and support the practice of public dialogue (deliberative workshops with members of the public usually held over several days) around UK Government science policy. In this time period it was an arms-length body, working as part of what was then the Department for Business Innovation and Skills. Since it was set up in 2004 the Sciencewise programme has carried out more than 50 influential public dialogue projects (Sciencewise, 2019) in partnership with Government departments and agencies around important issues, such as the target to reduce greenhouse gas emissions in the UK by 80% by 2050 and the decision to legalise so-called 'three parent babies' through the use of mitochondrial transfer techniques. The programme was initially created in response to a number of high-profile public science controversies which the House of Lords Science and Technology Committee argued necessitated more direct, two-way dialogue with the public around science policy issues (Jenkin, 2000). The work of the programme is overseen by civil servants but carried out by third-party contractors (during the period of study it was the consultancy firm Ricardo-AEA, the British Science Association, and the charity Involve) and self-employed participation and evaluation experts. This public participation organisation can be seen as a part of a broader move across much of Western Europe and North America to institutionalise practices of public participation since the early 2000s (Irwin, 2006). During this contract period the Sciencewise programme made greater efforts to increase its profile and influence in Government (Pallett, 2015). The focus and activities of early contract periods of the programme around developing a clear model for public engagement and capacity building around Government are summarised by Pallett and Chilvers (2013). A particular target for Sciencewise actors was the Cabinet Office, due to its central role in defining changes in policy-making practice and in overseeing 'civil service learning'. It was hoped that, given the changes in civil servant training called for in the reform plan, there would be an opportunity to make competencies in public engagement or dialogue a core part of the expectations and training of civil servants, giving Sciencewise actors a clear reason to engage in discussions around civil service reform.
Sciencewise had previously been regarded by most departments as a Government communications initiative, which was therefore not a substantial contributor to processes of policy making. However, in the period following 2012's Civil Service Reform plan Sciencewise moved into more ambiguous territory, sitting strategically but sometimes uncomfortably between debates about deeper democratic engagement on one hand and evidence-based policy (EBP) on the other; both of which had the potential to give the programme greater legitimacy in policy making. Viewed as a method of evidence-gathering for government policy, public dialogue had the potential to be considered as a much more central part of government business, and would be dealt with by a government department's evidence and research team, thus having a higher status and more policy influence than a communications initiative. However, as institutionalised procedures for gathering and evaluating policy evidence were geared towards quantitative methods, it was feared that this approach may lead many to dismiss public dialogue as too anecdotal and not representative enough to be considered as robust policy evidence. Viewed as a more normative and political act, public dialogue could instead be treated as a way of testing the reception of certain policies and also an instrument of political legitimacy. In this frame the deliverers of public dialogues could potentially be more creative with their methodologies and more open about the normative dimensions of such projects.
This paper explores the relationship between debates about EBP at this time and increasingly institutionalised forms of public participation in UK Government policy making, in the context of the Civil Service Reform Plan. It will do this by first linking the current literature concerning the re-emergence of the EBP debate in the UK with recent literature on the concurrent institutionalisation of public participation. Three ethnographic vignettes will then be presented, based on a multi-sited ethnography carried out around the Sciencewise programme during 2013, including semistructured interviews, participant observation and document analysis. These vignettes illustrate the multiple challenges and effects of the Sciencewise programmes attempts to contribute to and benefit from the EBP debate.
The first vignette explores a process for horizon scanning and policy agenda setting which the Sciencewise programme contributed to. In this case policy makers seemed to have little respect for the formalised evidence-gathering process used which involved expert elicitation and deliberative public engagement. Furthermore, the 'experts' involved in this agenda-setting process were not necessarily more systematic in their use of evidence to back up claims than the participants in the public dialogues. Yet the expert views were used to frame the terms of the public dialogues. The second vignette focuses on Sciencewise's internal organisational routines for evaluating projects and producing usable evidence for policy. In this case it was found that pressures to construct policy-relevant evidence from public dialogue projects and about publics sometimes worked against processes of open experimentation and reflection on failures which would have improved organisational learning. The audit function played by evaluation and evidence-making procedures, fulfilling a need for Sciencewise to prove it was giving value for money to its funders, was often experienced as restrictive in terms of trying to do good public dialogue. The final vignette concerns Sciencewise actors' deliberate attempts to influence broader policy debate about the meaning and practice of evidence-based policy. These forays showed that expanding definitions of evidence to incorporate new elements, such as the outputs of public dialogues, did not automatically mean that these processes were taken more seriously or had more influence on policy processes. In some contexts Sciencewise found that it was strategically more powerful to present public dialogue quite differently as a democratic act. These vignettes illustrate that the question of what counts as credible evidence for policy has not been definitively settled, and that being considered as evidence for policy does not always enhance the credibility of practices like deliberative public engagement.

Retheorising evidence-based policy
Debates about EBP were revived in the early years of the coalition Government through broader debates about policy making and the civil service, and in the run-up to Sir Mark Walport's inauguration as the new Government Chief Scientific Advisor in April 2013. This latest version of the debate centred on a strong perception that civil servants lacked the capacity -especially scientific and mathematical skills -to process and use evidence effectively. For example, a 2011 Institute for Government publication noted a 'lack of culture and skills for using rigorous evidence in the civil service' (Rutter, 2012: 5). The other key feature of the EBP debate at the time was the focus on the 'robustness' of evidence used and the potential for using more quantitative and quasi-experimental methods in policy making (for example, Haynes et al, 2012). This argument responded to the perceived success of behavioural economics in Government through the introduction of the Behavioural Insights Unit, excitement about the potential for 'big data' to revolutionise policy making and planning around Government services, and the strong promotion of the use of methods like Randomised Controlled Trials in policy areas like education and justice (for example, Haynes et al, 2012).
The momentum behind these arguments spurred the creation of the Alliance for Useful Evidence, a body run by Nesta (National Endowment for Science, Technology and the Arts) which was a co-sponsor of the 'What Works' centres, a joint venture by Nesta and the ESRC on behalf of Government. The What Works centres were a flagship Government policy in 2013, which aimed to build on the success of the health body NICE (National Institute for Health and Care Excellence) and apply quasi-experimental methods in policy areas involving service delivery (Cabinet Office, 2013;Mulgan and Puttick, 2013). These initiatives championed the inclusion of a much greater diversity of sources of evidence or kinds of knowledge in policy making but were also based on a rigid set of assumptions about what constituted legitimate and rigorous evidence or credible policy knowledge, and therefore how this knowledge should be used in policy making (Cabinet Office, 2013). Both initiatives adopted a positivist and quantitative framework to assess evidence, even in reference to topics in the domain of the social sciences.
Forms of public participation in science decision making have not been a common feature of the governance of the modern liberal state, which has instead classically looked to science and scientific expertise, both for an apparently objective and legitimate basis for policy decisions, and for neutral and independent judgements of the consequences of state decision making (Jasanoff and Wynne, 1998;Jasanoff, 2012). Throughout much of the twentieth century expert judgement and knowledge alone were considered to be an adequate basis for decision making, with a particular reliance in the UK on methods like cost-benefit analysis and technology assessment in solving complex decisions, such as the development of nuclear power plants and associated infrastructures (Wynne, 1982). This has been characterised as the 'Decide-Announce-Defend' approach to the making and retrospective public justification of technical decisions (Webler et al, 1995). Welsh and Wynne (2013) characterise the dominant imaginary of the public and its role in technoscience between the 1950s and early 1990s as being passive non-entities, unable to play any meaningful role in decision making other than expressing grateful acceptance.
At the start of the twenty-first century many of the important institutional developments around science and science policy concerned the move towards initiating more of a two-way dialogue with citizens around scientific issues which affected their lives. In the UK this move was characterised as a shift from the focus on public understanding of science (PUS) towards an emerging approach which emphasised public engagement with science (PES) (Michael, 2011;Pieczka and Escobar, 2013). A series of high-profile public science controversies, including the BSE crisis and debates about the risks of using the MMR vaccine, also made it clear that the PUS movement had not succeeded in overcoming public distrust and dissent towards government science and scientists (Jasanoff, 2005). The Sciencewise programme was created in 2004 as part of the Science and Society Unit within what was then the Department for Trade and Industry. Sciencewise's creation followed strong advocacy for the involvement of citizen voices in science policy making from academics (for example, Wynne, 1996) and some policy makers (Jenkin, 2000;POST, 2001;2002), as well as emerging amongst other consensus-driven initiatives of New Labour or 'Third Way' politics (for example, Thorpe, 2010). This institutionalisation of deliberative approaches to public engagement with science policy in the UK during the 2000s has been referred to as the 'new scientific governance' (Irwin, 2006). These developments seem to have occurred quite separately to conversations about EBP, and were based on contrasting epistemologies; emphasising the value of qualitative evidence in giving rich insights into public perspectives rather than aiming for the quantitative standards of robustness and replicability emphasised in EBP discussions. This led a recent Nesta report to conclude 'democracy and evidence are not happy bedfellows' (Breckon et al, 2019: 6), despite arguing for a multitude of opportunities for deliberative public engagement to contribute evidence to policy making.
Key elements of this 'new scientific governance' have been the emergence of new models of best practice around public engagement and knowledge making, accompanied by the rise of a new set of experts of participation. Chilvers (2008) mapped these actors as an emergent epistemic community in the UK with shared norms and modes of knowledge making, arguing that a specific set of methods for facilitating and presenting the findings of public participation processes were being narrowed and standardised in the community. Lee (2014) has observed similar trends in North America, and both authors have noted the growing power of these new experts to orchestrate and control public participation processes and to speak for the public in policy-making processes. It has even been suggested that the increasing level of standardisation of models of participation -through training programmes, bestpractice guidance, and the dominance of particular individuals and organisations -has led to creation of 'laboratories of participation' (Bogner, 2012) where conversations and outputs are so tightly controlled as to be virtually irrelevant to the real-world contexts of the issues under discussion. The most fundamental criticism of this form of governance has been that the institutionalisation of these new practices has done little to change the broader organisational cultures of governing institutions (Wynne, 2006;Bickerstaff et al, 2010). This suggests that, though widely practiced, deliberative public engagement processes, based on small numbers of participants, nuanced facilitation and the collection and analysis of qualitative social science data, still lack legitimacy in policy organisations.

Methodology
This paper draws on data from a multi-sited ethnography of the Sciencewise programme carried out during 2013 using a combination of semi-structured interviews, participant observation and document analysis. 'Multi-sited ethnography' (Marcus, 2007) is a methodological approach increasingly drawn upon by STS (Science and Technology Studies) scholars as they move out of the laboratory to study the more diverse locations of science and science policy (Hess, 2001;Thompson, 2004;Ellis and Waterton, 2005;Scott and Du Plessis, 2008;Gehrke, 2014;Laurent, 2017). In the anthropological frame, multi-sited ethnography has been suggested as an alternative to conventional approaches to ethnography that require deep and long immersion in one space, in order to develop an understanding of the multiple and often contradictory processes feeding into and influencing any given organisation, group or situation, and even constituting the spaces themselves (Marcus, 2007).
Another feature of multi-sited ethnographies is that ethnographic understanding of the organisation is not assumed to only be a result of participant observation, but can also be gained through the use of other methods such as semi-structured interviews and document analysis (Marcus, 2007). Thus all of the sites studied around Sciencewise were explored through a combination of these methods. Participant observation was adopted when possible and permitted by Sciencewise actors (including in steering group and programme board meetings), and was then followed up and enriched through semi-structured interviews with actors associated with each space, attempting to get good coverage of different kinds of roles within and around Sciencewise. In total 27 people were interviewed, some with relevance to and involvement in one or more of the organisational spaces studied. Through participant observation and interviews the researcher came into contact with virtually every member of Sciencewise staff during the period of research. The researcher has never worked for Sciencewise.
Document collection and analysis, including official documents, internal documents, preparatory materials for events and online materials such as blog posts, were also used to supplement understanding of the spaces under study and to follow how certain ideas travelled and were refined. Documents were found through the websites of the organisations involved, including Sciencewise's own website, through their relationship to particular events studied, and through the use of less formal online methods, including following the main Sciencewise actors on Twitter and Linkedin and through following the Twitter hashtags for particular events. These online materials themselves also served as objects for analysis. Full ethical approval for this work was granted by the University of East Anglia on the condition that interview and observational data were fully anonymised and stored in a secure manner. For this reason the data used in this study have not been made available in an open repository.
All of the data collected through the methods outlined above were analysed inductively using the qualitative data analysis software ATLAS.ti. The coding of the data was iterative and inductive, responding to new patterns and issues emerging from the documents, such as the civil service reform agenda discussed. A full list of field notes and documents analysed can be found in Pallett (2014). In order to protect the identity of interview respondents as well as the status of a number of high-stakes processes which were underway during the period of research, interview respondents are not listed or numbered. Rather, where interview quotes appear the speaker will be identified in a way that is relevant to the themes and processes under discussion but is judged not to fully disclose their identity, for example as a Sciencewise management actor or a non-Sciencewise actor.

Between democratic acts and evidence fit for policy
In the following vignettes the paper explores three significant processes through which actors in the Sciencewise programme tried to negotiate the uneasy dance of both affirming the democratic legitimacy of public participation processes and positioning them as a crucial element of 'the new evidence-based policy'. Due to the continued predominance of quantitative and positivist standards for assessing the quality of evidence in EBP discussions, public participation processes which are based on qualitative social science methodologies continued to be dismissed as lacking in rigour and robustness in policy settings, despite their potential to help policy makers engage with public values and uncertainty.

A novel approach to horizon scanning
In the development of the proposal for the running of the 2012-2015 contract of the Sciencewise programme conversations focused on what all of the parties involved had mutually identified as a lack of clarity and systematisation around central government horizon-scanning procedures, and therefore a potential role for Sciencewise in experimenting with and promoting new methods for horizon scanning. As one actor involved in the process put it, "you know, could there be a more systematic way to think about what…, the space of possible public dialogues and then,… at least need to have a view of… what's not being done as well as what's being done". As a result, the idea of funding some sort of expert workshop for government horizon scanning, in partnership with Robert Doubleday (Academic and director of the Centre for Science and Policy at the University of Cambridge), was written into the initial proposal document to BIS (The Department for Business, Innovation & Skills), and then put into action when the new contract was awarded. The horizon-scanning project later developed into a three-stage process taking place during 2013, involving an expert workshop hosted by the Centre for Science and Policy, a series of public dialogue workshops carried out by Ipsos-MORI on behalf of Sciencewise, and finally a workshop hosted by the Parliamentary Office of Science and Technology (POST) at the Houses of Parliament.
For the Sciencewise actors I interviewed, this horizon-scanning project had two main functions. The first was to systematically identify the 'hot topics' around which dialogue could be done. This would then help to indicate if there were any important science and technology policy issues that Sciencewise had not engaged with, as well as providing a potentially useful resource for other parties. The second function of the exercise was to use the evidence set built up as a tool for starting conversations with hard-to-reach government departments and those that Sciencewise had not worked with before, potentially leading to high-profile public dialogue projects. By including a prominent public dialogue element in the horizon-scanning process, Sciencewise actors hoped also to be able to demonstrate the value of public dialogue during these conversations and reinforce their own position as knowledge brokers of public attitudes and views. It was anticipated by many of the actors involved that the exercise could also fulfil a more straightforward advocacy and networking function, by introducing all of the participants in the expert and policy workshops, many of whom were high-profile policy makers unfamiliar with the concept of public dialogue, to Sciencewise and its work.
The Centre for Science and Policy (CSaP) expert workshop lasted a full day and involved around 50 policy makers and academics (both social and natural scientists) with expertise related to science and technology policy. I attended the workshop as a scribe and contributor. A list of around 200 emerging policy issues was identified before the workshop through interviews and email feedback from key stakeholders, including academics, industry representatives, science journalists and some civil servants. A vote on which were the most important of this list of issues was used to narrow down the list to 120, which were discussed during the workshop itself, and eventually narrowed down to 30 key emerging policy issues through small group decisions and a plenary discussion at the end of the day. All of the sessions in the workshop were fast-paced, requiring those involved to make quick decisions and compromises, often concerning issues falling well outside their usual domain of expertise; furthermore, participants more regularly resorted to anecdotes and other narratives than they did to more 'scientific' forms of evidence, in order to justify their arguments. The workshop was strongly geared towards aiding the production of a publishable paper, eventually published in the journal PLOS One (Parker et al, 2014), outlining the 30 emerging policy issues chosen, with small sections written by different workshop participants. Topics identified included, for example, democracy in the digital age; antimicrobial resistance; and sudden environmental change.
The 30 emerging policy issues identified in the expert workshop were then used as the basis for a public dialogue exercise run by the market research company Ipsos-MORI, involving three six-hour-long public dialogue events in London, Manchester and Cambridge, involving a broadly representative sample of the public with little or no prior interest in the issues under discussion. The academic paper was used to produce the stimulus materials, and the participants' discussions were also structured around the same seven broad policy areas used in the expert workshop and paper. A short part of each workshop invited the participants to put forward their own ideas about future science and technology-related policy challenges. For the rest of the time in the public workshops, though the participants' views on the issues were openly elicited and they were encouraged to debate with one another, it was made clear by the facilitators that the information on the slides represented the expert view and there was little scope for participants to challenge their substantive content. The final part of the event involved the whole group of participants placing the issues they had chosen as most important on a large matrix which ranked them in terms of their importance as emerging policy issues, and the extent to which there was a need for further public dialogue on them. This matrix ranking formed a central part of the report resulting from the public dialogue process which some of the participants from the three events were invited to help with drafting over a three-hour session in London.
The POST horizon-scanning meeting, held in a committee room in the House of Commons, lasted around one hour and had two Lords and one MP in attendance. At the start of the session the deputy director of POST, who had attended the CSaP workshop, contextualised the event amid the revival of government interest in horizon-scanning exercises and presented the 30 emerging policy issues under discussion, categorising them in terms of how they fared in an initial polling of interested policy makers. In the following discussion much attention was focused on the quadrant of public views on the importance of the policy issues identified and their relevance to further public dialogue work. However, the bulk of the discussion focused on the issue-ness of the issues presented. Some of the policy makers present felt that there was a lot of overlap between the issues, and that the issues were of different types meaning that they could not be easily compared; however, they had not been provided with the full descriptions of each of the issues from the expert process and academic paper. It was suggested that the issue of health care provision for an ageing population was an issue requiring a Royal Commission, while it was asserted that many of the other issues were also inappropriate for a horizon-scanning exercise, given that they were current rather than future policy issues. Finally, one of the Lords concluded that in future horizon-scanning processes the selection of issues should be carried out by policy makers, rather than other experts and citizens, in order to ensure they were relevant and usable within policy processes. Following the meeting POST produced only an internal report on the horizon-scanning exercise, which raised concerns about the methodology of the expert workshop and emphasised that POST's involvement in the whole exercise was purely experimental. The research found no evidence that the organisations involved had taken any ideas from the process forward.
This vignette illustrates the challenges of orchestrating and producing outputs from public participation processes which are seen as legitimate and credible in 'the policy room'. In a very short encounter which did not allow for any of the issues identified to be discussed in depth or the methodologies of public engagement to be justified, the outputs of this process were apparently dismissed out of hand. The public participation element of this project was also implicitly attributed lower status in the progress of the project by the fact that it was held after and largely shaped by the expert workshop, and the way in which lay participants were given little scope to suggest their own issues for discussion. While the methodology used in the public dialogue element of the project was arguably much more robust, and certainly more established and 'tried and tested' than the Delphi format of the expert workshop, this did not seem to enhance its credibility. However, the conduct of the POST meeting also suggests that policy makers viewed even the (peer-reviewed) expert contributions to the project as lacking credibility, instead asserting the importance of discussing such questions in the formal political domain. This event undermines claims of the general resurgence of interest in EBP in UK governance settings.

Producing rigorous evidence
The idea of collecting, storing and disseminating 'best practice' was a central Sciencewise aim which strongly informed the layout of the website and many organisational routines, stimulating the production of case studies and guidance documents. The idea of Sciencewise itself as a repository or producer of institutional memory also shaped internal views of how Sciencewise could contribute to making EBP. The creation of social intelligence reports -labelled on the Sciencewise website as 'What the public says' -was a new initiative of the 2012-2015 programme contract. It was hoped that these reports would give a more credible and lasting insight into the shape of societal attitudes towards these different science and technology related issues, getting beyond accusations that public dialogue projects only offer a snapshot of the views of a small group, but also to provide a richer picture than that provided by public opinion surveys. While the social intelligence work provided an opportunity for Sciencewise actors to learn about and reflect on particular topics and to develop perspectives on the state of science policy making in that area, the reports were mostly conceived of within Sciencewise as an advocacy tool. It was hoped they would demonstrate the value of public engagement to policy makers and that they could be used to robustly identify areas for future dialogues projects.
The social intelligence work was seen as bolstering the storage and translation of institutional memory, both within Sciencewise and within Whitehall. One Sciencewise management actor justified the social intelligence work like this: as well as doing the open-ended deliberative stuff we were getting a lot of calls with people saying well "what do you know about how the public thinks about these things? Not just what it says but how it thinks" um… I think we realise that attitude and opinion research only has so much value unless you understand how people are thinking about these things underneath. So the social intelligence work we did was an attempt to, not just sort of map where the public was, but to… in terms of opinions, but to tease out some of the underlying reasons and values behind that, as far as we knew from the research, from social media and all the rest.
These reports were not just about synthesising insights from Sciencewise projects, but also bringing these together with research from elsewhere. Furthermore, through this work Sciencewise actors hoped not only to present and bring together existing knowledge about relevant topics, but also to use this knowledge to identify broader trends and drivers which might be relevant to policy makers. This approach clearly advanced Sciencewise's advocacy objectives, positioning the programme as a knowledge broker of public attitudes, but it also arguably represented an attempt to provide a new way of storing and translating relevant policy knowledge about the public.
Public dialogue evaluations were the main mechanism within Sciencewise to collect and store evidence from public dialogue projects. Independent evaluators were appointed at the start of each dialogue project, so that they could follow the progress of the public workshops and offer advice and reflections during the process. The evaluation reports would usually be published several months after the initial reports on the dialogue projects to allow the evaluators to conduct interviews with some of the actors involved, process survey data from the dialogue's participants, and observe any broader effects of the project on policy processes. The first draft of the evaluation report would be circulated among key actors in the dialogue project in advance of the wash-up meeting, which would involve the dialogue evaluators, contractors and commissioners reflecting on the successes, failures and key lessons of the project, facilitated by the project manager. Though these discussions were not formally recorded, it was intended that they would feed into the production of the final evaluation report, and the shorter Sciencewise case study write-up of the project, both of which would be available on the Sciencewise website. Public dialogue, activity and whole programme evaluations were used extensively for synthesising knowledge into advocacy and informational materials on Sciencewise's website, such as dialogue case studies, guidance documents or the FAQs.
The development of public dialogue case studies, and the way these evaluation reports were used in the programme-wide evaluation, indicates that they also provided an important audit function, as might be expected in an arm's-length government programme. Several interview respondents recognised this as a limitation, for example: [w]e have to be honest, that our evaluation of Sciencewise has to fit the criterion objectives that BIS set in its funding model, so in a sense there are things that we have to evaluate to prove that the money's been spent well, um… so on one level you could say it's a sort of operational evaluation. (Sciencewise steering group member) Sciencewise felt the need to bring in 'neutral' observers to measure and verify the quality and impact of its projects, to demonstrate that it was being held to account and to provide evidence of the programme's effectiveness. Several Sciencewise internal documents even use 'evaluation' and 'producing credible evidence' interchangeably. Thus, while Sciencewise evaluation procedures offered an important mechanism for potential organisational learning and reflection, on the way they commissioned and ran projects, the forms of public engagement they were promoting, or the topic areas they were engaging with, this was not the only or primary function of these mechanisms. Furthermore, other concerns, such as the need to provide convincing evidence of the success and impact of Sciencewise projects and therefore not to acknowledge failures or problems, potentially precluded opportunities for honest reflection and broader learning.
The yearly programme evaluations were also seen as a source of organisational learning about 'big-picture' issues. For example, the 2009-2010 programme evaluation was cited as a key source for the new projects and ideas contained in the 2011 proposal document, such as the horizon-scanning project. Through conducting interviews with Sciencewise and policy actors, the programme evaluation reports had been useful in identifying generally agreed areas for improvement or opportunities for new initiatives. In common with the project evaluations, the programme evaluations also served a key self-auditing function and were constructed with a policy audience in mind, meaning that findings had to be clearly and very concisely stated.
This vignette about evidence synthesis and evaluation procedures in Sciencewise illustrates the challenges of producing evidence to fulfil a standard and framework set by a distant body with very different assumptions about and ways of assessing evidence. On the one hand this close auditing and evidencing of the impacts of the Sciencewise programme helped to demonstrate its positive policy impacts and the value of the approach to a sometimes-sceptical audience of policy makers in control of budgets. It led to the production and synthesis of vast amounts of academic and market research about public views around science and technology, with direct relevance for government decision making. On the other hand, these audit regimes were constantly experienced as restrictive and sometimes even counterproductive to the aims of running good public dialogue processes or faithfully representing the views of citizens. Key lessons became lost and obscured through communication and translation problems in large chains of multiple actors involved in the evaluation, oversight and orchestration of each project, while the need for Sciencewise to constantly justify its existence and compete for further funding sometimes prevented an honest assessment of the failures and oversights of particular projects. Connectedness with EBP agendas had a very ambivalent effect on Sciencewise's internal procedures and broader standing, while at the same time there was little evidence that public participation processes were being taken more seriously as a source of credible evidence for policy.

Debating evidence-based policy
During the period of study Sciencewise actors became involved in national debates about EBP in several ways. One common way for Sciencewise actors to contribute to the debate was through 'thought leadership' work, writing for their own or external publications. Sciencewise actors also tried to engage specifically, with debates about the impacts of civil service reform and the inauguration of the new Government Chief Scientist Mark Walport, on the use of evidence in policy through its involvement in a prominent policy seminar series, and by inviting then DEFRA Chief Scientist Professor Ian Boyd to a steering group meeting.
Related to the prominence of the EBP agenda, there was an ongoing and unresolved debate within Sciencewise during 2013 on whether public dialogue was research or a democratic act (see Liabo et al, 2018). Due to their backgrounds in conflict resolution and stakeholder engagement, several of the Dialogue and Engagement Specialists felt quite strongly that public dialogue should be viewed as a form of market research, as it only included those with no predefined stake in an issue and it was not aimed at resolving a particular state of affairs. Actors who had more of an advocacy role within Sciencewise were more likely to emphasise the normative dimension of public dialogue, seeing it as a democratic trial or experiment which opened up a particular policy issue to citizen input and scrutiny.
This view is also reflected more generally in Sciencewise's gradual drift towards a greater focus on advocacy. However, several actors were also willing to take a more interpretive or strategic view of this debate, recognising that at different times and in different contexts it would sometimes be appropriate and constructive for public dialogue to be viewed as research or evidence, and at times it would be beneficial for public dialogue to be seen as a source of democratic legitimacy and accountability.
Many of my interview respondents felt that there was a lot at stake for Sciencewise in these EBP debates and initiatives. On the one hand, many recognised that public dialogue had the potential to be considered as a much more central part of government business if it was viewed as a method of evidence-gathering for government policy; for example, the director of the Alliance for Useful Evidence's contribution to the Sciencewise anthology notes, '[p]ublic dialogue can be an important part of the evidence base for social policy and practice' (Breckon, 2013: 15). In an interview exchange one Sciencewise actor with significant experience in Whitehall explained why they felt public dialogue needed to be regarded as evidence to really be taken seriously and remembered within Government: that's why this concept of 'is it evidence' is really important because the government departments do evidence-based reviews, I listened to a presentation a couple of weeks ago by [a cross-departmental body] but… I don't know whether thinking about what the public thought about the topic a few years ago would count as evidence. So we're back in that big circle of 'What is evidence? 'Is dialogue evidence?' 'What is the impact on policy makers?' 'What is the impact on decision makers?' Me: So I mean, from your perspective dialogue is strengthened if it is seen as evidence in these processes?
Respondent: Absolutely, yeah, absolutely, and that's a major cultural shift that would be required, by government departments, and particularly the strong scientific departments, because to be frank the only evidence that ever gets used is either pure science or economics.
This exchange captures the hope shared by many in the management and steering structures of Sciencewise that if public dialogue was categorised as 'evidence' it would therefore be dealt with by a government department's evidence and research team, giving it a higher status, more policy influence and a greater likelihood of being remembered than a project viewed as a communications initiative. For some this also meant that Sciencewise needed to reconsider how it presented, gathered evidence from, and evaluated public dialogue projects; for example, using more quantitative measures, allocating more resources to evaluations, and adopting digital methods in order to satisfy requirements for statistical significance through 'scaling up'. However, interview respondents were also concerned that assessing public dialogue as evidence within EBP frameworks which were overwhelmingly geared towards quantitative methods would lead to dialogue being dismissed as too anecdotal and not statistically representative enough to be considered as robust evidence for policy.
Sciencewise was involved as a partner in organising a series of seminars called 'Future directions for scientific advice in Whitehall', created to reignite debates about the role and practice of scientific advice around government in the run-up to the inauguration of the new Government Chief Scientific Advisor, Sir Mark Walport, in 2013. The seminars were co-organised alongside the Institute for Government (IfG), the Alliance for Useful Evidence (run by Nesta), and the Science Policy Research Unit at Sussex University. Each organisation hosted a seminar, followed up by the CSaP annual conference in April 2013 which shared the same name as the seminars, and where an edited collection, including short contributions from academics and policy makers based on the seminars, was launched (Doubleday and Wilsdon, 2013).
The IfG seminar entitled 'Culture clash -bridging the divide between science and policy', held in November 2012, focused on the perceived divide between scientists and policy makers, stressing that the latter tend to have backgrounds in humanities and social sciences, with some speakers suggesting that this was a direct and negative influence on the government's capacity to listen to scientific advice and evidence. Other speakers -such as the then shadow minister for Innovation and Science, Chi Onwurah -highlighted the roles and structures of science advisory bodies and discussed the scope to include scientists in other departmental governance structures. The Alliance for Useful Evidence seminar, 'Broadening the evidence base: science and social science in social policy', hosted at Nesta in January 2013, was concerned with the use of social science in policy making, and much of the conversation focused on calls for more social scientists to be conversant with large-scale quantitative and quasi-experimental methods, which were perceived to be of most use to government.
The Sciencewise seminar, 'Experts, publics and open policy', hosted by Lord Krebs at the House of Lords in January 2013, was the only one in the series to explicitly discuss the role of citizens in policy making. The speakers were: Gemma Harper, chief social researcher for DEFRA; Roland Jackson from Sciencewise; and Lord Krebs, former chairman of the Food Standards Authority (later known as the Food Standards Agency). Roland Jackson (the Sciencewise chair), representing Sciencewise, used his talk to link public dialogue to the prominent policy idea of open policy making, explaining how he saw the two as complementary concepts, and arguing that policy could only be truly open if it involved dialogue with citizens. Gemma Harper and Lord Krebs struck a much more sceptical tone about public participation, with Harper highlighting what she saw as problems with assessing the 'anecdotal evidence' produced by public dialogue projects, alongside other forms of quantifiable evidence. Lord Krebs talked through the example of the 'GM Nation?' public dialogue in 2004, which he saw as emblematic of the potential pitfalls and dangers of inviting public participation.
For Sciencewise actors the programme's participation in this seminar series was part of its advocacy work, raising the profile of public dialogue and attempting to get Sciencewise involved in agenda setting and debates around EBP, science advice and civil service reform. For one Sciencewise management actor: "that was very much about profile raising. It was very much about getting the public aspect in amongst that agenda". As another management actor put it, the justification was "[t]wo-fold I suppose, one to um… you know, get our ideas into a form of policy box if you like, and secondly to be seen to be a part of that, and promote what we're doing a bit more. So raise awareness and contribute some ideas". Through the seminars, and the subsequent publications and blog posts, Sciencewise advocates attempted to position public dialogue as an important source of evidence for good policy making, and continually aligned the practice of public dialogue with open policy making, sometimes drawing on specific examples of Sciencewise dialogue projects as examples of best practice in open policy. Sciencewise's position in the EBP debate remained ambiguous, in part due to the kinds of attitude expressed during the Sciencewise seminar which were sceptical about the robustness of evidence produced by public dialogue processes, but also due to debate within Sciencewise about whether or not public dialogue was social research.
Similar tensions were played out in the October 2013 Sciencewise steering group meeting, when the DEFRA Chief Scientist, Professor Ian Boyd, was invited to speak in the focus section of the meeting about his perspectives on public participation in policy making. He based his remarks on a comment piece he had written for the journal Nature the month before (Boyd, 2013), addressing the EBP debate from a scientist's perspective and exploring what scientists could do to make their research more policy-relevant. Boyd emphasised the need for scientists and policy makers to engage with the public and to take public views seriously, and he argued that it was part of democratic process to engage with and accept the apparently irrational or controversial views of democratic representatives. However, the crux of Boyd's argument, in the paper and in the meeting, was that scientists had to be better at using statistical methods in their work, in order to avoid the creation and amplification of bias in their data, and to ensure the evidence they produced was robust and credible. When Boyd was challenged by several members of the steering group on whether he felt that Sciencewise produced 'robust evidence' in these terms, his response was that provided they used the right statistical methods to eliminate bias then it would be.
For several of my interview respondents this encounter encapsulated the challenges of getting those in Government to understand the nature of public dialogue and see it as a valid basis for policy making. Several also expressed annoyance that Boyd had clearly not been fully briefed on what Sciencewise did before the meeting, preventing any meaningful exchange about the use of different forms of evidence in policy. Others felt that the meeting had been worthwhile, due to a more subtle agenda of gradually introducing actors like Boyd to Sciencewise ideas and practices, contributing to a more diffuse movement around Government where 'one day' public dialogue might be accepted as credible policy evidence. Furthermore, one Sciencewise management actor felt that Boyd's remarks had moved between several contradictory visions of the public and public participation, some of which were more in line with the Sciencewise approach. It was also pointed out that having Boyd and his PA present at the lunch before the meeting had facilitated more informal interactions with Sciencewise management actors, discussing topics such as the impact that earlier public engagement around bovine tuberculosis (a controversial policy issue at the time) might have had on the development of Government policy and on public reactions around this issue.
To me, this exchange also revealed strong power differentials in debates about the use of evidence in policy. Despite their superior expertise in matters of public participation, Sciencewise management actors and steering board members felt unable to challenge the fundamental premise of and assumptions behind Ian Boyd's talk at the meeting. It was seen as more important to have him 'onside' but apparently confused about what they considered to be fundamental tenets of public dialogue, than to risk losing him as a potentially powerful advocate.

Discussion and conclusions
This paper has presented a case study of the Sciencewise programme's interactions with a revived EBP debate in and around the British Government. While the programme was arguably already influential, having overseen more than 30 public dialogue projects around important areas of policy making, revived EBP debates presented the potential to improve the legitimacy of public dialogue and the Sciencewise programme among policy makers. In particular, the proposal to broaden definitions of what counted as credible evidence for policy, to incorporate knowledge from the social sciences and evidence related to policy implementation, posed a significant opportunity for actors related to the Sciencewise programme to contribute to the debate and further promote public participation as an essential and valuable input to policy making.
However, research conducted around the programme during 2013 reveals significant tensions around the extension of the definition of EBP. In the context of the experiment with horizon scanning, or in interactions with policy actors and EBP advocates through the policy seminar series and Ian Boyd's steering group visit, public dialogue was frequently dismissed as lacking the necessary rigour to be counted as evidence in policy processes. These judgements were based on positivist and quantitative frameworks, which suggested there had been little attempt to broaden definitions of evidence for policy -though this is not to argue that policy actors adopt a positivist stance in all contexts, rather that positivism appears to be the default in the context of EBP debates. In the vignettes presented above it did not seem that public dialogue processes were being taken more seriously, or became more likely to be incorporated into policy, when they were presented as part of EBP. While there are examples of ways in which presenting public dialogue as evidence helped the Sciencewise programme gain legitimacy initially, for example its involvement in the 'Future directions' policy seminars, there were also many ways in which it led to important work being dismissed as lacking in rigour or not being representative of the broader population.
Engagement with the EBP debate also led to tensions within the Sciencewise programme about how to categorise public dialogue. Mirroring the outward-facing debate, competing perspectives were put forward on whether public dialogue should be considered as a research method (producing evidence) or a democratic act. These two categories also mapped onto the programme's twin aims of substantively improving policy through public participation and broader advocacy for public participation. Internally, mechanisms and frameworks for evaluation were very important for codifying evidence and broader lessons from public dialogue processes. However, these audit processes also encouraged the partial presentation of evidence by providing a framework which was not always appropriate for assessing public dialogue outputs, and by necessitating the occlusion of failures or opportunities for improvement.
These findings point to the conclusion that there is a broader politics at play in what gets counted as evidence and how the evidence is judged. Despite many rhetorical calls during the period of research by high-profile figures for definitions of evidence to be broadened in the context of EBP, positivist and quantitative frameworks of evidence continued to be the main resource for assessing legitimacy and rigour. The EBP debate seems to struggle to move beyond these positivist and quantitative logics, to consider other definitions of quality and rigour which have been well-established in the social science methods literature for at least two decades. Therefore approaches and frameworks held together under the EBP umbrella also struggle to deal with ambiguity and uncertainty -though they are axiomatic characteristics of pressing science policy issues (Stirling, 2008) -as well as the idea that issues and findings are contextual and may remain in flux, as public perspectives do (Felt and Fochler, 2010). The research presented in this paper therefore suggests that trying to incorporate small-n, qualitative and creative work into these frameworks will always result in them being assessed as sub-standard.
Bodies promoting public participation like the Sciencewise programme may instead benefit from considering other sources of legitimacy and credibility in policy making if they wish to make a difference to policy agendas (Jasanoff, 2012). Democratic legitimacy is an oft-debated but nevertheless fundamental feature of the modern liberal state. It is not something which can easily be captured by standards or frameworks, but rather something to be struggled over and experimented with. However, in the quest for legitimacy for public participation processes as a democratic act, the role of values and existing power inequalities in shaping the debate is far more openly acknowledged and explicitly reflected upon. This is potentially a more productive basis for arguing for the value of public participation, than in the context of EBP frameworks which are equally value-laden but far more tightly controlled.

Funding
This work was supported by an ESRC 1 + 3 PhD studentship (grant number ES/ I902201/1). Strategy) for their cooperation and support during this research. Thanks go to the editors of this special issue and to two anonymous reviewers for their incisive and constructive comments on earlier versions of this paper. Thanks also to Jason Chilvers, Peter Simmons and Martin Mahony for comments on an earlier version of this paper. Special thanks go to the research participants for generously giving up their time and sharing their thoughts.

Conflicts of interest
The author declares no conflicts of interest.