What does Pawson (2006) mean by ‘evidence-based policy’? Drawing on examples when appropriate, critically discuss the efficacy of adopting this approach in evaluation research.
Evidence-based policy (EBP) is a reoccurring theme in political rhetoric. Whenever a new policy is announced, it is invariably accompanied by a myriad of studies from various academic fields that are utilised to underline the necessity of the policy as well as why it is likely to have the desired effect. Using empirical evidence in order to formulate and justify public policy seems self-evident. However, many prominent social researchers, such as Ray Pawson,are critical of the way in which evidence-based policy is defined and, more importantly, of the current approach to using evidence within the policy-making process. This essay will begin by offering a critical overview of the interventionist approach that is currently applied to EBP. A comparison will then be made between ‘interventionism’ and Pawson’s ‘realist’ approach, before examiningPawson’s assertion that realism is essential to achieving good policy. This comparison will also be accompanied by some relevant examples that have arisen as a result of these different approaches. These examples, along with various critical assessments of interventionism will be used as a foundation forarguing in favour of an incremental shift towards a realist approach to EBP. Finally, potential problems and insufficiencies in Pawson’s specific approach to EBP within a broader political context will also be highlighted.
EBP implies that legislation is formed as a result of careful consideration, combined with extensive empirical evidence that appropriately draws upon the scientific method. Consequently, it may at times be assumed that evidence-based policy isautomatically reasonable. However, there are many challenges associated with adequately accumulating, analysing, and applying evidence to something as complex as public policy. One of the main challenges is thatsuch actions often result in unintended, negative consequences (Pawson,Greenhalgh,Harvey& Walsh 2005). Many prominent social researchers attribute such adverse side effects to the interventionist approach that dominates mainstream politics (Kazi&Spurling 2000; Sanderson 2002). In order to fully understand the reason behind this approach, the preliminary stages of interventionist methodology must first be examined. These stages are recognised as follows:problem identification, which involves measuring and rankinga specific aspect of society, such as crime;public disclosure, in which the information gathered in the previous stage is made publicly available;sanction instigation, which typically involves the broader community taking action by staging a boycott or lobbying the government to take action; andmiscreant response, in which the failing parties are publicly shamed and criticised,thereby theoretically influencing an improvement in future performance (Pawson,Greenhalgh, Harvey & Walsh 2005).
Each stage within this series of events is often easily justifiable to both the public and policy makers alike, particularly in instances in which the call for policy reform is emotionally driven or instigated by a dramatic event. Pawson (IBID) argues, however, that the theories underlining each stage of this process are fallible and capable of failing at any point. In stage one, for instance, a problem may be falsely attributed to something that correlates with incidences of the problem. For example, the anti-vaccination movement insists that vaccinations are responsible for the increase in the rate of autism among children. As a result, advocates of this movement passionately campaign in favour of government intervention in order to prevent parents and doctors from vaccinating children (Calandrillo 2005). Further investigation, however, has revealed that this rise in autism is far more likely attributable to a combination of improvements in diagnostic methods, as well as to the fact that a range of social disorders have been added to the official autism classification (Tanne 2005). The reason that vaccinations were initially identified as being the cause of a rise in autism is because they are typically administered around the same time that children will begin to exhibit symptoms of a disorder(Offit 2011). Nevertheless, the medical profession has been forced to endure every stage of the process outlined above as a result of theincorrect ‘problem identification.’
Assuming, in another circumstance, each stage of the interventionist process is valid, issues still arise regarding the universal applicability of the resulting policy. This question arises because the interventionist process is extremely fragile, as it is embedded within various social systems (Pawson,Greenhalgh,Harvey& Walsh 2005). In other words, a policy that successfully addresses homelessness in the urban parts of Sydney,for example, will not necessarily have the same relevance in rural areas of NSW. Moreover, the logic behind the two fundamental strategies of current evidence-based policy, ‘meta-analysis’ and ‘narrative review’,is argued to be insufficient(Pawson,Greenhalgh, Harvey & Walsh 2004). Pawson(2006) is particularly critical of the use of meta-analysis, referring to it as ‘systematic obfuscation’(p.38). While narrative review is certainly more sympathetic with the realist approach, both strategies have been criticised for sharing common limitations with regards to providing a template for important policy decisions (Pawson 2001). Both perspectives areonly capable of making inferences from evidence that is already available (IBID). To return to the earlier example of the anti-vaccination movement, it would not matter which of these two strategies was used to evaluate the issue. The limited information available, namely that ‘children are being diagnosed with autism soon after being vaccinated’, only allows for the conclusion that vaccines may or may not cause autism. Note that the assertion ‘vaccines cause autism’ cannot be ruled out, nor can a more likely alternative be offered.
This point is extremely significant, as the interventionist process relies heavily upon both meta-analysis and narrative review (IBID). Thus, the risk of public policy being subject to weak inductive reasoning, thereby leading to bad policy decisions, is considerable (Pawson 2002). This risk appears to increase alongside public outrage, and its consequences can be dire. Fortunately, the anti-vaccination movement did not gain enough momentum to have a discernable effect on legislation. However, there are other examples,in which government intervention fuelled by public outrage has been prematurely imposed. One case study related to the US sex offender notification and registration programme (commonly known as Megan’s Law) reveals that the initiative was rushed into law following the murder of Megan Kanka by a convicted sex offender, who was living anonymously in her community. Understandably, the horrendous nature of the crime was at the forefront of the policy process, and evidence of its effectiveness only trickled in as an afterthought (Pawson 2002). Not surprisingly, subsequent research has shown that Megan’s law has not effectively addressed any of the potential issues in relation to released sex offenders and has instead become an increasingly expensive initiative due to the associated logistic difficulties (NCJRS 2008)
In order to formulate the kind of EBP to whichPawson is referring, the hypothesis must be tested against new evidence,different variables must be added and subtracted in order to measure and compare the outcome,before ruling out all other alternate possibilities prior to reaching a final decision. This intricate processdiffers fundamentally between the type of EBP to whichPawson refers to and the interventionist approach. While astrong argument is made, suggesting that the interventionist process is inherently flawed, its key advantage over the realist alternative is its expediency. As previously mentioned, the realist approach demands a far deeper understanding of issues surrounding policy development. Like the scientific method upon which it is modelled, the realist approach is time-consuming and resource intensive (Solesbury 2001). Moreover, the public demand required in order to push through reform is often short-lived and is likely to focus on other issues by the time detailed empirical analysis is able to yield useful results. Pawson (2001) also acknowledges this shortcoming, stating, ‘Evaluation research is tortured by time constraints. The policy cycle revolves quicker than the research cycle.’
Finding a politically viable method for Pawson’s EBP is perhaps the most significant obstacle to making realism a credible alternative to the status quo. Pawson(2001) struggles to overcome this challenge, as he cautiously proposes that there may be some promise in a method of ‘realist synthesis’ (p. 24).Nevertheless, Pawson writes at great length on the merits of the realist synthesis method in many of his publications,which suggests that his caution is simply due to the fact that the idea is still in its infancy, rather than to a lack of confidence in the idea.
As the name suggests, a ‘realist synthesis’ operates by combining methods of systematic review that are already in use for the purpose of producing a completely new method. It should be noted that this method recognises the benefits in different types of methodology that are typically considered to be involved in a paradigm war, such as ‘quantitative’ versus ‘qualitative’, ‘positivism’ versus ‘phenomenology’, and ‘outcomes’ versus ‘process’ (Pawson 2001). Such an approach allows a realist synthesis to be more compatible with various social complexities, as it is sympathetic to the use of a multi-method, multi-disciplinary base of evidence(Pawson,Greenhalgh,Harvey& Walsh 2005).
The logic of realist synthesis contrasts greatly with the interventionist process that was outlined above. While the interventionist methodremains linear and rigid, Pawson’s realist synthesis is far more flexible and is comprised of three main components. The first entails identifying the subject matter, where a basic theory outlining how a policy will function and in what circumstances it will apply is submitted. The second stage involvesthe use of primary case studies.In this stage, primary data is collectedby means of various types of research methods, which can be expected to yield different results regarding the efficacy of the policy. The final stage results in a revised theory. In this stage, the information gathered in the previous stage is used to test, revise, and refine the preliminary theory. While it is possible for the data collected in the second stage to validate or discredit the original theory, the more likely outcome is simply that policy makers gain a deeper understanding of an issue, thereby allowing for more effective policy reform in the future (Pawson 2006, p. 74).
As with any model, the realist synthesis is an oversimplification. In reality, the process is far from smooth, asPawson readily acknowledges when he explains that a maze of back loops and repetitions are likely to be encountered before the synthesis is brought to completion (IBID, p. 102). These obstacles may be considered equally abundant in classic research methods. The key difference is that Pawson’s method embraces these obstacles, rather than suppressing them, as they are considered to be among the most important aspects of research and scholarship (MacLure 2005).
Pawson submits many compelling arguments for a realist approach to EBP. Anyone who recognises the effectiveness of the scientific method is likely to sympathise with his position. Introducing a method for EBP that takes a detached and rational approach to various social issues will almost certainly minimise anyaccompanying logical fallacies and human error that so often have unintended consequences.It is also likely to result in a much more effective type of public policy that improves over time, in a manner very similartothat of science and technology. Whetheror not a ‘realist synthesis’ is the best method by which to deliver this outcome remains to be seen.
Pawson successfully identifies specific problems within interventionism and offers sound and reasonable alternatives. However, his ideas fall short when it comes to addressing underlying issues of human nature that often cause interventionism to have problems to begin with. In this regard, Pawson’s assertions can, at times,come across as rather idealistic, as they presume a perfect world in which public policy remains unaffected by emotional outrage, political fads, or manipulation by unions and lobby groups. When referring to EBP,Pawson implicitly expects that these things will not matter. Interventionism, on the other hand,operates the way in which it does precisely because these things do matter. Thus, it is possible that Pawson’s notion of EBP is simply nota politically viable alternative. Hopefully, with an incremental shift towards realism, however, EBP will eventually begin to resemble something akin to what Pawson is advocating.
Calandrillo, SP 2005,Vanishing vaccinations: Why are so many Americans opting out of vaccinating their children? Washington, The Berkeley Electronic Press, viewed 5 November 2011,http://law.bepress.com/alea/15th/bazaar/art24/
Kalucy, L2007,‘Evidence-based policy: A realist perspective (a review)’, Australian and New Zealand Journal of Public Health, vol. 31, no. 2, doi:10.1111/j.1753-6405.2007.00043.x
MacLure, M 2005,‘Clarity bordering on stupidity: Where’s the quality in systematic review’,Journal of Education Policy, vol 20, no. 4,pp. 393–419.
Offit, PA 2011,Deadly choices: How the anti-vaccine movement threatens us all.USA, Basic Books.
Pawson, R 2006,Evidence-based policy: A realist perspective, CA, Sage Publications.
Pawson, R 2002, Evidence-based Policy: The Promise of ‘Realist Synthesis,’ UK Centre for Evidence-Based Policy and Practice, vol. 8, no. 3,pp. 340–358.
Pawson, R 2001,‘Evidence-based policy: I. In search of a method’, ESRC UK Centre for Evidence Based Policy and Practice, CA,Sage Publications.
Pawson, R, Greenhalgh, T, Harvey, G &Walshe, K 2005,‘Realist review:A new method of systematic review designed for complex policy interventions’,Journal of Health Services Research & Policy, vol. 10, Suppl. 1,pp. 21–34.
Pawson, R, Greenhalgh, T, Harvey, G&Walshe, K 2004,Realist synthesis: An introduction.ESRC Research Methods Programme, University of Manchester.
Popay, J (Ed.)2006,‘Moving beyond effectiveness in evidence synthesis: Methodological issues in the synthesis of diverse sources of evidence.’National Institute for Health and Clinical Excellence.
Solesbury, W 2002,‘Evidence-based policy: Whence it came and where it’s going’, In A. Editor & B. Editor (Eds.),Public Administration,vol. 80, no. 1, pp. 1–22. USA: Blackwell Publishers, Ltd.
Tanne, JH 2005,Increase in autism due to change in definition, not MMR vaccine.BMJ, vol. 330, p. 112, doi: 10.1136/bmj.330.7483.112-d
Zgoba, K, Witt, P, Delessandro, M &Veysey, B 2008,Megan’s law: Assessing the practical and monetary efficacy. New Jersey, The Research & Evaluation Unit Office of Policy and Planning New Jersey Department of Correction, viewed 5 November, 2011 https://www.ncjrs.gov/pdffiles1/nij/grants/225370.pdf