Transcript Approaches to Public Policymaking, Policy Analysis

Approaches to Public
Policymaking, Policy Analysis
& Evaluation Research
Kathy Luckett
University of Cape Town
Research Paradigms
 Not rigid paradigmatic incommensurability
 A map for navigating choppy waters around
policy analysis and evaluation methodological
debates
 Post-positivist – experimental, pragmatic
 Interpretative – constructionist, poststructuralist
 Critical – PAR, empowerment evaluation,CSH
 Critical Realism – theory-based evaluation
Post-positivist: Quasi-experimental
 Popper, Campbell & Stanley (1963, 1966), Lasswell, Rossi,







Lipsey & Freeman
Based on methods of the natural sciences, statistical
measurement techniques
Social science can contribute to improved governance or
management
Establish cause & effect relations bet policy/ programme
objectives, inputs & interventions --------- outputs, outcomes &
impact
Human performance can be objectively measured tv. Efficiency
& effectiveness criteria
Evaluator: objective, neutral, rational
Problem: to secure internal validity of evaluation results
By 19702 disillusion set in, shift to quasi-experimental methods
(pre- & post tests, times series, comparison group designs)
Critique: Quasi-experimental models
 Uses a model developed for closed systems for open




social systems
Adopts a flat ontology – reality = regularities bet
observable, atomistic objects & events (ignores the
non-observable)
Causality = regularities bet variables within stat. sig.
samples
Claims about causation usually unclear and
unconvincing
Can only provide descriptions (for a few variables on
large populations), seldom explanation
Post-positivist: Pragmatic
(dominant model)
 Developed from ‘new public management’ , the
‘evaluative state’ – wants practical, workable results,
useful for decision-making
 Takes policy/ programme goals as focus of evaluation
 Methods: a) open-ended case study (improvement)
e.g. Patton
b) closed-system sets up criteria & performance
indicators to measure performance & accountability
of individuals & institutions e.g. programme
accreditation
 Sets up criteria and performance indicators to
measure performance & accountability by institutions
and individuals – a closed system
Critique: Pragmatic Models
 Assumes stable external environment
 Difficult to set measurable objectives, criteria
& indicators for actual performance
 Difficult to control variables in open soc
systems - possibility of rival explanations,
difficult to prove cause & effect
 Ignores context & stakeholder meanings,
‘black box’ evaluation – seldom diagnostic
 Can be prescriptive, leading to conformity
Interpretive: Constructionist
 1970s – 80s ‘linguistic turn’, 1980s policy sociology:
meaning socially constructed, human action culturally
and discursively mediated – rejection of naturalism
 Vickers (1995) policymaking as communicative
activity for institutional regulation, a process of normsetting
 Neo-institutional theory emphasises cognitive and
normative factors in policy adoption and
implementation
 Guba & Lincoln (1989), 4th generation evaluation:
focus on subjective stakeholder meanings, values &
interests, evaluator as facilitator, truth as agreement,
evaluation useful to insiders
Critique: of Constructionist Models
 Over-socialised, emphases subjectivity at
expense of structure, truth located in
subjectivities of respondents
 Ignores systemic asymmetries of power
 Inability to rise above context
 Relativist ontology
Interpretive: Post-structuralist
 Foucault’s ‘geneaology’, Ball (1993), Gale (2000)
 Discourse is socially constitutive, in dialectical relation to




practice – sets up systems of power/ knowledge, norms &
values
Policy as political artefact – as text & discourse – with unequal
material & discursive effects that should be exposed
Policy has a normalising & regulatory role, sets up subject
positions that constrain ways of speaking & thinking
Technologization of language for institutional ends
How do certain discourses become dominant?
What discourses are at work when those who govern, govern?
How do they become institutionalised & supported legislatively,
professionally & financially?
Critique: Post-structuralist Models
 Weak on method, selectivity of data, dominance of
researcher as interpreter, tendency to jump from data
to (preconceived) narrative
 Quest to successfully link the micro and macro levels
of analysis difficult to achieve
 All of social life gets reduced to discourse,(materiality
of the social world gets lost)
 Knowledge reduced to conditions of its production
and interests of its producers (epistemological
relativity)
Critical: Emancipatory
 Neo-Marxist insights, Frankfurt School (Habermas
empancipatory interest)
 Critical policy analysis (the ‘argumentative turn’)
policy discourses construct social problems & policy
solutions, policymaking a form of argument to
persuade & manufacture consent
 Challenge: how do discourses become
institutionalised & reflected in institutional practices?
 Ulrich (1994) Critical systems heuristics: policy to be
normatively acceptable to those affected by it, value
clarification – diff groups of stakeholders
Critical: Emancipatory
 Developmental evaluation (Patton)
 PAR
 Empowerment evaluation (Fetterman 1996)
 Transformative evaluation (Mertens 2005)
 Development of evaluees, giving voice to the
silenced, inclusion of marginalised groups
affected by the results
Critique: Emancipatory Models
 Utopian: the ‘better argument’ is produced through




power not rational dialogue - all communication
already penetrated by power
Why should the involved (the powerful) bother to take
into account the views and concerns of the affected
(the powerless)?
Cannot work under conditions of coercion requires a
fully functioning public sphere
Needs to hold material conditions and structures as
contexts for vlaues & interests
Post-structuralists: consensus is neither possible nor
desirable
Critical Realist: Theory-based
 Bhaskar (1978, 1998), Sayer (1992, 2000)
 Reality is stratified – empirical (experiences), actual




(events) & real (non-observable structures & causal
powers)
Holds tog ontological realism + epistemological
relativism
Both agency & structure have causal powers – attend
to both (analytically separate)
Openess of the social world, plurality and
contingency of causality
Key to successful intervention = change of social
practice
Critical Realist: Theory-based
 Pawson & Tilley (1997) Realist evaluation: what




works, how, for whom and under what conditions?
(builds in context & subjectivity)
Evaluator to make programme theory explicit & to
check it out with stakeholders: C + M = O
tests assumptions about causal relations & change
Tests goal realisation, but places in context of wider
social explanation
Evaluation can be cumulative – middle range theories
Critique: Demanding to operationalise, timeconsuming
Conclusion
 Be aware of tradition & model you’re working
in - & of other possibilities
 Complex nature of policy analysis &
evaluation justifies methodological pluralism
 But don’t use methods opportunistically,
select according to values, purpose of
evaluation, stage of the policy/programme
cycle & practical constraints
 Think purpose (teleology), ontology,
epistemology 1st – then methodology!