Transcript Lecture 5
EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Nick Saxton Fall 2014 Agenda • American Evaluation Association (AEA) conference experience(s) • Improvement- and accountabilityoriented evaluation approaches • Social agenda and advocacy evaluation approaches • Eclectic approaches • NOTE: If we do not get through all of this material tonight we will continue next week… AEA experience • What was your most ‘profound’ experience at the conference? • What did you learn (or not learn that you expected to)? • What did you like most? • What did you like least? Improvement- and accountability-oriented approaches • (Ideally) are expansive and cover all aspects related to merit, worth, and significance of a program or other type of evaluand • Intended to inform program improvement and/or decision making (i.e., accountability) • Extremely difficult to implement in practice – Stufflebeam – Scriven Approach 15: Decision- and accountability-oriented studies • Advance organizers – Relevant decision makers, decisions to be made, accountability requirements, etc.. • Purposes – Decision making—whether formative or summative • Sources of questions – Stakeholders (typically limited) • Questions – How can the evaluand be improved? – Does the evaluand meet accountability requirements? • Methods – Any relevant method for determining how to improve a program or provide information regarding accountability • Pioneers – Lee Cronbach, Daniel Stufflebeam • Use considerations – Generally directed toward program management and staff • Strengths – Scope • Weaknesses – Emphasis on program management and staff Approach 16: Consumeroriented studies • Advance organizers – Complex conceptions of ‘values’ (i.e., criteria) • Purposes – Typically, but not always, summative – Extent to which consumers’ needs are met • Sources of questions – Relevant sources of values (see Scriven’s Key Evaluation Checklist [KEC]) • Questions – How well does the evaluand meet consumers’ needs? • Methods – Any relevant method for determining how well consumers’ needs have been met • Pioneers – Michael Scriven • Use considerations – Emphasis is on instrumental use • Strengths – Independence • Weaknesses – Independence Approach 17: Accreditation and certification • Advance organizers – Accreditation or certification requirements • Purposes – Whether accreditation or certification requirements are met • Sources of questions – Accreditation or certification requirements • Questions – Have accreditation or certification requirements been met? • Methods – Any relevant method for determining how well consumers’ needs have been met • Pioneers – Various • Use considerations – Accordance to accepted standards • Strengths – For making informed judgments • Weaknesses – Emphasis on inputs and processes Social agenda and advocacy approaches • Predominately aimed at increasing social justice • Giving power to marginalized/disenfranchised groups • Tend to favor ‘qualitative’ methods and a constructivist/constructionist perspective • Stress cultural pluralism, relativism, and multiple realities • Only a small emphasis on determining merit, worth, or significance Approach 18: Responsive or stakeholder-centered evaluation • Advance organizers – Stakeholders’ concerns • Purposes – Varies widely, depending on stakeholders’ needs • Sources of questions – Stakeholders • Questions – What has the program achieved? – How well has the program been implemented? – What do experts ‘think’ about the program? • Methods – Most often qualitative • Document analysis • Observations • Interviews • Pioneers – Robert Stake • Use considerations – Primary stakeholders • Strengths – Intended to address stakeholders’ concerns • Weaknesses – External credibility Approach 19: Constructivist evaluation • Advance organizers – Rejects the notion that merit, worth, and significance can be ‘objectively’ determined • Purposes – To ‘construct’ stakeholders’ experiences • Sources of questions – Pluralistic values of stakeholders • Questions – Emergent • Methods – Predominately qualitative • Pioneers – Evon Guba, Yvonne Lincoln • Use considerations – Often conflicting accounts and no definitive judgments • Strengths – Stakeholder involvement • Weaknesses – Time, costs, and—most importantly— relativist results Approach 20: Deliberative democratic evaluation • Advance organizers – Democratic participation – Dialogue – Deliberation • Purposes – Democratic participation • Sources of questions – Stakeholders/evaluator(s) • Questions – What is the merit and/or worth of the program? • Methods – Any relevant methods for gathering evidence • Pioneers – Ernie House • Use considerations – Should represent interests of all relevant stakeholders • Strengths – Democratic participation • Weaknesses – Ambitious demands required to execute the approach Approach 21: Transformative evaluation • Advance organizers – Social justice and equity • Purposes – Recognizing ‘situational’ nature of knowledge claims • Sources of questions – Least advantaged groups of stakeholders • Questions – None that can be identified in advance • Methods – Generally mixed-method • Pioneers – Donna Mertens • Use considerations – Inclusion of marginalized stakeholders • Strengths – Emphasis on social justice • Weaknesses – External credibility Eclectic evaluation approaches • No particular philosophical or methodological orientation • Draw from various evaluation concepts and methods to secure ‘useful’ evaluation findings • Tend to invoke the Program Evaluation Standards Approach 22: Utilizationfocused evaluation • Advance organizers – Potential users and uses • Purposes – Information necessary for intended uses by intended users • Sources of questions – Intended users, often determined through situational analysis • Questions – Specific questions articulated by intended users • Methods – Any relevant method for addressing intended users questions • Pioneers – Michael Patton • Use considerations – Situational, and tailored to intended users intended uses • Strengths – Maximizes evaluation impact • Weaknesses – Too much user input and involvement Approach 23: Participatory evaluation • Advance organizers – Promoting buy-in and use • Purposes – Tends to be directed toward program improvement • Sources of questions – Intended users and other stakeholders • Questions – Specific questions articulated by intended users and other stakeholders • Methods – Any relevant method for addressing intended user and stakeholder questions • Pioneers – Brad Cousins • Use considerations – Situational, and tailored to intended users intended uses • Strengths – User-friendly • Weaknesses – Sometimes, poor technical quality