Evaluation for the People

Research Methods for Community Change

Reassessing the Pyramidal Structure of Evaluation Research

  The typical structure of evaluation is quite pyramidal, with interlocking directorates of funding organizations, government institutions, and academic evaluators imposed on lower levels of service agencies, bureaucrats, and community members The top imposes programs on the bottom, which implements projects to fulfill the program mandates Funders and Interlocking Elites Programs Service Agencies Bureaucrats Projects Community Members Adapted from Stoecker p. 65

Programs and Projects

  Programs are comprehensive social change systems that are generally broad in scope with long term goals  LBJ’s Great Society Program Projects are delimited implementation of specified program goals usually bounded in time and space by statute  The Texas Workforce Commission or more specifically, the Upper Rio Grande Workforce Development Board

Evaluation on the “Pyramid Scheme”

 Evaluation is rarely done of entire programs, it is commonly found a the level of projects   Elites don’t mind evaluating others… so long as they are not evaluated in turn As a practical matter, even socially aware researchers must comply with this unstated basic structure of evaluation research

Navigating the Political Landscape – Anticipate or Be Tackled

 Keep the following basic questions about political opportunity structures in mind when proposing to engage in policy evaluation and policy formation as an SMO member  How is formal power distributed?

  Power brokers Linkage figures     How is informal power distributed?

 Power brokers  Linkage figures What are the relations between formal and informal power structures?

  Which is more powerful?

Which is your SMO better connected to?

What are the controversial “sore” issues?

 Degree of polarization   Elites Public Where does the project fit into the community’s political cleavage structure?

 Will it build bridges or dig ditches?

Taking Research From the Activist’s Perspective

  To many activists, the groundwork that they do to prepare for a successful action campaign is not “academic” research This view is slightly erroneous

Old Research Paradigm

 Classical academic research is to be value neutral   The researcher is to have no “horse that he/she is backing” This is to avoid slanting questions, design, and methods to get “desired answers”

Post Modern/Realistic Research Paradigm

 No one is value free. The best we can hope for an open accounting of the researcher’s biases   This means that there is no reason that a frank and open activist cannot do quality research There is really no such thing as “in too deep,” unless the researcher decides to not reveal how deep they are in

Intensive and Extensive Approaches

  Intensive – case study/qualitative approaches concentrate in detail on one or a few examples of a phenomena Extensive – survey/quantitative approaches try to discern general patterns of behaviors or events

Intensive

 The goal is really to get deep enough into the situation to derive causal predictions  By getting into intense scrutiny, we try to eliminate alternate explanations for a an event   With enough scrutiny, we also can become quite familiar with possible history and maturation processes that extensive analysis tends to gloss over The problem with this approach is that findings tend to be idiosyncratic (unique to the case)

Extensive

 The goal here is generalizability based on causal hypotheses gained from initial intensive studies   By dealing with repeated observations over space and time, we can make a case for the observed connection being a causal relationship or law of human behavior The problem with such studies is that findings rarely investigate extensively multiple lines of causality  One line is preferred and any others are by default either used as controls or are not even investigated

Project-Based Research Cycle

 The project-based research cycle that Stoecker (2003) uses is akin to the logic described in Levin’s research cycle (see CRIJ 3300 powerpoints) Evaluation Implementation Diagnosis Prescription

The Goals are Deceptively Simple

     Engage community members in evaluation Diagnose community needs and strengths Define potential solutions that a community would find acceptable and consonant with community norms Use the community evaluators to help gain acceptance of the implementation of the selected solution(s) Evaluate achievement of objectives according to both sponsors and community clients

Finding One’s Place in the Cycle

    Diagnosis     New services are in demand New problem exists, cause unknown General need to be “in touch” with community clients Strategic planning Prescription   Finding best practices (common solutions) for our problem Seeking to decide if best practice(s) will apply to our community  Seeking to efficiently mobilize SMO and service provider extant resources Implementation   Moving aspect of SMO to the fore in aiding community Using POS (political opportunity structure, see Tarrow 1994) to get policy enacted   Public Private (corporate) Evaluation   Seeking to find if our project had an impact Seeking to determine if a change in strategy or tactics is needed to meet SMO goals and community needs Adapted from Stoecker 2005, p.76.

Who are These Community Persons?

     Staff and volunteers of local social movement organizations and movements The academic institutions and government institutions that house evaluation units   Interested college students that either research the community out of a personal connection or are committed to evaluation projects as part of their degree Academic researchers who are engaged in community service or contract-research with sponsors of community affecting projects Funding organizations Service providers that do not house evaluation units General community members who come in for a wide variety of personal reasons

Who are These Community Persons? (cont’d)

Government or Academic Institution Service Provider SMO General Community that Has Both Assets and Needs SMO Service Provider Government or Academic Institution Linkage Adapted from p. 46 of Stoecker (2005)

Staff and Volunteers of SMOs

1  These folks are the foot soldiers of project based evaluation – the civil society version of street level bureaucrats  They have the history and maturation knowledge essential for good research design and implementation 1 SMO – Social Movement Organization, the general class of organizations that community organizations belong to.

College Students and Faculty

 These are the “specialists” or “guns for hire”  They have the methodological expertise to merge with the case expertise of SMO members to complete the designs and implementation plans  Keep in mind, the hero of “Have Gun Will Travel” was not a callous fellow. He was actually an idealist. Just because you are a methodological expert does not mean that you are aloof

General Community Persons

 These folks add perspective    SMO members tend to be a bit overcommitted at times Academics may not know enough facts on the ground Thus incorporating additional views can be a real bonus, if done with a view to improving the study design

Steps for Doing Project-Based Research

     Choosing questions Designing the methods Data collection Data analysis Reporting findings

Choosing Questions

  The selection of what to ask is always a negotiated process between stakeholders The difference with project-based research is the relative emphasis one puts on the stakeholders  Here community stakeholders take moral (if not actual) priority over financial and institutional stakeholders  Theoretically the people matter most, only after their needs are addressed do the considerations of finances and legal mandates come into play

Designing the Methods

   Design can be top-down or bottom-up Top-down – prefabricated designs are placed in many contexts to insure comparability and reliability Bottom-up – using community input contextually relevant designs are employed to insure validity and consensual implementation of studies

Data Collection

 A personal favorite…  I like to use students to do phone surveys and intercept surveys. The students generally belong to the El Paso “community”  This increases their ability to converse in a linguistically appropriate manner with respondents  I would be abysmal at doing such interviews myself and hiring folks from even a prestigious firm like Princeton Research Associates would not make situation much better

Data Analysis

  Community contact brings nuance to the analysis  Non-SW US surveyors rarely understand the ire they raise when not considering Latino/Hispanic a race History and maturation processes for time series analysis are fundamental  The totally bungled testimony by ASARCO’s expert witness on the health impacts of reopening the plant foundered on the out of towner’s ignoring the impacts that the plant would have on Cd. Juárez and that Juárenses would make it an issue at an El Paso hearing

Reporting Findings

  While the word of an impartial judge is the epitome of legal jurisprudence, it does tend to lack for the persuasive impact one might desire  Thus the classical academic discourse of findings lacks as Israelis say, “Zazooah” or “Ooomph” Having a trusted community member express findings that are counter to their known positions on an issue is both highly credible and attention-getting  The challenge is getting a reputable community member to express such counter-ideologicals  The best way to do this is to have credible community members from all sides involved in the entire evaluation process so that they buy into the end products

So How Do We Bring the Community in?

   Try to be useful Use a multimethod evaluation style Listen as well as talk, learn as well as teach

Useful

  Not every evaluation need be stripped down to only the quintessential nuts and bolts of measurement and data analysis. In fact, to do so is generally counterproductive Like in all genuine forms of politics, logrolling is an essential practice  Trade of services – as part of the evaluation, provide some community needed functions   Help organize community archives Help generate donor lists

Multimethod Evaluation

   Despite my desires to the contrary, not all of us are statisticians at heart… nertz!

Thankfully, nor are all of us folklore specialists like my sister-in-law We need both extensive/quantitative and intensive/qualitative methods

Multimethod Evaluation (cont’d)

  As my favorite Aesop’s fable goes, the ability of three blind men to describe a elephant/pachyderm depended on two things   Touching multiple sites of the “big tusks” (elephant) Coordinating findings about the “wrinkly skinned” (pachy-derm) critter So do our evaluations

Don’t Be Your Professor or a College Freshman

   Dr. Levin is a notorious talker and a lousy listener College freshmen are notorious for not actively engaging in classroom discussions Genuine collaboration needs a maximization of all parties to both listen and talk

What to Think About When Collaborating with Community SMOs

   SMO skills SMO financial and personnel capacity Compatibility of evaluation with SMO goals

SMO Skills

  By default, SMOs ought to be able to contribute  Community member contacting expertise  Street credibility In addition, SMOs may be able to contribute  Archives  Funder/donor databases

SMO Financial and Personnel Capacity

     Clerical staff Surveyors Data entry specialists Historians Morale boosters – never underestimate the impact of a warm relationship on research productivity

Compatibility of Evaluation With SMO Goals

 Try to limit the input on evaluation goals to consensus goals that SMOs from several sides can agree on  This requires you to be slightly vague   This does not mean you lie!

This means keeping the goals to what is to be evaluated. Refrain from goals about what is to be done with the findings

Flipping the Coin to the Other Side – SMO Questions

    Can the academic/institutional evaluator listen to the SMO when needed?

Can the academic/institutional evaluator keep deadlines?

Can the academic/institutional evaluator speak to the SMO in its own language?

What credentials does the academic/institutional evaluator have?

Can the Evaluator Listen?

 Despite Dr. Levin’s notoriety, the Planned Parenthood Center of El Paso decided to take the plunge.   Since that decision, Dr. Levin has learned a new vocabulary of three letter words If the formal evaluator can’t listen, the SMO should drop the relationship  Otherwise they risk having “words put in their mouth”

Can the Evaluator Keep Deadlines?

   In the period between retiring from the federal service and prior to starting at IPED, I have had nothing but deep schooling in the inability of academics to keep schedules  It is a norm for academics to     Come late Turn in reports late Turn in budgets late Never write contracts out fully Solution 1 – work only with academic research units  They are built to work with federal and state budgeting and grant application schedules Solution 2 – work with private research units that share interests with your SMO  They too generally work on business schedules

What Credentials Does the Evaluator have?

  Credentials are a useful way to separate out the wheat from the chaff, but have one serious deficiency – young talents/institutions have limited credentials Get as much input from the evaluator’s colleagues on the evaluator’s abilities as possible

Tackling the Four Aspects of Project-Based Research

 Diagnosis, Prescription, Implementation, and Evaluation are all worthy of discussion on their own. Thus the remainder of this lecture, like the Stoecker text, will address them separately

Diagnosis

    Forming a core group Needs assessment Asset mapping Melding needs and assets Evaluation Diagnosis Implementation Prescription

Forming a Core Group

 All social policy projects, like most complex games, require teams. Project success hinges on building a group that can  Survive mild degrees of internal conflict   Has enough breadth of external contacts to build allies and exploit POS Either intentionally links in with the existing issue arena institutional structures or intentionally decides to “work around” the existing issue arena institutions

Needs Assessment

 As discussed last week, needs assessment is a complex form of

evaluation

 In this respect, I differ with Stoecker, who views only outcome evaluation/impact assessments and process evalutations as evaluation

Extensive Needs Assessment

 Extensive needs assessment  This is essentially what we reviewed last week  Surveys   Census data SWOT analysis

SWOT Analysis

 Survey of Strengths, Weaknesses, Opportunities and Threats  This is a focus group approach  Bring in core stakeholders for a few 1-2 hour discussion sessions to    Discussion 1 – Generate a list of SMO or community successes and failures within a relevant time frame Discussion 2 – Generate a list of SMO or community strengths and weaknesses within a relevant time frame Discussion 3 – Maximize strengths and avoid weaknesses by utilizing POS of issue arena

Intensive Needs Assessment

   Beyond the quantitative assessment of community needs from surveys or census data exist the views of informed stakeholders The basic SWOT method does not address the perceived depth of needs, polarization on goals, etc Intensive assessments will add greater emphasis in SWOT to building lists of ranked priorities for needs  Weaknesses and threats will be prioritized on which need to be addressed most

Asset Mapping

    Some of us find a glass of water half empty (your Prof.), others find the same glass to be half full (his pal Marika). It takes both perspectives to realize what real potential the glass holds Asset mapping is all about seeking to know what resources the general community, relevant SMOs, and service providers have as opposed to what they need The goal behind asset mapping is to sensitize communities to their own potential to solve community issues without government largesse Appropriate tools include surveys and census data

Melding Needs and Assets

    In truth, the best approach to a diagnosis of a body politic is the same as of a body – you need to know both what is healthy and what hurts In the course of plotting out what hurts, you know what might be healthy In the course of plotting out what is healthy, you know what you can rely on to deal with what hurts It is much more sensible to approach a community diagnosis with a mind to seek to find the gap between internal needs and assets. This will avoid the overstatement of community and SMO helplessness that needs assessment often encouages

Prescription

    Service versus policy Finding alternatives Evaluating alternatives Choosing an alternative – C-B/C E Analysis Evaluation Implementation Diagnosis Prescription

Service Versus Policy

Service  Inward-focused on community     Concrete plans Narrow application to a specific goal and context Can have policy changes in it Think of this as an experiment in Kuhn’s (1970) sense Policy  Outward-focused on setting agenda for best practices solution to problem   Abstract rules that service prescriptions will fill in Wide application  Think of this as a paradigm in Kuhn’s (1970) sense

Finding Alternatives

 To come up with a vision that is broad, long-range, and has substantive meaning     Dig in deep in the existing literature  Use academic books, online academic journals, and research websites Work with trade publications  In many service professions the federal government sponsors trade publications. So do unions. Find the union websites and you will find the publications Ask friends in the field for their experiences  We all know someone who “knows someone.” Screw up your courage and ask around Brainstorm with stakeholders   This is another focus group approach Beware of groupthink!

Evaluating Alternatives

      Seek the field recognized best practices  Often the trade publications are better sources for overall assessments of what works and what are duds Pull evaluation criteria from prior formal evaluation studies   While specific criteria are not always available for a new issue, the basic methods (C-B analysis) and measures ($, QLI, etc) are very adaptable If you have access, convene a panel of experts Add specific standards from the theoretical literature Bring in the community context   Different communities have different baseline conditions, what most folks define as poor in El Paso is really “poorer” than what most folks define as “poor” in Chicago This will also help to establish the practical limits of the expected impacts Bring in the project’s stated goals Seek the stakeholder’s input for measurement criteria   After all, it is their lives that are impacted the most Expect this input to be political in nature  This can help to integrate dissenting groups early in the process though

Choosing an Alternative

    Rank criteria   When resources are scarce, the easiest method is bottom line priority Other methods are paired comparisons and Q-sorts Calculate benefits and costs  We will go into this in detail with Levin and McEwan (2000) later this semester Do forecasts  We will look at forecast modeling next week as part of the REMI and REDYN models Do decision tree analysis  We will cover this in Ch. 12 of Stokey and Zeckhauser (1978) during November

Implementation

   Research as action Community research Target research Evaluation Diagnosis Implementation Prescription

Research as Action

  Organizing the community on its own behalf  The goal here is to use the project to bring the community to support SMOs and service providers – “Help us to help you”  Getting Hudspeth residents to support their public libraries Knowledge for its own sake  To get a community more aware of its own problems so as to address them more effectively  Educate an AIDS susceptible community about safe sex – the PPCEP-IPED project for the CDC

Community Research

    History  Most communities have persons and local archives – the Rogers Park Historical Society has a great photo of some 1900s farm horses fertilizing what would become David’s childhood backyard Living documentaries  The community’s knowledge embodied via plays, “home movies,” exhibitions, local cable TV shows Community media  Local newspapers, the internet, and local cable GIS, ARCView, and “Mapping the Neighborhood”  The National Institute of Justice pushes crime mapping heavily on us poor criminologists  NIJ loves to fund GIS based research in to the epidemiology of crime

Target Research

  This is where you research an organization or policy in order to target it as needing to be changed by public policies This is an essentially confrontational method

Target Research (cont’d)

 Types of Targets   Government (agencies)  Texas Women, Infants, and Children Program help service website speeds Corporations – includes financial corporations    Wells Fargo lending practices, ASARCO Nonprofits  Boys and Girls Club in NE El Paso embezzlement dispute Foundations  Kaiser Foundation’s pattern of choices in spending money on health care issues in the US tends to avoid questions about reassessing the valuation of medicines, but tends to push questions about supplementing purchaser ability to pay – in the scale of things, this is not a great sin, but it is a needless limitation on research

Government

   Governments are the typical targets of research since they are the primary source of service provision and evaluation funds These evaluations can range from purely pro forma validation to highly investigative A key tool for such research is the federal Freedom of Information Act (FOIA) and its state and local equivalents  You can open most government records upon request, if the Homeland Security bills have not closed them off

Corporations

  Corporations are another typical target Their actions are often traceable through stockholder reports, Equal Employment Opportunity Commission complaints, National Labor Relations Board complaints, local Better Business Bureau complaints, and financial area specific databases

Evaluation

   External versus participatory Outcome versus process The participatory approach Evaluation Diagnosis Implementation Prescription

Evaluation

  As we discussed last week, evaluation can come in many forms, from early stage needs assessments and formative evaluations to the more end stage process evaluations and impact assessments Stoecker shows a preference here for outcome evaluations/impact assessments, but I think this has more to do with the organization of the text than that he rejects needs assessment as being a forma of evaluation

External Versus Participatory

External  Controlled by someone not directly involved in the project to avoid the evaluator “going native”  Use     When project stakeholders are riven by factions When sponsors or agencies demand it When time is an issue When highly specialized skills are required    Participatory Partly controlled by persons directly involved in the project  Use Full control would really have a bias problem – most of us really cannot self critique    When there is a manageable consensus on goals and methods When SMOs and service providers truly want to get critiqued When SMOs and service providers have time and skills to commit to project evaluation

Outcome Versus Process

Outcome  Focus is on the net impact of a project on the welfare of a target community   Tends to be holistic in evaluation measures  Many measures will be combined to present a “complete picture” at a single point in the project – the “end”  No social project has a real “end” unless all subjects exposed and those they contact are dead – e.g. a whole community is exterminated Will use standard array of social, economic, and biomedical indicators of well being – income, education, leisure time, toxic content of air or soil, incidence of cases of disease etc Process  Focus is on the steps towards reaching an outcome   Can be incremental in evaluation methods  Often multiple methods and points in the process will be examined and adjustments in both process and evaluation will be done “on the fly” Will use array of efficiency indicators – service time, number of cases processed, complaints registered, etc

The Participatory Approach

 SMO members, service providers, and general community members can be fruitfully involved in      Determining the question Choosing the methods Gathering the data Analyzing the data Presenting the data

Determining the Question

  While a specialist has a prefabricated set of questions to ask about a project, those most engaged in the context will be able to add specific questions or eliminate ones that are unlikely to show sufficient variance due to context The naysayers among the stakeholders may be co-opted at this stage by adding a question or two

Choosing the Methods

  Again, the specialist will have a prefab set of common indicators of project success Stakeholders will know which are practical to obtain, which are expensive but manageable, and which are simply boondoggles

Gathering the Data

  Like Dorothy once said “There’s no data like free data”… or was that “There’s no data like home cooked 2 data”? Anyway, the best data is that which is   High fidelity Low cost Stakeholders who have an interest in getting useful  evaluations will be the persons best in position to Access original source data  Be/recruit volunteer labor 2 This is not an instruction to cook (fake your data). If you do this, I and every other evaluator worthy of the title will hound you to your grave for dishonoring our profession.

Analyzing the Data

    Herein lies the evaluator’s best chance to be an educator  While participation in other stages may involve education, it tends to be more about alerting people to skills they already possess, not giving them new ones As a teacher of evaluation methodology for the MPA program and the Criminal Justice program, I can attest to you that the skills needed for being a skilled evaluator are in short supply in El Paso. You have a valuable commodity, but like most forms of education, it gains value by teaching your consumers how to use it and do rudimentary stages of it on their own Have the SMO members, service providers, general community members, and other interested stakeholders come in for     Training on SPSS, SAS, STATA, etc A working knowledge of what the term “statistically significant” means Basic cost-benefit/effectiveness analysis Decision analysis Remember, the idea is to give them enough to  Make them aware of the potential benefits to them from using evaluations  Make them aware of the need to bring you or a colleague in for specialized analysis services  Yes, I guess this is saying that we want you to get people addicted to evaluation

Presenting the Data

  Picture this: a city budget analyst stands in front of a city council meeting and is roundly booed by all and sundry for giving them news they don’t want to hear about programs that must be cut to meet the council’s demand for a balanced budget without raising taxes. Sound familiar?

Avoid! How? Simple! Bring the SMO members, service providers, and other stakeholders into the presentation process    That way they are unlikely to boo, since they would be booing themselves They would be more understanding of the actual implications of the findings It allows findings to leak early… yes, leaks can be good sometimes  Policymaking subsists on strategic leaks to co-opt and demobilize opposition

Bringing it All Around Again

  Having touched on the high points of the project-based research cycle, let’s be aware of some of its limitations  It tends to be too darn inward focused if the specialist is edged out to the margins of the process  It is the evaluation specialist’s job to look beyond the project to the program and to social welfare as a whole    It cannot and targets from criticism  Participatory process tend to marginalize experts in the name of an ersatz “average Jane/Joe” should not entirely insulate the evaluators or evaluation Not all projects are created equal. Let’s face it, some projects stink and should be terminated. Someone has to take the heat for the flawed design, implementation, or less than kindly breaking of the bad news It tends to be antagonistic towards traditional service providers (e.g., government agencies)  Sorry “Neos” of the world, but the government is rarely a “Matrix” enmeshed in extracting a parasitic existence from you Nevertheless, the project-based research cycle’s participatory emphasis is well-worth the risks and limitations enumerated above