Transcript Document
National Institute on Disability and Rehabilitation Research
Understanding the Government Wide Shift in Accountability: Using Performance Management to Plan for Outcomes & Demonstrate Results
Technical Assistance Guide Prepared for Web Cast on AT Act Outcomes Wednesday, October 22, 2003 by Margaret L. Campbell, Ph.D. National Institute on Disability and Rehabilitation Research U.S. Department of Education
Purpose of Technical Assistance Guide National Institute on Disability and Rehabilitation Research
2
To provide NIDRR staff, grantees, applicants and reviewers with a comprehensive resource for understanding the government-wide shift in accountability and how the concepts and tools of performance management can be used to:
Plan for outputs & outcomes
Improve programs
Monitor progress, and
Document results and communicate successes
National Institute on Disability and Rehabilitation Research
Part 1:
The “What and the Why” of the Accountability Shift
National Institute on Disability and Rehabilitation Research
4
What is the Government-Wide Shift in Accountability?
•
Shift from primary focus on:
Dollars (how much is spent & on what)
Activities & processes (what you are doing to/with whom and how well you are doing it)
Productivity – (how much are you doing)
•
To an expanded focus on :
Performance - (are you meeting stated objectives and standards of quality), and
Results – (what you are achieving and who’s benefiting?)
Shift in Accountability is Not “Either- Or” Situation National Institute on Disability and Rehabilitation Research
5
Rather amounts to an added dimension of accountability for “results.”
Still must meet accepted standards of practice for:
Continuous quality improvement
Program development and delivery of services, and
Consumer involvement
Systems change & advocacy
National Institute on Disability and Rehabilitation Research
6
What Does “Accountability for Results” Mean?
In general, “accountability for results” refers to accomplishments that are under the direct or indirect influence of project resources and activities & occur within the boundaries of a funding/budget cycle.
Within the NIDRR context, this translates into accountable for three types of results:
Outputs
– planned and produced to date
Short-Term Outcomes
– anticipated and actual, and in some cases
Intermediate Outcomes
or actual – anticipated
Origins of the Accountability Shift in Government: GPRA National Institute on Disability and Rehabilitation Research
7
In 1993, Congress passed the Government Results and Reporting Act, which established uniform requirements for strategic planning, annual planning and reporting, and agency-grantee partnerships.
Key Provision of GPRA: Provides a new definition of program evaluation – “an assessment, through objective measurement and statistical analysis, of the
manner
programs and
extent achieve
to which federal intended objectives.”
National Institute on Disability and Rehabilitation Research
8
Recent Policy Initiatives Reinforcing the Accountability Shift
The President’s Management Agenda (PMA) FY 2002, with its emphasis on:
Performance management & measurement
Objective criteria to assess program results (i.e., the PART)
Performance-based budgeting
The Dept. of Education’s New Strategic Goals , 2002-2007, with an emphasis on:
Transforming Education into an Evidenced-based Field.
National Institute on Disability and Rehabilitation Research
Part 2:
The Language and Logic of Outcomes Planning and Managing for Results (Key Concepts and Definitions)
National Institute on Disability and Rehabilitation Research
10
What is Meant by “Managing for Results?”
“Managing for results” requires developing a new mindset based on outcomes-oriented goals and performance measures, as opposed to activity-oriented goals and objectives .
The primary focus is on planning for what outputs and outcomes will be achieved not on what activities will be conducted.
For most of us, this involves re-expressing existing program goals and objectives the language of planned outputs and anticipated outcomes.
into
National Institute on Disability and Rehabilitation Research
11
Concepts and Tools You Need to Succeed
Performance management
Performance measurement
Performance goals & measures
The Performance Spectrum
Logic modeling – i.e., reverse mapping
Components of Performance measurement
Target systems and/or populations
National Institute on Disability and Rehabilitation Research
12
What is Performance Management vs. Performance Measurement?
Performance management is an approach to planning and evaluation that utilizes the concepts and tools of performance measurement and logic modeling to identify performance goals and assess progress towards goals.
Performance measurement “involves the ongoing monitoring and reporting of program accomplishments, particularly progress towards pre established goals” (see GPRA requirements).
What are Performance Goals?
National Institute on Disability and Rehabilitation Research
13
According to OMB, performance goals are the desired levels of performance expressed as measurable objectives against which actual achievement can be compared.
To be complete, performance goals should incorporate performance measures and performance targets into timeframes .
National Institute on Disability and Rehabilitation Research
14
Definitions of Performance Goals (Cont’d.)
Performance measures are the quantitative or qualitative metrics and indicators that are used to demonstrate progress towards goals and document results/ accomplishments.
Performance targets are the quantifiable or otherwise measurable characteristics that tell how well a program or project must accomplish a performance measure.
National Institute on Disability and Rehabilitation Research
15
The Spectrum of Performance: Outcomes-Oriented Goals
Although the PART emphasizes outcome goals , that focus on changes and/or improvements in the target system, performance goals can also be stated in terms of “planned outputs” (and sometimes “quality of activities”).
The key : All performance goals should clearly distinguish between components of performance and there should be a logical connections among them with outputs supporting outcomes (i.e., outcome-oriented performance goals).
What is Logic Modeling?
National Institute on Disability and Rehabilitation Research
16
A vehicle for dialogue, planning, program improvement and evaluation
A graphic representation or “blueprint” of the key elements of a program or project, and how it will work under certain conditions to “solve” identified problems.
A helpful tool for identifying outcomes and depicting the “chain of events” that link inputs and activities to outputs and outcomes.
What Does A Basic Logic Model Look Like?
Prepared by the NIDRR PPB&E Division 1
HOW Inputs & Infrastructure Program Activities PROGRAM DELIVERED Outputs Span of Accountability for Results WHY Short-Term Outcomes Intermediate Outcomes Customers Who Use Outputs RESULTS FROM PROGRAM Longer Term Outcomes EXTERNAL CONDITIONS INFLUENCING SUCCESS (+/-) ANTECEDENT/ MEDIATING MEASURES
1 Source: Adapted from McLaughlin, J. A. & Jordan, G. B. (In Press/2003) Logic models: a tool for describing program theory and performance, in Wholey, et. al.:
Handbook of Practical Evaluation
. Jossey-Bass. The lead author may be contacted by email at: [email protected]
Note:The above logic model does not include the standards by which the quality, relevance, and productivity of performance will be measured and evaluated across the spectrum. For NIDRR grantees this is provided by the “centers of excellence” (CoE) criteria. Updated on June 27, 2003
What are the Components of Performance Measurement?
National Institute on Disability and Rehabilitation Research
18
Inputs & Infrastructure: The human and financial resources and systems needed to conduct a high quality, outcomes-oriented program or project.
Examples: agency priorities and requirements, host institutional support and leveraging, staff competencies and expertise, management practices and evaluation plans, partnerships and collaborations, and previous accomplishments.
National Institute on Disability and Rehabilitation Research
19
What are the Components of Performance Measurement? (Cont’d)
Program Activities: Are the action steps, tasks, procedures, and services performed in conjunction with implementing your planned program of public awareness, technical assistance and training, Outreach, and interagency coordination to carry out objectives and produce results.
National Institute on Disability and Rehabilitation Research
20
What are the Components of Performance Measurement? (Cont’d)
Outputs
: Are the direct results of program activities and consist of the findings or conclusions, products and services produced and reported to external audiences.
Outputs typically are reported using statistics or
quantitative counts
– e.g., the # of individuals trained, newsletters distributed, publications produced, & participants attended.
But they also can be described
qualitatively
terms of the nature of the findings or discoveries. in
National Institute on Disability and Rehabilitation Research
21
What are the Components of Performance Measurement? (Cont’d)
•
More On Outputs:
Outputs are important in their own right as indicators of productivity; they also are the essential building blocks of outcomes .
However, not every output will have a corresponding outcome, nor should it.
What are the Components of Performance Measurement? (Cont’d) National Institute on Disability and Rehabilitation Research
22
Utility: Although not an official component of the performance spectrum as defined by GPRA, NIDRR has included utility because of its centrality to our Mission, and because demonstrating the “perceived usefulness” of outputs to sub-groups within the broader target systems constitutes an important bridge between outputs and short-term outcomes.
National Institute on Disability and Rehabilitation Research
23
What are the Components of Performance Measurement (Cont’d)
Outcomes: Are the anticipated or actual effects of program activities and outputs and constitute changes or improvements in the target systems being served. Although the emphasis is on planning for outcomes, outcomes may also occur as unexpected changes.
Outcomes are the most complex component of PM and are difficult to define outside of the context of a specific program and target system.
National Institute on Disability and Rehabilitation Research
24
What are the Components of Performance Measurement (Cont’d)
Factors to take into account in planning for and documenting outcomes:
The basic outcomes equation is : Outcomes = change/improvement in the target population or system, which is a function of: Outputs + Documented External Use
Outcomes typically occur sequentially over time and, as a result, may affect more than one target system. Because of this, we refer to “chain of outcomes ,” starting with short term outcomes and progressing to intermediate and longer-term outcomes.
National Institute on Disability and Rehabilitation Research
25
Types of Outcomes
Short-term outcomes are expected or actual changes or improvements in the identified target system that are more under the direct influence of center activities. Short-term outcomes represent the first level of change that must occur in order to bring about intermediate outcomes. Typically, short-term outcomes are focused on changes/improvements in the learning & knowledge system at the individual or environmental level of analysis.
National Institute on Disability and Rehabilitation Research
26
Types of Outcomes
Intermediate outcomes are the “gold standard” in terms of accountability for results. They consist of expected or actual changes or improvements in the “action” system that occur in part as a result of the use or adoption of project outputs. Unlike short-term outcomes that occur under the direct influence of program activities, there may be other causal factors contributing to the achievement of intermediate.
National Institute on Disability and Rehabilitation Research
27
Types of Outcomes (Cont’d)
Longer-term outcomes are the desired end-results of a program, and constitute changes or improvement in the overall condition of a population or system. Given their scope, longer term outcomes usually take more than one funding cycle to achieve, and therefore are not required under the “accountability for results” umbrella. Their primary function is to serve as critical anchor points in the planning process.
Examples of Types of Outcomes (Changes or Improvements) at the Individual Level National Institute on Disability and Rehabilitation Research
28
Focus on Learning and Knowledge Focus on Action Awareness Knowledge Attitudes Skills Opinion Motivation Function Behavior Practice and/or Clinical care Advocacy Choice or decision-making Empowerment/Sel f-Efficacy Focus on Conditions Physical or Mental Health Status Employability Independent Living Community Integration, Participation Economic Self Sufficiency Quality of Life
National Institute on Disability and Rehabilitation Research
29
Examples of Types of Outcomes (Changes or Improvements) at the Environmental Level Focus on Learning and Knowledge Level or Availability of New Knowledge Models, frameworks or guidelines Methods or Tools Products and/or devices Services Technologies Focus on Action Use or Adoption of New: Knowledge Models, frameworks or guidelines Methods or Tools Products and/or devices Services Technologies Standards Policies, laws and/or regulations Focus on Conditions Capacity of systems or society: Employment Rates Rates of poverty among PWD Rates of Secondary Conditions Rates of Premature Mortality Delivery Systems Environmental Access (barrier elimination) Rates of participation
National Institute on Disability and Rehabilitation Research
30
What is a Target System?
Formally defined , a “target system” refers to a unit of analysis or bounded set of interacting elements. In the NIDRR context , a target system or population refers to the level(s) of the environment and/or to the group(s) of individuals that program activities and outputs are intended to affect.
Specifying the particular sub-groups or sample populations within the broader target system that your program intends to affect is key to identifying anticipated outcomes.
National Institute on Disability and Rehabilitation Research
31
Examples of the Types of Target Systems Typically Involved in NIDRR Funded Projects
Individual level systems : persons with disability and/or their family members, clinicians, service providers, other researchers, policy makers, and industry representatives.
Environmental-level systems : the knowledge base in a field, treatment/practice standards, product development standards, availability of new methods, tools, products, services, and technologies, policies and/or service delivery systems, manufacturers, and societal conditions.
Part 3:
National Institute on Disability and Rehabilitation Research
How to Apply Performance Management Tools to Plan for Outcomes & Manage for Results
Objective of Outcomes Planning
National Institute on Disability and Rehabilitation Research
33
To identify a limited number of high priority, problem-focused, reasonably ambitious and measurable outcomes oriented performance goals that:
Reflect NIDRR priorities or statutory requirements, and
Capture the anticipated effects and benefits of project activities & outputs on identified target populations and/or systems.
Two Ways to Get to Outcomes: Prospective vs. Retrospective Planning National Institute on Disability and Rehabilitation Research
34
Prospective outcomes planning is the preferred strategy, but it requires starting with a “managing for results” mindset and identifying performance goals and measures to guide activities and monitor progress (slides 35- 42 ).
Retrospective outcomes planning is less than ideal, but is how most of us beginning to get our feet wet. It involves identifying outcomes after-the-fact and collecting data to support them (slides .
National Institute on Disability and Rehabilitation Research
35
Prospective Outcomes Planning Step 1: Preparation
Describe the overall problem you are trying to “solve” and specify what part or parts of the problem will be addressed by your program activities (i.e., theory of the
problem clarifies scope of your project).
Identify the specific sub-groups or sample populations within the broader target systems that you anticipate will change/ improve as a result of your program.
Explain how your program activities & outputs will change or benefit the specific target systems or populations you have identified (i.e., theory of the program)
National Institute on Disability and Rehabilitation Research
36
Prospective Outcomes Planning Step 2: Constructing a Logic Model
Translate requirements and objectives of program into anticipated outcomes, starting with longer-term, end-results and working backwards to identify intermediate and short term outcomes (see Worksheet below); Identify the specific target populations and systems to be affected by each type of outcome Plan the outputs that will be produced to bring about anticipated changes in target systems; Design the activities that must be conducted to produce planned outputs; Identify the resources needed to implement activities; and Document the contextual factors that could influence the success of your program.
Inputs and Infrastructure Worksheet for Constructing a Logic Model of Performance for Your Program
Prepared by the NIDRR PPB&E Division
1 Important Tip: When planning, begin with the end in mind and work from right to left.
When implementing, begin with inputs and infrastructure and move from left to right Program Activities Outputs Span of Accountability for Results Utility Short-Term Outcomes Intermediate Outcomes Longer-term Outcomes What resources and management practices are you using, or developing, to conduct planned activities and produce desired results consistent with the priority and program requirements?
What activities are you currently conducting, or plan to conduct, across program requirements to fulfill objectives, produce intended outputs, and assess their relevance and utility for target audiences? How well are you implementing these activities; and what is your progress to date, including problems encountered and actions taken?
What products, services, and information are you providing, or do you plan to provide to target audiences or customers to support anticipated outcomes? And what is your productivity to date?
What evidence do you have of the potential merit and/or usefulness of your outputs?
What changes or improvements will occur in awareness, understanding or knowledge and for which target populations as a result of your program activities and outputs? And, what measurable indicators will you use to track progress towards achieving these short-term outcomes/ performance goals?
Internal and External Factors That Can Influence Success What changes or improvements do you anticipate will occur in behavior, practice and/or policy, and for which target populations as a partial result of the use or adoption of new knowledge generated by your program? And, what measurable indicators will you use to track progress towards achieving these intermediate outcomes/ performance goals? What are the long-term changes or improvements in societal conditions that you hope to contribute to overtime through your program activities, outputs and short-term and intermediate outcomes?
1 Source: Adapted from McLaughlin, J. A. & Jordan, G. B. (In Press/2003) Logic models: a tool for describing program theory and performance, in Wholey, et. al.:
Handbook of Practical Evaluation
. Jossey-Bass. The lead author may be contacted by email at: [email protected]
Note:The above logic model does not include the standards by which the quality, relevance, and productivity of performance will be measured and evaluated across the spectrum. For NIDRR centers and model systems this is provided by the “centers of excellence” (CoE) criteria.
Revised: June 27, 2003
National Institute on Disability and Rehabilitation Research
38
Prospective Outcomes Planning Step 3: Formulate Performance Goals
Convert outcome statements into performance goals by:
Identifying or developing the performance measures that will be used to document progress and demonstrate accountability for results (i.e., outputs, short-term and intermediate outcomes).
Establishing a baseline and identifying performance targets that must be met to achieve outcomes.
Establishing the timeframe within which progress & results will occur.
Prospective Outcomes Planning Step 4: Evaluation of Performance National Institute on Disability and Rehabilitation Research
39
Develop an evaluation plan that identifies the types of evidence you need to collect to determine how well your program is working and if it is achieving the anticipated results.
Specify the sources and methods of data collection you will use to measure performance and demonstrate progress towards outcomes-oriented goals (see
Table on next slide for examples of data sources & methods).
Examples of Sources and Methods of Data Collection for Demonstrating Outcomes National Institute on Disability and Rehabilitation Research
40
Source of Information
Participants (consumers, persons with disabilities)
Family Members
End-users & Other Stakeholders
Key Informants
Practitioners
Documents
Data sets/ public use data files Method of Collecting Information
Usability Assessments, pre/post tests
Observation
Interviews or surveys
Focus Group
Record reviews
Citation analysis
Literature Reviews
Secondary Data Analysis
National Institute on Disability and Rehabilitation Research
41
Prospective Outcomes Planning Step 5: Evaluation of Performance
Integrate your performance management plan into the ongoing operation of your program activities to measure performance, collect outcomes data and monitor progress.
National Institute on Disability and Rehabilitation Research
42
Prospective Outcomes Planning Step 6: Analysis & Documentation
Analyze the qualitative and quantitative performance data to document progress towards anticipated outcomes.
Interpret the data to determine if an outcome has occurred or is likely to occur within the specified timeframe. Making this determination relies on qualitative judgments of quantitative and/or qualitative data.
Document results and communicate accomplishments to external audiences is an evidence-based claims-making process, in which grantees present evidence to support the claim that their program activities & outputs have resulted in changes/ improvements in the identified target system.
National Institute on Disability and Rehabilitation Research
43
Retrospective Outcome Identification: Problems with Approach
Missing pre-established performance goals to guide activities and outputs
Missing specification of target systems where outcomes are expected to occur
Missing a logic model that clarifies the interconnections among activities, outputs and outcomes and between short-term and intermediate outcomes
Missing performance measures and systematic outcomes data to monitor progress and demonstrate results
National Institute on Disability and Rehabilitation Research
44
Retrospective Outcome Identification: Suggested Implementation Strategies
Form an AT Act Work Group to translate statutory requirements into a “limited
number of high priority” (i.e., 2-3) and
measurable outcomes-oriented performance goals for each core activity: public awareness, TA & training, outreach, and interagency coordination. (suggested # of outcomes is
Review and categorize accomplishments produced to date into outputs and types of outcomes (short-term, intermediate and long-term).
National Institute on Disability and Rehabilitation Research
45
Retrospective Outcome Identification: Implementation Strategies (Cont’d.)
Link accomplishments to newly formulated performance goals and identify the types of data available and the data elements needed to demonstrate progress towards outcomes and document the contribution of AT Act project activities and outputs.
Revise web-based annual performance reporting form for FY 2004 to collect necessary data and rethink how existing data can be analyzed to provide evidence of progress or achievement of outcomes.
National Institute on Disability and Rehabilitation Research
46
Common Weaknesses in Prospective & Retrospective Outcomes Identification 1. Lack of specificity in “what” and “where” 2. Too many outcomes and expressed too narrowly or too broadly 3. Outcomes expressed as speculations vs. as expectations 4. Confusion between levels of the performance spectrum (i.e., activities vs. outputs and outcomes and short-term vs. intermediate outcomes) 5. Missing supporting links between outputs and short-term outcomes 7. Missing evidence of outcomes
National Institute on Disability and Rehabilitation Research
47
Exercise: What’s Missing in the Following AT Act Project Outcomes?
•
Outreach: Within the last grant cycle, State X provided technical assistance to 4, 127 individual and demonstration of loan of equipment to 1,415 individuals.
•
Training: State X provided 102 training sessions attended by 3,8288 which resulted in increased knowledge levels that have prompted systems to address AT needs and responsibilities within policies and procedures.
National Institute on Disability and Rehabilitation Research
48
Exercise: Illustration of Reformulated Outcome Statement
•
Original Statement Systems Change Intermediate Outcome: State AT program worked with the state election officials on the HAVA state plan which resulted in adoption of the FEC 2002 access
•
standards for voting machines.
Suggested Reformulation: Conduct legislative initiatives that result in the development (short-term outcome) and adoption by states (intermediate outcome) of new standards for voting machines to increase access for adults with moderate to severe disabilities.
Part 4:
National Institute on Disability and Rehabilitation Research
Lessons Learned:
The Utility and Challenges of Implementing Performance Measurement
National Institute on Disability and Rehabilitation Research
50
Benefits of Utilizing Performance Management
Provides a common language and integrated framework to facilitate:
Strategic planning & program design
Identification of performance goals
Program integration – coordinates the contributions of multiple projects and tasks
Objective measurement of progress towards goals
Program monitoring & improvement
Program accountability -- documentation of results, and
Communication of success stories
National Institute on Disability and Rehabilitation Research
51
Challenges of Applying Performance Management Concepts & Tools
Developing a “managing for results” mindset .
Incorporating requirements of performance measurement into the traditional standards of excellence for service delivery, consumer involvement, and productivity.
Learning to re-express core program objectives into the language of outcomes-oriented performance goals (outputs and outcomes).
Deciding what evidence you need to collect and how to measure performance and determine how your program is working (i.e., quality and relevance/utility) and if you are achieving planned outputs and anticipated outcomes.
Challenges of Applying Performance Management (Cont’d) National Institute on Disability and Rehabilitation Research
52
Integrating the tools of performance management into the operation of your program.
Using the information gathered to improve program management, evaluate performance, achieve results, and provide evidence to support claims of short-term & intermediate outcomes.
Documenting results and communicating successes using a “claims-making” process based on qualitative judgments of quantitative and/qualitative data
National Institute on Disability and Rehabilitation Research
53
Lessons Learned from Implementing Performance Management in Business & Government
“What gets measured gets done.”
(Drucker)
“If you don’t measure results, you can’t tell success from failure “ (C. Mindel).
According to OMB, the number one cause of federal agencies getting a low score on the new Program Assessment Reporting Tool (PART) is NOT having results-oriented performance measures to gauge the outcomes of a program ( http://www.transparentgovernment.org/tg/an alysis.htm
).
Lessons Learned from Implementing Performance Management (Cont’d) National Institute on Disability and Rehabilitation Research
54
Accountability for results involves a partnership - NIDRR can’t satisfy GPRA and PART alone! To succeed in the new era of performance-based budgeting, NIDRR needs data on accomplishments from grantees like you.
Key Assumption: All grantees have accomplishments; what missing are the performance goals and outcome data.
Key Assumption: “If you can demonstrate results, you can win public support” (C. Mindel).
National Institute on Disability and Rehabilitation Research