Transcript National Institute on Disability and Rehabilitation Research

Towards KT Measurement:
What Constitutes KT Outcomes
and Outcomes Monitoring in
the NIDRR Context
(An Evaluation Perspective)
Presentation to National Center of
Dissemination of Disability Research
(NCDDR) Workshop:
December 3, 2007
Margaret L. Campbell, Ph.D
Senior Scientist for Planning and Evaluation
National Institute on Disability and Rehabilitation Research
Working Assumptions and
Perspective of Presentation
 Focus is on KT as a set of strategies for accelerating
the uptake of R&D findings to improve systems and
benefit end users – i.e., the desired result of KT is
“getting to outcomes.”
 R&D outcomes do not occur without some type of
KT, including publication.
 Initial questions posed by NCDDR for this segment
of the workshop are in many ways ahead of NIDRR’s
development of KT and integration into planning and
evaluation. Because of this, it is necessary to
reformulate the objectives of this presentation.
Presentation Objectives
 Illustrate how NIDRR’s existing strategic framework can
be used to clarify what constitutes an outcome and to suggest
possible theoretical KT outcomes.
 Describe components of NIDRR’s existing evaluation
framework, explain current strategies used to monitor and
assess the quality of program outcomes, and present
quantitative results from recent Annual Performance
Assessment Expert Review (APAER) process.
 Use examples of “noteworthy” accomplishments from 2007
APAER for Participation and Community Living to illustrate
the kinds of outputs and outcomes grantees are reporting and
whether they correspond to suggested theoretical KT
outcomes.
Part 1: Illustrate Utility of
NIDRR’s Strategic Framework
 NIDRR’s Long-Range Plan and Logic Model help clarify and
identify possible theoretically KT outcomes to be tracked and
evaluated for quality?
 The Long-Range Plan identifies strategic goals and
objectives and provides a roadmap to the kinds of program
outcomes NIDRR is seeking;
 The Logic Model identifies outcome arenas that
correspond to strategic goals, types of R&D outcomes, and
intermediary and intended beneficiaries.
NIDRR’s Strategic Goals, LongRange Plan – 2005-2009
 To advance knowledge through capacity-building:

By increasing capacity to conduct and use high quality and relevant
disability and rehabilitation research and related activities designed to
guide decision making, change practice and enhance the lives of
individuals with disabilities.
 To advance knowledge through research and related
activities:

By generating science-based knowledge, technologies and applications
in order to contribute to policy; change practice; and enhance the lives of
individuals with disabilities.
 To advance knowledge through translation by utilizing
science-based knowledge, technologies and applications:

By promoting the effective use of science-based knowledge,
technologies and applications to contribute to disability and
rehabilitation policy; improve practice; and enhance the lives of
individuals with disabilities.
Brief Background on Logic
Models (Making sure we’re on the same
page)
 What is a logic Model? Graphic blueprint of how a program
is supposed to work to achieve benefits for participants.
Ideally, a logic model depicts both the “theory of the problem”
and the “theory of program.”
 Utility of logic modeling? In additional to serving as a
valuable communication tool, logic modeling ideally helps
researchers, program managers and evaluators:
Chart the pathways by which program activities and
outputs will have their desired effects or outcomes;
Identify “indicators of success” or milestones along the
outcomes pathway that occur with the “span of
accountability” of the program.
Components of NIDRR’s Logic
Model for Research Outcomes
(See Handout)
 Situation or “theory of the problem” to be addressed
 Outcome Arenas -- correspond to strategic goals of the LRP for
CB, R&D and KT and point to where program results occur.
 Types of Outcomes:
 Short-term – advances in understanding, knowledge, skills
and learning knowledge that occur through discoveries,
theories measures and methods, and interventions, products,
devices, etc.
 Intermediate – adoption and use of short-term outcomes to
change/improve policy, practice & system capacity
 Long-term – changes/improvements in overall conditions of
the system or population (e.g., increased access and
improved rates of employment)
Components of NIDRR’s Logic
Model for Research Outcomes
(Continued)
Together, these outcomes:
• Reflect a hierarchy or sequence of anticipated results in
order of program implementation;
• Depict the program’s change strategy; and
• Help to identify areas of possible implementation
failure.
 Beneficiaries
 Intermediate Beneficiaries or Initial Users
 Intended Beneficiaries or End-Users

Lessons Learned in
Implementing NIDRR’s Logic
Model
 Still a work in Progress. Some of the gaps include:
 No specific KT & CB outcomes are identified;
 No depiction of the pathways by which outputs of NIDRR’s
investments in R&D become outcomes and what is the role
of KT is this process;
 Similarly, no depiction of the technical and infrastructural
developments or “milestones” necessary to support
advances in knowledge and changes in policy, practice and
systems capacity. CHIR is addressing this challenge by
identifying “indicators of success” that approximate and
anticipate KT outcomes;
 Stated another way, no acknowledgement that outcomes
evolve over time and do not emerge fully developed and to
scale across a target population of beneficiaries or system.
Possible Future Development of
NIDRR Logic Model
 Ideas for Possible Future Development:
 Elaborate CB and KT outcome arenas;
 Chart pathways by which KT activities and outputs lead to
KT outcomes and facilitate R&D outcomes; and
 Identify “indicators of success” that can be used to mark
progress towards KT outcomes.
 Caveat:
 For the above developments to occur, we need more input
from stakeholders and more data on the performance of the
NIDRR’s new KT centers.
 So what can we do in the meantime?
Proposed Interim Strategy
 We can assume that:
KT outcomes do not differ theoretically from R&D
outcomes in their type or sequence or in the intended target
populations of users and beneficiaries.
 The only differences are in the specific knowledge areas
being advanced and to some degree in the methods used to
achieve outcomes.
 We can use the tools of logic modeling, along with the ongoing
refinements in KT NIDRR is making under Pimjai Sudsawad’s
leadership, and the experience of other federal and non-federal
agencies to suggest possible theoretical KT outcomes that
could be tracked through NIDRR’s existing evaluation system
to determine their frequency and appropriateness.

Building Blocks for Identifying
Possible KT Outcomes
 CIHR’s initial definition and operationalization of KT and
results of recent initiatives described early in this workshop.
 On the U.S. side, results of the T.R.I.P. initiative (Translating
Research into Practice) sponsored by the Agency for
Healthcare Research and Quality (AHRQ) that suggest to
accelerate research into policy and practice research users
need three types of KT outcomes:
 Research results about what works and what doesn’t,
 Knowledge about how others have implemented findings
within their organizations, and
 Knowledge about how the culture and values of an
organization affect the uptake and continues use of the
findings.
Possible KT Outcomes within
the NIDRR Context – For
Discussion Only
 Possible Short-Term Outcomes:



Increased understanding and knowledge of the KT
concept and methods, as reflected in results of KT training
and use and/or adoption of KT outputs;
Rigorous research results about which KT strategies and
methods work and which do not and in which settings and
for which target populations. This would be reflected in the
increased availability of “best practices.”
Increased knowledge about how the culture and values of
research users, including consumers, and organizations
affect the uptake of R&D findings.
Possible KT Outcomes within
the NIDRR Context – For
Discussion Only (Continued)
 Possible Intermediate Outcomes:



Change in R&D policy as reflected in increased funding of
KT initiatives by federal and non-federal organizations;
Change in R&D practice as reflected in increased
utilization of KT strategies and methods by practitioners;
Increased capacity to support the exchange, synthesis and
ethically sound application of R&D findings to change
policy, practice, as reflected in infrastructural
developments at a systems level.
Possible KT Outcomes within
the NIDRR Context – For
Discussion Only (Continued)
 Possible Long-term Outcome:

Reduced time intervals between the generation of R&D
findings and the ethically sound application of findings to
eliminate disparities between people with disabilities and
the general population in employment, health & function,
and participation and community living.
Part 2: Describe NIDRR’s
Evaluation Framework
 Current Status: More emphasis on outcomes monitoring
versus outcomes measurement and evaluation
 Outcomes monitoring constitutes an ongoing system of
reporting and tracking program operations and results
typically through the use of administrative information
systems and performance indicators, and is used to answer
questions that ask for a description of “what’s happening.”
 Outcome evaluation extends beyond monitoring to an
examination of the extent to which outcomes are caused by
the program. Outcomes evaluation requires stronger
evidence and seeks to answer questions like “What
difference does the program make and what about the
program is responsible for the difference”?
Components NIDRR’s
Performance Monitoring &
Evaluation System
 Annual Performance Reporting (APR) form -- primary data
warehouse for NIDRR’s outcomes monitoring and evaluation
system is the.
 Performance Measures – out of 7 programmatic GPRA
measures 2 pertain directly to KT.
 Annual Performance Assessment Expert Review (APAER)
process
 Phase 1 – external quality review of individual grantee
accomplishments documented in APR and related to one of
major life domain of NIDRR Logic Model; and
 Phase 2 – expert panel review of aggregate
accomplishments to assess progress towards NIDRR longterm goals.
How NIDRR Currently
Measures KT Performance:
GPRA Measures
 To assess the effective use of NIDRR-funded scientific-based
knowledge, technologies, and applications, the agency’s
collects data from the APR to determine:
 The average number of publications per award based on
NIDRR-funded research and development activities in
refereed journals.
• Results: For calendar years 2004 to 2006, the average
number of peer-reviewed publication per award for the
five eligible program categories is 2.71, 3.81, and 2.71,
respectively.
 The number of new or improved NIDRR-funded assistive
and universally designed technologies, products, and
devices transferred to industry for potential
commercialization.
• Results for FY 2006 & 2007 available 12-31-07.
How NIDRR Assesses Quality of
Program Outputs & Outcomes -Phase 1 of APAER
 Purpose: To satisfy GPRA R&D measure #2.1, which calls for
"the number of accomplishments developed and/or tested with
NIDRR funding that have been judged by expert panelists to be
of high-quality and to advance the field," the agency contracts
with an outside organization to provide external reviews of
individual grantee accomplishments associated a major life
domain under the Long-Range Plan.
 Focus of 2007 APAER: Participation and Community Living
(P/CL) Domain
 Data: 84 individual accomplishments (76 outputs and 8
outcomes) documented in the APR from 37 awards were
considered eligible for external review based on completeness
of information.
How NIDRR Assesses Quality of
Program Outputs and Outcomes:
Phase 1 of APAER (Continued)
 Methods:


Reviewers were asked to rate the quality and contribution
to the field of each accomplishment using a 5-point scale
for outputs and a 3-point scale for outcomes, with each
scale ranging from no contribution to outstanding
contribution.
Ratings were collected from 12 reviews. To establish a
basis for estimating rater effects, 10 accomplishments
were rated by all reviewers with the remaining 76
accomplishments overlapping across reviewers and rated
by three reviewers. A total of 272 ratings were available
for outputs and 34 for outcomes.
How NIDRR Assesses Quality of
Program Outputs and Outcomes:
Phase 1 of APAER (Continued)
 Data Analysis: Used Many-faceted Rasch Measurement
(MFRM) statistical technique to control for rater effects in
arriving at a composite rating for each accomplishment.
 Using this approach, it is possible:
 To determine if there are significant differences in the
ratings across accomplishments and reviewers and then to
identify which accomplishments are more highly and
which are more lowly rated and which reviewers are more
severe and which are more lenient in their ratings;
 Adjust the ratings based on reviewer severity and produce
a composite rating that is reported in terms of logits (or
log-odds probabilities), which represent equal-interval
scales that that allow for analysis using parametric
statistical techniques.
How NIDRR Assesses Quality of
Program Outputs and Outcomes:
Phase 1 of APAER (Continued)
 Quantitative Results: Of the 84 accomplishments rated:

49 (or 58%) were judged to be of high-quality and advancing
the field based on adjusted composite rating, which is
equivalent to a rating of substantial or higher on the observed
scale.

Of these, only 6 were determined to be outcomes, which may
be due primarily to the way performance data were collected
in the APR with an emphasis on publications, tools and
measures; technology products, and informational products.
 Significance: First time NIDRR has had reliable data to satisfy
GPRA performance measure 2.1, and to use to establish a
baseline against which we can demonstrate progress towards
long-term goals, consistent with PART requirements.
Part 3: Examples “Noteworthy”
Accomplishments: Phase 1
APAER
 Preliminary Impressions of Qualitative Data: Of the 43
outputs and 6 outcomes judged to be advancing the field:
 The vast majority were the result of R&D activities, with
few direct links to KT activities. Again, this may be due to
the way performance data are collected in the APR;
 None of the 6 outcomes and only 1 of the 43 outputs
appears to correspond to the types of suggested KT
outcomes presented above;
 However, more than half of the outputs describe findings
that appeared to be “ready” for the application of KT
strategies to accelerate use or uptake based on strength of
presented;
 And, several of the accomplishments described utilizing
KT strategies in the generation of the output – e.g., use of
networks or partnerships among researchers, developers,
and industry representatives.
Part 3: Examples “Noteworthy”
Accomplishments – Ready for
Application of KT? # 1




Program Mechanism: RRTC
NIDRR LRP Domain: Employment
Accomplishment Type: Output, Peer-reviewed Lit Review
Objective and Key Findings:
 Summarizes research knowledge from fields of disability,
economics, health care, and labor force research, to
describe the nature of barriers to paid work and economic
security for people with mental disorders; and
 Describes how findings can be applied to systems change
activities that can be pursued at local, state, and federal
levels.
 Also summarizes NIDRR-funded research that was used
by the President’s New Freedom Commission on Mental
Health.
Part 3: Examples “Noteworthy”
Accomplishments – Ready for
Application of KT? #2




Program Mechanism: RRTC
NIDRR LRP Domain: Participation and Community Living
Accomplishment Type: Output, Peer-reviewed Pub
Objective and Key Findings:
 Uses most recent available data (2002) to present three per
participant expenditure comparisons between Medicaid
HCBS waivers (which require that participants have an
institutional level of care need) and institutional care: (1)
program expenditures; (2) total Medicaid expenditures, and
(3) estimated total public expenditures.
 Analysis estimates that when compared with Medicaid
institutional care in 2002, HCBS waivers produced a
national average public expenditure saving of $43,947 per
participant.
Part 3: Examples “Noteworthy”
Accomplishments – Ready for
Application of KT? #2 (Continued)




Program Mechanism: RRTC
NIDRR LRP Domain: Participation and Community Living
Accomplishment Type: Output, Peer-reviewed Pub
Potential Policy Implications:
 As long-term care policy makers struggle with competing
challenges including state budget deficits and pressures to
expend home and community based services (HCBS),
there is a pressing need for information on the comparative
cost of Medicaid HCBS and institutional care.
Part 3: Examples “Noteworthy”
Accomplishments – Already
Applying KT Strategies #3




Program Mechanism: RRTC
NIDRR LRP Domain: Participation and Community Living
Accomplishment Type: Output, Community of Practice
Objective & Desired Outcomes:
 Developed Early Childhood Family Support CoP to foster
wisdom-based action, including integration of best
available research, for the ultimate benefit of families.
Outcomes envisioned:
• For families and practitioners that they become
informed and inspired to implement wise solutions.
• For researchers that they will be guided by families and
practitioners as they design future research.
Part 3: Examples “Noteworthy”
Accomplishments – Already
Applying KT Strategies #3




Program Mechanism: RRTC
NIDRR LRP Domain: Participation and Community Living
Accomplishment Type: Output, Community of Practice
Objective & Desired Outcomes (continued):
• For policy leaders that they work in with families,
professionals, and researchers to translate best available
research and family wisdom into meaningful policy.
 KT Strategies: Received RUSH grant to:
 Assess the effectiveness of CoP implementation;
 Measure gains in knowledge utilization based on random
sample of CoP members; and
 Utilize mixed-method design to document changes in
behavior of stakeholder groups related to using CoP.
Conclusions & Next Steps:
 NIDRR has only just embarked on the road to measurement
of KT outcomes.
 Much work needs to be done to get agreement on KT
outcomes; chart pathways of KT outcomes; formulate KT
outcome goals, and identify indicators of success.
 However, significant process has been made in:
 Improving performance reporting and data collection
systems;
 Establishing a performance monitoring system to track KT
accomplishments; and
 Launching a performance assessment expert review
process to rate individual grant accomplishments for
contribution to advancing the field and to assess the
overall progress towards NIDRR’s long-term goals.
Thank you