Transcript Lessons Learned from Evaluating NSF Programs

Evaluating NSF Programs
Dr. Jennifer Giancola Carney, Abt Associates
September 18, 2008
Agenda
• Two NSF program evaluations (IGERT & CAREER)
– Design & findings
– Rationale for methods used
– Limitations of methods used
– Lessons learned
• Q&A discussion
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
Integrative Graduate Education and Research
Traineeships (IGERT) Program
• Since 1998, PhD training program (DGE)
• Grants to universities who develop new IGERT-related
programs (most $$  student traineeships)
• Give PhD students interdisciplinary research experiences
and enhanced professional skills & perspectives
• Three phases of evaluation
• Implementation study (1999-2002)
• Impact study (2003-2005)
• Follow-up study of graduates (2006-present)
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
IGERT Evaluation: Began with a Logic Model
Program
Characteristics
Recruitment and
retention
Education for
the future
Self evaluation
Funding
Institutional
Characteristics
Pre-existing
IGERT-related
features
University/school
context
Project
Development
and
Implementation
1-3 Year
Outputs
4-6 Year
Outputs
Final
Outcomes
Successful
recruitment and
retention of
IGERT trainees
Graduates of
Program
Success of
trainees in
diverse careers
Education
component
Achievements of
IGERT trainees
Trainee research
productivity
Research
component
Institutional
change
Institutional
change
Collaborations
IGERT faculty
growth
IGERT faculty
growth
Multidisciplinary
theme
Ongoing
assessment
Trainee diversity
Adoption of
models of best
practice
throughout
graduate
education to
meet needs of
21st century
Leveraged funds
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
IGERT Implementation Study
• Annual Monitoring: “Who? What? When?” research questions
– Who participates and why? What activities are conducted?
– Annual web survey of program participants (PIs & trainees)
– Describe the program
recruitment strategies, training activities, faculty involvement
• Site Visits: “How? Why?” research questions
– What challenges have projects encountered? How have they
overcome them?
– Interviews with faculty, students, chairs, administrators
– Identify common challenges and solutions
project management, faculty engagement, implementing
interdisciplinary education within universities
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
IGERT Implementation Study (cont.)
• Examine implementation across projects and over time
– Mixed methods (quantitative & qualitative data)
• Data used for GPRA reporting, program management,
revisions to solicitations, sharing common solutions with IGERT
PIs
• Limitations
– Little information on longer-term effects of IGERT or
broader program impacts on faculty and the university
– No comparison to non-IGERT experiences to take into
account overall trends in graduate education
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
2003 IGERT Impact Study
• Impact study: So what?” research questions
– What have been the outcomes for participating IGERT
faculty and students as compared to non-participating
faculty and students?
– Has there been any institutional impact of IGERT funding?
• IGERT participants (PIs, dept chairs, faculty, students,
administrators) and Non-IGERT participants (comparison
group)
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
IGERT Comparison Group
• Provide a counterfactual for what would have been had IGERT not
existed. Needs to control for “academic quality” and variations among
STEM disciplines
• Matched each IGERT department to a non-IGERT department with
whom they compete for graduate students
– Vulnerable to selection bias: Outcomes may be due to preexisting characteristics of IGERT students, not to IGERT program
• Examples of reported findings:
– Can say: “IGERT trainees engage in more interdisciplinary
activities as graduate students than non-IGERT students.”
– Cannot say: “IGERT causes students to engage in more
interdisciplinary activities.” (Maybe these students would have
sought out i/d activities regardless.)
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
IGERT Impact Study (cont.)
• Benefits
– Examine value of IGERT for:
• Departmental recruitment
• Student preparation
• Faculty interdisciplinary involvement
• Institutional offerings and support for interdisciplinary
education
– Assess against counterfactual of “traditional” graduate ed.
• Limitations
– Focused on current participants
– Tested lots of outcomes – hypothesis generating (not confirming)
– No data on longer term outcomes for graduates
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
2006 IGERT Follow-up Study
• Graduate study: “What?” “So what?” questions:
– Where do IGERT graduates go and what do they do?
Are they any different from non-IGERT graduates?
Has IGERT helped prepare them for their chosen careers?
– IGERT graduates and comparison group of non-IGERT
graduates
• Presenting detailed descriptive data on IGERT graduates
• Limiting outcomes tested with comparison group to key outcomes
(hypothesis confirming, though still selection bias)
• Challenge: locating graduates
– Monitoring system had info on point of contact
– Easier to find those in academic positions versus non-academic
positions. Introduces sample bias into results – will conduct nonresponse bias during analysis.
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
Faculty Early Career Development (CAREER) Program
• NSF’s primary support mechanism for junior faculty
members since 1995.
– Grants to individual faculty members
• Support the research and early career advancement of
junior researchers
• Promote the integration of research and education:
– Individual awardees
– Changing university culture
• Most recent evaluation: 2005-2008
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
CAREER: Research Questions
• Descriptive questions re: perceptions of CAREER
– How do stakeholders at NSF perceive the CAREER
program and its relationship to the mission of NSF?
– How do faculty members in departments that host
CAREER awardee(s) view the CAREER program and its
relationship to their research and educational missions?
• Impact questions
– What is the impact of CAREER on the research activities
and career advancement of awardees?
– What is the impact of CAREER on the integration of
research and education by faculty members?
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
CAREER: Methodology
• Descriptive Study
– Interviews with NSF Program Officers
– Survey of 700 department chairs
– Site visits 22 departments
– Samples representative of population in question (but
no comparison group)
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
CAREER: Methodology (cont.)
• Quasi-experimental evaluation of impact on awardees
– CAREER Awardees
– Comparison group of Non-Awardees (same research
potential and interest in integration of research and
education)
• Unsuccessful CAREER applicants who won another
NSF grant as PI w/in 5 years of CAREER application
• Matched using propensity scores (reduces selection
bias)
• Limited outcomes tested (confirming hypotheses)
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
CAREER: Findings
• Description of how program goals are interpreted within
and outside of NSF
 Inform program management
• Description of characteristics of awarded PIs
 NSF program reporting (GPRA, etc.)
• Assessment of grant’s impact on awardees (“Receipt of a
CAREER award increases the likelihood of receiving
tenure”)
 Inform decisions about program continuation or
modification
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
Lessons learned
• Know thy program: Until you understand the
intervention, you cannot assess outcomes
– Logic model, articulate goals
– Develop indicators / measures of program success
• Clearly define your research questions
– Prioritize - you cannot evaluate everything
– Identify data needs for reporting, decision-making
– Be realistic (ask questions that can be answered about
indicators that can be measured)
• CAREER: “impact on institutional culture”
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
Lessons learned (cont.)
• Identifying appropriate comparison groups
– What’s the right counterfactual? Each comparison option
allows you to answer different questions. Choose the
option which best addresses the research questions.
• IGERT - other interdisciplinary programs? Same or different
institutions? All STEM students nationwide?
– Choose right level of rigor (developing or testing
hypotheses?)
– Consider risk of selection bias
– Change over time (longitudinal studies; pre/post)
– Take advantage of available data available
• National datasets
• CAREER – data available to do PS matching9/18/2008
Abt Associates AGEP Capacity Building Meeting Presentation
Lessons learned (cont.)
• Ground each subsequent phase in findings from previous
work:
– Work from exploratory / descriptive evaluation to more
summative / confirmatory evaluation. Each phase can
answer questions raised (or not answered) in previous
phases.
• IGERT: Implementation  Impact  Graduate Followup
• Take advantage of different levels of data collection
– Qualitative versus quantitative; single versus cross-site
• IGERT: Richness of single site visits enabled future
studies
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
Lessons learned (cont.)
• Think long term
– Begin evaluation when program begins
– Plan now for information you will need in the future
• IGERT: tracking graduates
• New study (GK-12) – building comparison group today
for work in future
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008
Questions?
Abt Associates AGEP Capacity Building Meeting Presentation
9/18/2008