Transcript Document

Measuring the Impact of IFLE Grant Programs
Evidence-Based Grant Design & Evaluation
U.S. Department of Education, International & Foreign
Language Education Program
Andy Gluck
Technical Director
ICF International
KimOanh Nguyen-Lam, Ph.D.
Advanced Training & Research
Division Director
Slide 2
Slide 3
Proposed Priority
Evaluation
accomplished
FLAS Survey on Post-Graduate Outcomes
 Mandated by the Higher Education Opportunity Act (HEOA) of 2008,
Section 601: “The Secretary shall assist grantees in developing a
survey to administer to students who have completed programs
under this title to determine postgraduate employment, education,
or training. All grantees, where applicable, shall administer such
survey once every two years and report survey results to the
Secretary.”
 To be administered every other year for 8 years (QUALTRICS)
 Comparison groups (national data sets: American Community Survey, the
NSF Survey of Earned Doctorates, the Survey of Doctorate Recipients, or the
Baccalaureate & Beyond Longitudinal Study)
IFLE Evaluation Contracts
2011-2013
 EPI: Education Policy International
– To assist IFLE staff in developing a comprehensive plan for analysis and
formative assessment of the UISFL, BIE and NRC programs.
– Data sources: IRIS database, Project Directors Surveys and Interviews
 BPA: Berkeley Policy Associates IMPAQ
– To Improve the effectiveness of IFLE’s programs and Increase
transparency and accountability
– Tasks:
• Analyze existing performance data/measures (Logic Model)
• Assist the development of outcome measures (GPRA),
• Develop evaluation materials for grantees (Evaluation Guide)
OMB May 18, 2012 MOU
Evidence-Based Grant Program Designs
Evidence-Based Program Design
•
Build evidence: use strong evaluation designs to test and find effective practices
•
Act on evidence: increase the share of funds that support evidence-based practices
•
Support knowledge diffusion and learning among grantees: Programs should disseminate
findings about what works and what does not to current and potential grantees
•
Build grantee capacity and use grantee input: Use training, technical assistance, collateral
materials, grantee learning networks, and information systems that help grantees implement
effective programs.
•
Return data to its suppliers with value added through analysis: Agencies that gather data from
grantees should return the data collected to grantees in ways that the grantees find useful.
•
Harness available high-quality data: Programs should look for available high-quality data sets
that can inform decisions and lower the cost of performance measurement, data analysis, and
evaluation.
•
Engage both program and research offices: The most effective programs enable researchers to
inform program design and implementation, and program staff to inform the research agenda.
A key recommendation from the 2011 grants improvement initiative was to revise EDGAR
to reflect policy goals of the department (ED).
As a result, ED developed a notice of proposed rulemaking (NPRM), published in the federal
register (77 FR 74392) in December 2012. The amendments promote objectives related to
evidence of effectiveness, project evaluation, simplified procurement procedures,
sustainability, productivity, and strategy to scale in parts 75 and 77 of EDGAR.
The notice of final rulemaking (NFRM) was published on August 13 and the changes became
effective 30 days later – September 13, 2013.
Performance Measures
GPRA Measures
Project Specific Measures
GPRA of 1993 - GPRA Modernization Act of
2010 requires federal agencies to develop and
report to Congress measures that are:
 Include both quantitative &
qualitative
 Support program goals
 Link directly to project’s
goals and objectives
 Align with institutional
goals
 Have clearly defined
outcomes



quantifiable annual and longterm
ambitious, yet achievable targets
from baseline data
based on program regulations
and goals
Slide 15
Performance Measures







Objective-linked
Responsibility-linked
Organizationally acceptable
Comprehensive
Credible
Compatible
Comparable with other
data (useful in making
comparisons, for example,
performance can be
compared from period to
period, with peers, with target
groups, etc.)
Additionally, we have to be
able to demonstrate that
NRC program and project
performance is:
Sustainable
Reasonable
Measurable
Replicable
 Easy to interpret
16
Project Performance Measures
– Student Outcomes: advanced proficiency
(knowledge/skills), degree completion, employability
– Faculty Outcomes: improved course curricula & delivery,
published & disseminated new knowledge
– Infrastructure Outcomes: courses, certificate, degree
programs, signed agreements/collaboration, sustainability
Other Research Study Questions:





Why should my institution care about international education?
Why should it be a priority in higher education or undergraduate program?
What are my institution’s overarching goal and priorities?
How can I align international education with them?
How do I set up an evaluation to produce concrete data to show the values of international abroad
education?
Education Abroad & academic performance (Kuh, 2009, McKeown 2010)
Education Abroad & college persistence and completion (GLOSSARY 2001-09, U of Minnesota 19992009, Indiana2009, UC San Diego 2008-09)
17
IFLE PERFORMANCE MEASURE FORM (GPRA Measures)
1. Performance
Measures
2. Activities
3. Data/
Indicators
4. Freq.
5. Data
Source
6. Baseline and Targets
BL
T1
T2
T3
T4
Slide 18
IFLE PERFORMANCE MEASURE FORM (Project-Specific Measures)
2.
Performance
Measures
1. Project Goal Statement
3.
4. Data/
5. Freq. 6. Data 7. Baseline and Targets
Activities Indicators
Source
BL T1 T2 T3
T4
IFLE NEW GPRA MEASURES FY 2014
NRC, FLAS, CIBE, UISFL, AND DDRA
CIBE GPRA MEASURES
Centers for International Business Education (CIBE) Program provides funding to
schools of business for curriculum development, research, and training on issues of
importance to United States trade and competitiveness.
1. Percent of graduates of a doctoral or Master's, including MBA, program with significant
international business concentration (completed more than 12 semester units) at the
postsecondary institution who are employed in business-related fields, including teaching at
a `business school.
2. Percentage of CIBE projects that established or internationalized a concentration, degree, or
professional program with a focus on or connection to international business over the course
of the CIBE grant period. (Long-term measure).
3. The percentage of Centers for Business and International Education (CIBE) projects whose
business industry participants increased export business activities (Short-term measure).
4. Efficiency measure: cost per CIBE doctoral or Master's graduate employed in international
business-related fields, including teaching in a business school or program within eight years
21
after graduation as measured by the IFLE Tracking Survey
LRC GPRA MEASURES
Language Resource Centers (LRCs)Program provide grants for establishing, strengthening, and
operating centers that serve as resources for improving the nation's capacity for teaching and
learning foreign languages through teacher training, research, materials development, and
dissemination projects.
1. Percentage of LRC products or activities judged to be
successful by LRC customers with respect to quality,
usefulness, and relevance.
2. Percentage of LRCs products judged to be successful
by an independent expert review panel with respect
to quality, relevance, and usefulness. (In addition,
experts could survey end users of LRC products.)
Slide 22
NRC GPRA MEASURES
National Resource Centers (NRC) Program provides grants to institutions of higher education or consortia of
institutions of higher education to establish, strengthen, and operate comprehensive and undergraduate
Centers that will be national resources for teaching of any modern foreign language.
1.
Percentage of National Resource Centers that targeted and taught priority languages, as
defined by the Secretary of Education.
2.
Percentage of National Resource Centers that increased the number of intermediate or
advanced level language courses in the priority languages and/or LCTLs during the course
of the grant period. (Long-term).
3.
Percentage of National Resource Centers that taught intermediate or advanced courses in
any of the priority languages as defined by the Secretary of Education.
4.
Percentage of National Resource Centers that increased the number of certificate, minor,
or major degree programs in the priority and/or LCTLs, area studies, or international
studies during the course of the four-year grant period.
5.
Efficiency measure: Cost per NRC that increased the number of intermediate or
advanced level language courses in the priority and/or LCTL s during the course of the
Slide 23
grant period.
FLAS GPRA MEASURES
The Foreign Language and Area Studies (FLAS) Fellowship Program provides academic year and
summer fellowships to institutions of higher education to assist undergraduate and graduate
students in foreign language and either area or international studies.
1.
Percentage of Foreign Language and Area Studies (FLAS) academic-year fellows who
increased their foreign language reading, writing, and/or listening/ speaking scores by at
least one ILR equivalent proficiency level.
2.
Percentage of Foreign Language and Area Studies (FLAS) graduated fellows who
secured employment that utilizes their foreign language and area studies skills within 8
years after graduation based on FLAS tracking survey.
3.
Percentage of Foreign Language and Area Studies (FLAS) master’s and doctoral
graduates who studied priority languages as defined by the Secretary of Education.
4.
Efficiency measure: cost per Foreign Language and Area Studies (FLAS) Fellowship
Program academic year fellow who increased his/her reading, writing, and/or
listening/speaking language score by at least one proficiency level
24
UISFL GPRA MEASURES
The Undergraduate International Studies and Foreign Language (UISFL) Program provides funds to institutions
of higher education, a combination of such institutions, or partnerships between nonprofit educational
organizations and institutions of higher education to plan, develop, and carry out programs to strengthen and
improve undergraduate instruction in international studies and foreign languages.
1. Percentage of UISFL projects that added or enhanced
courses in international studies in critical world areas and
priority foreign languages as reflected in the course syllabi.
(Short-Term).
2.
Percentage of UISFL consortium projects that established
certificates and/or undergraduate degree programs in
international or foreign language studies. (Long-Tem)
Slide 25
DDRA GPRA MEASURES
The Fulbright-Hays Doctoral Dissertation Research Abroad (DDRA) Program provides grants to
colleges and universities to fund individual doctoral students to conduct research in other
countries in modern foreign languages and area studies for periods of 6 to 12 months.
1.
Percentage of DDRA fellows who increased their foreign language
scores in speaking, reading, and/or writing by at least one ILR equivalent
proficiency level.
2.
Percentage of Doctoral Dissertation Research Abroad (DDRA) fellows
who complete their degree in their program of study within 8 years of
receipt of the fellowship
3.
Percentage of DDRA fellows who found employment that utilized their
language and area skills within 8 years of receiving their award
4.
Efficiency Measure: The cost per DDRA fellow who found employment
that utilized their language and area studies skills within 8 years.
Slide 26