Transcript AERA-NCME 2009 PowerPoint template

®
Evaluation of the Quality of
Higher Education in the U.S.:
Role of Learning Outcomes
Assessment
刘欧, 博士
Educational Testing Service, Princeton
北京大学中国教育财政科学研究所
Aug 23 2012
Unpublished Work Copyright © 2012 by Educational Testing Service. All Rights Reserved. These materials are an unpublished, proprietary work of ETS. Any
limited distribution shall not constitute publication. This work may not be reproduced or distributed to third parties without ETS's prior written consent. Submit
all requests through www.ets.org/legal/index.html.
Educational Testing Service, ETS, the ETS logo, and Listening. Learning. Leading. are registered trademarks of Educational Testing Service (ETS).
®
Introduction
• Rapid development of higher education
– 14.8 million to 20.4 million students from 1999 to 2009 (U.S. Department of
Education, National Center for Education Statistics, 2011).
• National goal of higher education
– By 2020, America should have the highest proportion of college
graduates (Obama, 2009)
• Quality Assurance in higher education
– Accountability
– Comparable outcomes
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Need for Learning Outcomes
Assessment
• Accreditation
– Pressure on institutions to become accountable for student learning
• Accountability calls
– Voluntary System of Accountability (VSA)
– Transparency by Design
– Voluntary Framework of Accountability
• Public
– Understand how institutions operate
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Federal Input on Accountability
• Commission on the Future of Higher Education
– Accountability as one of four areas for urgent reform
– Report comments on the nation’s “remarkable absence of
accountability mechanisms to ensure that colleges succeed
in educating students” (U.S. Department of Education, 2006)
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Accountability Initiatives
• Voluntary System of Accountability (VSA)
– American Association of State Colleges and Universities
(AASCU) and the Association of Public and Land-grant
Universities (APLU)
– Public colleges and universities
– Institutions voluntarily join VSA to demonstrate student
learning
• 319 institutions have joined VSA
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Accountability Initiatives
• Transparency by Design
– Universities that provide distance education
– Serve adult learners
– WCET at the Western Interstate Commission for Higher
Education
• Voluntary Framework of Accountability
– Community college students
– American Association of Community Colleges and the
College Board
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Current Situation of Using Outcomes
Assessment for Accountability
• A core purpose of Voluntary System of Accountability
– Measure core educational outcomes: written communication,
analytic reasoning and critical thinking
• College Portrait
– A platform for participating institutions to demonstrate their
student learning outcomes and experiences
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
VSA Assessments
• Assessments
– ETS Proficiency Profile (Liu, 2008, 2009a, b, 2011a, b, 2012)
– Collegiate Learning Assessment (CLA; CAE, 2007)
– Collegiate Assessment of Academic Proficiency (CAAP; ACT,
2009)
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
The ETS Proficiency Profile
•
•
•
•
General college education outcomes
– Reading and critical thinking, writing, and math
Item format
– 108 multiple choice items
– 27 items for each skill area
Test forms
– Standard form (two hours)
– Short form (40 minutes)
Delivery methods
– Paper/pencil
– Online
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Value-Added Computation
• Definition
– Performance difference on standardized tests between
freshmen and seniors after controlling for admission scores
(Council for Aid to Education, 2007)
• Design
– Cross-sectional
• Freshmen and seniors are tested at the same time
– Analysis at institutional level
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Value-Added Calculation
First-year Students
Fourth-year Students
STEP 1:
Use Mean SAT score to
predict Mean EPP score
STEP 1:
Use Mean SAT score to
predict Mean EPP score
STEP 2:
Calculate Residuals
(Actual EPP– Expected EPP )
STEP 2:
Calculate Residuals
(Actual EPP– Expected EPP )
STEP 3:
Value Added =
Fourth-year Residual – First-year Residual
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
An Example of Value-added Report
Value
Added
Freshmen
Seniors
Mean SAT (or Converted ACT) Score
1081
1088
Expected EPP Score
445
452
7
Actual EPP Score
443
453
10
Actual versus Expected*
(difference in score points)
-2.3
1.3
3.6
Actual versus Expected**
(difference in standard errors)
-0.9
0.4
1.3
10
9
8
7
6
Performance Level***
5
4
3
2
1
12
Confidential and Proprietary. Copyright © 2009 Educational Testing Service.
®
Current Challenges in Outcomes
Assessment (Liu, 2011a)
• Insufficient evidence of what learning outcomes
assessment predicts
• Methodological issues with the current value-added
method
• Comparability of the three VSA-selected tests
• No evidence of the comparability of results between the
preferred longitudinal design and the current crosssectional design
• Unclear evidence of student motivation in taking lowstakes tests
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Insufficient Evidence of What
Outcomes Assessment Predicts
• Predictive validity with regard to other success
indicators
– GPA, retention, graduation, graduate school application, job
placement
• Previous research shows a positive correlation
between Proficiency Profile scores and college GPA
(Hendal, 1991; Lakin, Elliott, & Liu, 2012; Marr, 1995)
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Methodological Issues with the
Current Value-added Method
• Unit of analysis
– Institution vs. student
• Institutional characteristics
– Type, selectivity, size and settings, student profile
• Statistical methods
– Ordinary Least Squares (OLS) vs. hierarchical linear
modeling
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Comparisons of Two Value-added
Methods
• Multi-level model vs. Ordinary least squares (Liu, 2011b)
• Correlation of value-add index between two methods
– Critical thinking (r = .76)
– Writing (r =.84)
• Largest difference for an individual institution
– Critical thinking: 5 decile levels
– Writing: 4 decile levels
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Value-added Ranking: Critical Thinking
EPP Critical Thinking
10
10
9
9
9
Decile Rank
8
8
7
7
8
8
7
6
7
6
5
10
8
7
7
6
5
6
6
5
4
5
4
4
3
4
4
3
2
3
3
2
1
1
5
4
3
2
7
1
1
1
2
2
1
1
0
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
School IDs
OLS Ranking
HLM Ranking
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
23
®
Value-added Ranking: Writing
EPP Writing
10
10
9
9
9
Decile Rank
8
6
5
5
9
8
7
6
10
9
8
7
10
8
7
7
6
6
5
4
4
3
8
7
6
5
4
3
7
5
4
4
3
3
2
2
1
1
2
1
2
1
1
0
0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 22 23
School IDs
OLS Ranking
HLM Ranking
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Comparability of the Three Tests
• Tests differ in item format, test length, delivery mode,
scoring methods, even in construct definition
• 2007-2009: Fund for the Improvement of
Postsecondary Education (FIPSE) under the
Department of Education
– Test Validity Study
– Jointly conducted by ETS, CAE and ACT
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Test Validity Study
• Examine the correlation between tests designed to
measure similar constructs
• Examine whether item format has an impact on
student performance
• Examine the school-level reliability
• Examine the performance gains between freshmen
and seniors
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Conclusions from the Test
Validity Study
• Despite the seemingly differences among tests, after
controlling for unreliability, tests designed to measure
similar constructs correlated very highly (Klein, Liu, Sconing
et al., 2009).
• Constructs play a more important role in affecting
scores than item formats
• School-level reliabilities are high for all tests and
particularly high for multiple-choice tests
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Longitudinal and the Current
Cross-sectional Designs
• A group of freshmen and a group of seniors for the
institutional value-added report
– The freshmen and seniors are not the same group of
students.
• A potential problem is the unequivalence between
these two groups
– Seniors may be more selective due to retention reasons (Liu,
2009b)
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Longitudinal and the Current
Cross-sectional Design
• Need to track the same group of students from
freshman to senior year
– Test them twice
– Compare the results with results from the cross-sectional
design
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Student Motivation in Taking
Low-stakes Tests
• Outcomes assessment does not have a direct impact
on students
– low motivation could threat the validity of the test results
• Ways to monitor student motivation
– Student self-report
– Motivation survey : Student Opinion Survey (Sundre & Wise,
2003)
– Response time effort (Wise & Kong, 2005)
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Limited College Learning From
Learning Outcomes Data
From freshman entrance to the end of
sophomore year, students have improved
these skills (critical thinking, complex
reasoning, and writing), as measured by
the CLA, by only 0.18 standard deviation
(p. 35)
Arum & Roksa
(2011)
No statistically significant gains in critical
thinking, complex reasoning, and writing
skills for at least 45 percent of the
students (p.36)
®
Objectives
• Investigate the impact of motivation on low-stakes
learning outcomes assessment in U.S. higher
education
• Identify practical motivational strategies that
institutions can use
26
®
Research Questions
• What is the relationship between self-report
motivation and test scores?
• Do motivational instructions affect student test
scores?
• Do conclusions drawn about college learning gain
change with test format and motivational instruction?
27
®
Participants (N=757)
• One four-year research institution
– n=340, SAT/ACT
• One four-year master’s institution
– n=299, SAT/ACT
• One community college
– n=118, placement test scores
28
®
Instruments
• ETS Proficiency Profile
– Multiple-choice test
– Measures critical thinking, reading, writing, and mathematics
– Abbreviated version (36 items)
• Essay
• Motivation survey
– Student Opinion Survey (10 items; Sundre, 1999)
29
®
Motivational Conditions
• Created three motivational
conditions
• Embedded in regular consent forms
• Random assignment within a
testing session
30
®
Control Condition
Your answers on the
tests and the survey will
be used only for
research purposes and
will not be disclosed to
anyone except the
research team.
31
®
Institutional Condition
Your test scores will be
averaged with all other
students taking the test at your
institution. Only this average
will be reported to your
institution. This average may
be used by employers and
others to evaluate the quality of
instruction at your institution.
This may affect how your
institution is viewed and
therefore affect the value of
your diploma.
32
®
Personal Condition
Your test scores may be
released to faculty in
your college or to
potential employers to
evaluate your academic
ability.
33
®
Relationship between Self-report Motivation
and Test Scores
ETS Proficiency
Profile
RI
MI CC
Self –report
motivation
SAT/Placement
Test
RI
Essay
MI
CC
.17*** .26*** .22** .20*** .26*** .17*
.68*** .54*** .50
.31*** .32*** .29
Note. *p<.05. **p<.01. ***p<.001. RI=research institution;
MI=master’s institution; CC=community college
Copyright © 2012 by Educational Testing
Service.
34
®
Scale Score
How Motivational Instructions Affected the ETS
Proficiency Profile
464
462
460
458
456
454
452
450
448
462
458
0.41 SD
453
0.26 SD
Control (n=250)
Institutional
(n=257)
Personal (n=247)
Copyright © 2012 by Educational Testing
Service.
35
®
How Motivational Instructions Affected the Essay
4.50
4.40
4.30
4.20
4.10
4.00
3.90
3.80
4.46
4.29
4.07
0.23 SD
Control (n=250)
0.41 SD
Institutional
(n=257)
Personal (n=247)
Copyright © 2012 by Educational Testing
Service.
36
®
How Motivational Instructions Affected
Self-report Motivation
4.00
3.90
3.80
3.70
3.60
3.50
3.40
3.89
3.81
3.61
0.31 SD
Control (n=250)
0.43 SD
Institutional
(n=257)
Personal (n=247)
Copyright © 2012 by Educational Testing
Service.
37
®
Sophomore to Senior Score Gain
(In Adjusted Effect Size within Motivational
Condition)
0.45
0.42
0.41
0.40
0.35
0.30
0.25
0.25
0.23
0.21
EPP (multiple-choice)
Essay
0.20
0.15
0.10
0.05
0.02
0.00
Control (n=210)
Institutional
(n=215)
Personal
(n=211)
Copyright © 2012 by Educational Testing
Service.
38
®
Sophomore to Senior Score Gain
(In Adjusted Effect Size across Motivational
Condition)
0.80
0.72
0.65
EPP (multiplechoice)
Essay
0.60
0.40
0.20
0.00
0.00
-0.20
-0.40
Least motivated
sophomores, most
motivated seniors
Most motivated
sophomores, least
motivated seniors
-0.23
Copyright © 2012 by Educational Testing
Service.
39
®
Conclusions
(Liu, Bridgeman, & Adler, 2013, forthcoming)
• Motivational instructions have both a statistically
significant and substantially important impact on
outcomes assessment results
• Conclusions about value-added learning can
dramatically change with motivational instruction and
test of choice
40
®
Other Considerations
• Sampling of students
• Performance of English language learners (ELL; Lakin,
Elliott, & Liu, 2012) and international users
• Linking student learning outcomes to institutional
effectiveness
– Other variables that have an impact on learning
• Student motivation, learning experiences, financial status
etc
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Pending and Future Research
• A Chinese version of the ETS Proficiency Profile
• The OECD Assessment of Higher Education
Learning Outcomes (AHELO) project
–
–
–
–
General skills
Economics
Engineering
Value-added measure
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
References
•ACT. (2009). CAAP guide to successful general education outcomes assessment. IOWA City, IA: ACT.
•Baumert, J., & Demmrich, A. (2001). Test motivation in the assessment of student skills: The effects of incentives on
motivation and performance. European Journal of Psychology of Education, 16, 441-462.
•Braun, H., Kirsch, I, & Yamamoto, K. (2010). An experimental study of the effects of monetary incentives on performance
on the 12th grade NAEP reading assessment. Princeton, NJ: Educational Testing Service.
•Council for Aid to Education. (2007). CLA institutional report 2006–2007. New York: Author.
•Hendel, D. D. (1991). Evidence of convergent and discriminant validity in three measures of college outcomes.
Educational and Psychological Measurement, 51(2), 351-358.
•Klein, S., Liu, O.L., Sconing, J., Bolus, R., Bridgeman, B., Kugelmass, H., Nemeth, A., Robbins, S., & Steedle, J. (2009,
September). Test Validity Study (TVS) report. Supported by the Fund for Improvement of Postsecondary Education
(FIPSE). Online at: http://www.voluntarysystem.org/index.cfm?page=research
•Liu, O.L. (2008). Measuring learning outcomes in higher education using the Measure of Academic Proficiency and
Progress (MAPP™). ETS Research Report Series (RR-08-047). ETS: Princeton.
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
References
•Liu, O. L. (2009a). Measuring learning outcomes in higher education (Report No. RDC-10). Princeton, NJ: ETS.
•Liu, O.L. (2009b). Measuring value-added in higher education: Conditions and caveats. Results from using the Measure of Academic Proficiency and
Progress (MAPP™). Assessment and Evaluation in Higher Education, 34(6), 1-14.
•Liu, O. L., Bridgeman, B. & Adler, R. (Accepted). Learning outcomes assessment in higher education: Motivation matters. Educational Researcher.
•Liu, O. L. (2011a). Outcomes assessment in higher education: Challenges and future research in the context of Voluntary System of Accountability.
Educational Measurement: Issues and Practice, 30(3), 2-9.
•Liu, O. L. (2011b). Value-added assessment in higher education: A comparison of two methods. Higher Education, 61(4), 445-461.
•Marr, D. (1995). Validity of the Academic Profile. Princeton, NJ: ETS.
•Obama, B. (2009). President Obama’s Address to Congress. Retrieved from http://www.nytimes.com/2009/02/24/us/politics/24obama-text.html?_r=2
on Feb 20 2010.
•Sundre, D. L., & Wise, S. L. (2003, April). Motivation filtering: An exploration of the impact of low examinee motivation on the psychometric quality of
tests. Paper present at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
•U.S. Department of Education (2006). A test of leadership: Charting the future of American higher education. (Report of the commission appointed by
Secretary of Education Margaret Spellings). Washington, DC: Author.
•Wise, S. L., & DeMars, C. E. (2010). Examinee noneffort and the validity of program assessment results. Educational Assessment, 15, 27-41.
•Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in
Education, 18(2), 163-183.
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).
®
Questions?
Lydia Liu [email protected]
Copyright © 2010 by Educational Testing Service. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS).