Transcript Document

Lessons Learned from Industry:
Achieving Diversity & Efficacy in College
Success
ETS - College Board Invitational Conference
Washington, DC
Wayne Camara & Krista Mattern
September 8, 2008
Job Analysis
• Organizations use job analysis to determine what
work outcomes are desired.
• Sample individual outcomes (productivity, job performance,
retention) and organizational outcomes (efficiency, quality,
innovation, team work)
• Identify performance components (pc)
pc = {Declarative knowledge x Procedural knowledge x Motivation}
Knowledge
Goals
x
Cognitive skills
Interpersonal “
Ability, Interests, Education, Experience
2
Predictors
x
Level of effort
Persistence “
Importance & Prob.
Of Outcomes
Job Analysis – College Success
1. Identify desired performance outcomes for
individuals and organizations (college success)
(GPA, return, graduate, life after college – grad
school, certification)
Each outcome likely has different predictors
2. Identify performance tasks associated with
outcomes (persistence, academic ability, health,
engagement)
3
3. Identify or develop performance measures
(GPA, advisory ratings, self report data, dB of
student engagement)
Predicting Performance
Performance
Tasks (GPA,
Engagement,
Graduation)
Predictors
(Tests, Essay,
Ref,
Application)
4
Competencies
(Knowledge,
Skills, Abilities)
Goldstein, Zedeck, & Schneider (1993)
Group differences are not unique
to tests: They are present across
most educational measures
5
Large Mean Differences Persist on Cognitive
Ability Tests by Race/Ethnicity Remain
SAT CR+M
1200
1100
American Indian / Alaskan
Asian
1000
Black/African American
Hispanic
White
Other
900
800
1998 1999 2000 2001 2002 2003 2004 2005 2006 2007
6
College-Going Rates of High School Graduates Aged 18
to 24 by Ethnic Group, 1999-2006
70%
Asian
65%
60%
55%
White
50%
45%
Black
40%
Hispanic
35%
30%
25%
20%
98-99
7
99-00
Source: U.S. Census Bureau
00-01
01-02
02-03
03-04
04-05
05-06
8
Disparities Exist in HS Graduation, HS Drop
Out and College Ready
9
Source: Manhattan Institute, Public HS Graduation and College-Readiness Rates: 1991-2002,
http://Manhattan-institute.org/html/ewp_08.htm;* Condition of Education, 2007 Table 23-2
Graduation Rates in 2004 by ethnicity
79.4%
80.0%
76.2%
75.9%
70.5%
70.0%
67.1%
60.0%
50.0%
White
African Amer
40.0%
Hispanic
Asian
30.0%
20.0%
13.9%
10.0%
12.8%
13.8%
5.2%
11.6%
5.0%
10.9%
9.8%
7.3%
8.2%
6.3%
7.0%
6.5%
4.8%
7.6%
0.0%
HS Graduates
10
Associates Degree
Bachelors degrees
Masters degrees
Doctorate degrees
Published 3/7/2007 Title Awards conferred by Title IV institutions, by race/ethnicity, level of award, and
gender: United States, academic year 2004–05 (recalculated to eliminate students who with other or no
ethnicity reported). http://nces.ed.gov/ipeds/factsheets/pdf/fct_awards_conferred_03072007_5.pdf; Public HS
graduation rates: WICHE 3/2008, http://www.wiche.edu/policy/knocking/1992-2022/index.asp
Rationale for looking beyond Grades and
Tests
What is college success? Is it more than
grades and GPA? (Camara & Kimmel)
Develop measures that predict your goal
or desired outcome.
Employers test multiple measures:
openness, conscientiousness,
extraversion, agreeableness,
neuroticism
Military use today (GED).
Can do does not equal will do.
11
Predictors of College Success
College Skills
Content Knowledge
Achievement
Non-Cognitive
Personal Qualities/
Experiences/
Characteristics
Interests Vocations
Verbal Reasoning
Math
Motivation
Letters
Grades
Career Interests
Math Reasoning
Language Arts
Follow-through
Essay
GPA
Learning -Study
Skills
Writing
Science
Communication
Community Service
Weighted GPA
Interest in Major
Metacognition
Social Studies/
Humanities
Conscientiousness
Extra-curricular
Rank
Self Efficacy
Creativity
Foreign Language
Leadership
Work Experience
Courses Completed
Aspirations/
Practical
Knowledge
Language Proficiency
Other Personality
Literacy in Second Lang
Academic Rigor
Realistic Selfconcept
Spatial Relations
Health/Lifestyle
Teacher Ratings
AP/Honors Courses
Intellectual
Curiosity
Adaptability
Gender
School Size
Ethnicity
School Quality
Technology –
Research Skills
Ability to Benefit
Family Education/
Income
Ability to Pay
Residence
Tests Measure
12
School
Performance/
Context
Colleges Collect in some form
(applications, transcripts)
Not Collected in
Standardized form
Research collaboration with
Michigan State University
• Identify a broader domain of college student
performance:
• Review university mission statements and
department objectives
• Interview with university staff responsible
for student life
• Review of the education literature on student
outcomes
• Our systematic search (A JOB ANALYSIS OF
UNDERGRADUATE STUDENTS) resulted in 12
dimensions of student performance…
13
• Validate items with successful juniors – they are
the experts.
12 Dimensions of Student Performance
Broadening the Performance Domain in the Prediction of Academic
Success (Schmitt, Oswald, & Gillespie, 2004)
1. Knowledge, learning, mastery of general principles
2. Continuous learning, intellectual interest and curiosity
3. Artistic and cultural appreciation
4. Multicultural appreciation
5. Leadership
6. Interpersonal skills
7. Social responsibility, citizenship and involvement
8. Physical and psychological health
9. Career orientation
10. Adaptability and life skills
14
11. Perseverance
12. Ethics and integrity
Two “Noncognitive” Measures
• Situational judgment inventory
• A situation is presented along with several alternative
courses of action.
• The respondent is asked to indicate what she/he
would be most likely and least likely to do.
• Biodata
• Short, multiple choice reports of past
experience/background and interests/preferences.
15
Sample SJI Item for Leadership
You are assigned to a group to work on a particular project. When
you sit down together as a group, no one says anything.
a)-1 Look at them until someone eventually says something
b)Start the conversation yourself by introducing yourself
c)+1 Get to know everyone first and see what they are thinking about the
project to make sure the project’s goals are clear to everyone
d)Try to start working on the project by asking everyone’s opinion about
the nature of the project
e)You would take the leadership role by assigning people to do things or
ask questions to get things rolling
16
Sample Biodata Items for Leadership
1. The number of high school clubs and organized activities (such as
band, sports, newspapers, etc.) in which I took a leadership role
was:
a) 4 or more
b) 3
c) 2
d) 1
e) I did not take a leadership role
2. How often do you talk your friends into doing what you want to do
during the evening?
a) most of the time
b) sometimes (about half the time)
c) occasionally (about as often as others in my group
d) seldom or infrequently
17
e) never
Study 1:
Develop and refine the measures
• 644 MSU freshmen completed one of the two parallel forms
of the biodata and SJI instruments at the beginning of the
academic year.
• Results indicated significant incremental validity for some of
the scales above and beyond the validity of SAT/ACT
scores and existing measures of personality in predicting
college GPA.
• The biodata and SJI demonstrated the greatest incremental
validity when absenteeism, students’ self ratings, and peerratings of performance were examined ( .19, .22, and .14,
respectively).
18
Study 2 Examine Validity & Subgroup Differences:
10 Participating Institutions & 2,700 Freshmen
HBCU
N
Winston-Salem (public)
229
Spelman College (private)
254
Big Ten (public)
N
University of Iowa
19
335
Michigan State University
546
Ohio State University
304
University of Michigan
297
Indiana University
170
Other Institutions
N
University of Chicago (private)
168
Cal State – Fullerton (public)
223
Virginia Tech (public)
237
Predicting FYGPA: Total Sample across
10 Institutions (N = 2443)
Non cognitive
measures contribute
little beyond tests
and grades in
predicting academic
outcomes
20
Predicting Class Absenteeism: Total Sample across 10
Institutions (N = 899)
However, non
cognitive
measures will
predict non
cognitive
outcomes – better
than tests or
grades
(graduation,
attendance,
leadership,
engagement)
21
Percent of Students Selected:
Two Composites and Three Selection Strategies
Less selective
Moderately
selective
Top 85%
Group
Hispanic
AB
4.4 
Top 50%
7.6 
AB
AB+
AB
4.6
4.1 
4.9
3.9  5.5
AB+
(+.8)
(+1.6)
9.5
17.5  12.9
(+.1)
(-.4)
(-4.6)
9.6  13.6
(+4.0)
1.3 
76.4 
71.9
77.2  74.4
(-4.5)
(-2.8)
7.7
African-American
17.9 
19.8
(+1.9)
White
70.2 
67.9
9.9 
(-2.3)
AB = equally weighted composite of HSGPA and SAT/ACT.
AB+ = equally weighted composite of HSGPA, SAT/ACT, Biodata, and SJI.
22
Top 15%
AB+
(+.2)
Asian
Very selective
7.2
(+5.9)
Correlations of Non-cognitive Measures with
Cumulative GPA and Graduation
Variable
23
Cumulative GPA
Graduation
SAT/ACT scores
0.59
0.24
HSGPA
0.56
0.28
Knowledge
0.24
0.12
Continuous Learning
0.11
0.03
Artistic Appreciation
0.22
0.13
Multicultural Appreciation
0.12
0.11
Leadership
0.10
0.14
Responsibility
0.12
0.15
Health
0.13
0.08
Career Orientation
-0.18
0.02
Adaptability
0.01
0.07
Perseverance
0.02
0.10
Ethics
0.15
0.11
SJI
0.20
0.14
Note. Bold values are significant at p< .01. N ranges from 1560 to 1798 across variables.
Graduation is dichotomously scored (1, 0).
Study 3: Purpose & Research Questions
• 15 institutions (n = 4,164 for SJI and 7,645 for biodata)
• Purpose: evaluating the utility of the biodata and situational judgment
measures in as close to a real admissions situation as is possible
• Administer new measures to college applicants rather than college
freshmen.
• On an annual basis, collect class absenteeism, self rated performance of
the noncognitve dimensions, and commitment to the university from
enrolled students; institutions will provide course grades and retention.
•
University of Washington
•
Meredith College
•
Michigan State University
•
University of Southern California
•
Lafayette College
•
Furman University
•
Earlham College
•
University of North Carolina, Chapel Hill
•
Ohio State University
•
Kenyon College
•
Purdue University
•
Gonzaga University
•
Spelman College
•
University of Puget Sound
•
Johnson and Wales University
24
Incremental Validity of Biodata Measures
2
2
Overall R
2
R
Outcomes
N
R (HSGPA,SAT)
BARS
57
0.023
0.443*
0.420*
OCB
57
0.017
0.392
0.374*
Deviance
57
0.025
0.373
0.348
Turnover Intent
58
0.077
0.248
0.172
Academic Satisfaction
58
0.008
0.353
0.345
Social Satisfaction
58
0.077
0.294
0.218
FYGPA
84
0.201*
0.335*
0.134
Absenteeism
58
0.061
0.234
0.173
• To preserve N in these regressions, the SJI was not included because of a
relatively low response rate to this measure.
• It is worth noting that small sample sizes, such as those observed in these
analyses, can seriously limit the ability to detect significant relationships due to
decreased statistical power.
25
Next Steps
• In need of a demonstration project – Implement with Research
across a few colleges!
• Encourage applicants to complete on-line as part of
admissions and only use data as a “plus factor.”
• Provide incentives for applicants to complete the new
measures and institutions to track student success over time.
• Likely outcomes will be more diversity, broader talent, greater
retention, and standardized – defensible measures to evaluate
applicants fairly and objectively.
• Increased efficiency and judgmental decisions based on data
and comparability.
• For more information, go to http://www/iopsych.msu/cbstudy
26