Transcript Interactive Systems Group

Fifth International Conference on Intelligent Technologies
December 3, 2004
Dealing with Uncertainty
in a Model of
Computer Science
Graduate Admissions
Nigel Ward
University of Texas at El Paso
(a 12 minute pre-banquet talk
at a small 3-day gathering
of soft-computing researchers)
The Problem
10,000+ CS grad school applicants a year
many wasted applications
some disappointed applicants
A Solution
enable applicants to predict acceptance decisions,
using a web tool
a model of applicant strength + models of admissions criteria
The Acceptance Estimator Concept
demonstration
How to Combine GRE Scores?
Two common styles: avg/sum and min:
“we expect a GRE V+Q+A of at least 2100”
“we expect at least 600 V, 700 Q and 650 A”
A compromise:
stronger scores weighted less, but not zero*
1.33 for weakest, 1.0 for middle, .67 for strongest
(an ordered weighted averaging operator)
* cf Carlsson, Fuller and Fuller in Yager and Kacprzyk, 1997
Sample Computation
raw
value
(RV)
normalized rank
value NV
R
ranking
factor
RF
contribution
level CL
600
100
#1
.67
67
quantitative 650
0
#3
1.33
0
analytical
writing
62
#2
1.00
62
verbal
4.5
Explaining Apparent Diversity
admissions policy
for department x
standard
model of
applicant
strength
departmentspecific
threshold
> GQ x
X’s published
admissions
policy and
statistics
Estimating the Scaling Parameters
To apply OWA, we must normalize scores first
what is the GRE Q score corresponding to a 3.7 UTEP GPA?
JNTU
y = 0.001x + 0.6692
1.00
0.90
0.80
0.70
Mumbai
y = 8E-06x + 0.5651
GPA
0.60
GPA
0.70
系列1
線形 (系列1)
0.50
0.60
0.40
JNTU
0.30
0.20
0.50
0.40
系列1
線形 (系列1)
0
50
100
150
4.00
200
3.50
GRE
GRE Composite
0.20
Mumbai
3.00
0.10
2.50
0.00
-300
-250
-200
-150
-100
-50
0
50
Grades
-50
y = -0.0039x + 2.9309
0.30
0.00
-100
UTEP
Grades
0.10
100
系列1
線形 (系列1)
2.00
150
GRE
1.50
U. Texas at El Paso
1.00
0.50
0.00
-120
-100
-80
-60
GRE
-40
-20
0
Weighting the Scores
factor
GRE V
IW
.7
GRE Q
1.0
GRE AW
.7
GPA (if US)
2.5
GPA (JNTU, Madras)
2.5
GPA (other Indian)
2.0
…
letters of recommendation
varies
CGRE =
∑CL xi IW i
i
∑IW i
i
Complexities in Recommendations
•
•
•
•
commeasurate with GREs and GPA
can be a plus or a minus
are fundamentally optional
are not expected to have specific points
so no ranking factors
• vary in influence
so the importance weight computation is vital
Modeling Recommendations
Leading recommender is a
describing you as a
=
scaling factor
=
weight
warmth score
Summary of the Computation
Subtract Baseline and Scale
Raw to get Normalized Value:
Order Normalized Values
and apply Ranking Factors
to get Contribution Levels:
Weight and Sum:
NVi = (RVi - BV i) x SF i
CLi = NVi x RFi
2
RF =
3
( 1+
i
where r is rank,
n is number of scores
GQ =
r-1
n-1
∑CL xi IW i
i
∑IW i
i
)
Factors in Admissions Decisions
In the Model
• GREs
• GPA
• in-major or recent GPA
• major
• letters of recommendation
• statement of purpose
• scholarships
• group membership
Not in the Model
• undergrad school
• GRE subject test (CS)
• TOEFL
• nationality/culture
• specific coursework
• research match
• publications
• etc.
Evaluation
55 UTEP applicant datafiles
applicant features
accept/reject decisions
compute
GQ score
accept / reject
> -25?
compare
51/55
successes
with failures
explicable
Modeling Other Departments
published data
for school X
applicant data
threshold for
school X
compute
GQ score
>
accept / reject
compute
GQ score
Does the Model Work for Departments?
200
150
100
CGRE
50
0
-50
0
1
2
3
-100
-150
-200
-250
NRC Effectiveness
4
5
overall
Does the Model Work for Departments?
200
150
100
CGRE
50
0
-50
0
1
2
3
4
5
overall
trend
-100
-150
-200
-250
NRC Effectiveness
Thus selectivity, as measured by the model, correlates with desirability, somewhat
The Diversity Behind the Numbers
Minimum scores of 550, 600 and 3.5 on the verbal,
quantitative, and analytical writing sections, respectively
(U. of Delaware)
Average scores of successful applicants to this program for
Fall 2002: GRE: 560 verbal, 770 quantitative
(U. of Houston)
Most students admitted have earned scores in excess of
650 for the Analytical and Quantitative parts
(Columbia)
200
200
150
150
100
100
50
50
CGRE
CGRE
Averages, Minimums, and Thresholds
0
0
0
-50 -50
0
1
1
2
22
3 33
-100-100
-150
-150-150
555
inferred
threshold
-200
-200
-250
-250
444
average
conj. of mins
overall
most-above
overall
trend
minimum sum
average
conj. of mins
NRC
Effectiveness
NRC Effectiveness
200
200
150
150
100
100
50
50
CGRE
CGRE
Averages, Minimums, and Thresholds
0
0
0
-50 -50
0
1
1
2
22
3 33
444
-100-100
555
avg vs. threshold:
~20 (0.1 GPA points)
-150
-150
==> departments
don’t have much variety (?)
-200
-200
-250
-250
NRC
Effectiveness
NRC Effectiveness
threshold vs. min: ~30 (0.15 GPA points)
==> departments don’t take risks (?)
average
conj. of mins
overall
most-above
overall
trend
minimum sum
average
conj. of mins
inferred
threshold
A View of the Applicant Pool
Number of
Applicants
minimum
average
acceptees
Overall Applicant Strength (GQ score)
A Blurred View
Number of
Applicants
minimum
average
acceptees
Applicant Strength measured by GREs only
Modeling Other Departments
published data
for school X
applicant data
threshold for
school X
compute
GQ score
>
accept / reject
compute
GQ score
Modeling Other Departments
published data
for school X
applicant data
threshold for
school X
compute
GQ score
>
compute
GQ score
adjustment
accept / reject
description
adjustment
margin
most above
-30
40
average
-20
20
hard minimums
10
40
soft minimums
10
30
Presenting Uncertainty
Some Sources of Uncertainty
•
•
•
•
•
•
user interface errors
lack of information about the applicant
incorrect fundamental assumptions
incorrect GQ-model parameters
incorrect modeling of departments’ criteria
inadequate information on departments
Try it Yourself!
http://www.cs.utep.edu/admissions/
Future Work
• verification on data from more
departments
• better parameter estimates on more data
• a more parameterized version to model
different departments better
• a centralized clearinghouse?
Benefits for UTEP
•
•
•
•
better informs potential UTEP applicants
increases site traffic, and applicant pool?
increases Google score
shows we understand student needs
• makes the world a better place