Transcript Sparse Factor Analysis for Learning Analytics Andrew Waters, Andrew Lan, Christoph Studer, Richard Baraniuk Rice University.

Sparse Factor Analysis for
Learning Analytics
Andrew Waters, Andrew Lan,
Christoph Studer, Richard Baraniuk
Rice University
Learning Challenges
One-size-fits-all
Inefficient,
unpersonalized
Poor access to high-quality materials ($)
Slow feedback
cycle
Personalized Learning
Adaptation
– to each student’s background,
context, abilities, goals
Closed-loop
– tools for instructors and students
to monitor and track their progress
Cognitively informed
– leverage latest findings from the
science of learning
Automated
– Do this automatically data
Data
(massive, rich,
personal)
Jointly Assess Students and Content
Latent factor decomposition (K concepts):
•
•
•
•
Which concepts interact with which questions
How important is each concept for each question
Which questions are easy / difficult
How well have students mastered each concept
Do this solely from binary Q/A (possibly incomplete) data
Statistical Model
Partially
observed
data
Inverse link function (probit/logit)
Intrinsic difficulty
of Question i
Concept mastery of Student j
Concept weight for Question i
Model Assumptions
Model is grossly undetermined
We make some reasonable assumptions to make it tractable:
- low-dimensionality
- questions depend on few concepts
- non-negativity
• SPARse Factor Analysis (SPARFA) model
• We develop two algorithms to fit the SPARFA model to data
SPARFA-M: Convex Optimization
Maximize log-likelihood function
• Use alternate optimization with FISTA
[Beck & Teboulle ‘09] for each subproblem
• Bi-convex: SPARFA-M provably converges to
local minimum
SPARFA-B: Bayesian Latent Model
Use MCMC to sample posteriors
C
μ
Z
Y
Efficient Gibbs’ Sampling
Assume probit link function
W
Sparsity
Priors:
Key
Posteriors:
Ex: Math Test on Mechanical Turk
-0.50
2.37
3.98
0.92
1.10
1.40
3
0.29
1.05
1.03
0.79
High School Level
0.45
1.26
-0.54
-0.04
34 questions
100 students
4
1.37
0.27
0.89
1.92
5
2.13
-0.81
0.87
0.82
1.40
0.56
SPARFA-M
w/ 5 concepts
0.50
2
2.07
2.37
-0.46
1
-1.32
1.05
1.47
-1.42
1.62
-0.67
Visualize W, μ
Tag Analysis
Goal: Improve concept interpretability
Link tags to concepts
T1
C1
T2
C2
.
.
.
.
.
.
TM
CK
Algebra Test (Mechanical Turk)
-0.50
2.37
3.98
0.92
1.10
1.40
3
0.29
1.05
34 questions, 100 students
1.03
0.79
0.45
1.26
-0.54
-0.04
4
1.37
0.27
0.89
1.92
5
2.13
-0.81
0.87
0.82
1.40
0.56
0.50
2
2.07
2.37
-0.46
1
-1.32
1.05
1.47
-1.42
1.62
-0.67
Concepts decomposed into
relevant tags
Synthetic Experiments
Generate synthetic Q/A data, recover latent factors
Performance Metrics:
Compare SPARFA-M, SPARFA-B, and non-negative variant of K-SVD
EW, N=100, K=5
0.5
m
0.75
0.75 Q=50
Q=200
Q=50 Q=100
Q=100 Q=200
Q=50 Q=100 Q=200
0.5
0.5
m
0.25
E
K=5
Em,, N=100,
N=100, K=5
E
Em
Q=50 Q=100 Q=200
EC
EW
0.5
EC, N=100, K=5
0.25
0.25
0.25
0
MB K
MB K
MB K
0
MB K
MB K
MB K
00
MB
BK
K M
M
MB
B KK M
MBBKK
Ex: Rice University Final Exam
7.64
7.64
3.48
0.71
7.64
7.64
7.64
3.07
2.83
Signal processing course
0.69
0.87
1
2.93
7.64
2.08
2.83
5
1.73
2.11
1.95
2.62
-2.64
1.64
2
2.07
7.64
2.49
-0.72
1.96
1.22
3
0.77
2.16
3.00
2.07
4
3.13
44 questions
15 students
100% observed data
3.48
7.64
3.48
3.27
3.48
0.02
7.64
3.48
3.00
3.00
-0.85
0.12
SPARFA-M, K=5 concepts
Student Profile
Student Profile: Student’s understanding of each Tag
Average Student Profile on Rice Final Exam
Student 1 Profile on Rice Final Exam
SPARFA automatically decides which
tags require remediation
STEMscopes
0.08
1.82
-0.25
4.74
4.78
8th grade Earth Science
80 questions
145 students
0.34
1.08
0.16
0.66
2.14
0.77
1.41
5.04
2.68
1.70
0.84
1.37
1.03
0.01
1.33
1.51
1.00
-0.10
0.21
2.42
2.14
1.31
-0.25
5
1.56
1.14
-0.59
0.79
1.90
-0.10
2.14
0.83
-0.10
0.98
1
1.60
2.23
2.05
0.02
1.21
1.24
0.64
2.28
0.11
0.70
0.61
2.58
3
1.37
0.93
0.06
1.08
1.59
0.44
-1.10
0.56
4
0.00
1.25
2
1.31
1.53
2.52
-0.75
0.81
-1.32
2.19
2.16
0.22
0.81
1.28
1.06
-1.19
1.61
2.58
1.71
0.71
4.91
5.05
2.14
SPARFA-B: K=5 Concepts
Highly incomplete data:
only 13.5% observed
STEMscopes – Posterior Stats
Randomly selected students
Single concept
(Energy Generation)
Student 7 and 28 seem
similar:
S7: 15/20 correct
S28: 16/20 correct
Very different posterior variance:
Student 7:
Student 28:
Mix of easy/hard questions
Only easy questions – cannot determine ability
Conclusions
• SPARFA model + algorithms fit structural model
to student question/answer data
– Concept mastery profile
– Relations of questions to concepts
– Intrinsic difficulty of questions
SPARFA can be used to make automated
feedback / learning decisions at large scale
 Go to www.sparfa.com