Transcript Thriving at the Liberal Arts Colleges: What We are Learning in Predictive Modeling in Higher Education (PPT)

Thriving at the Liberal Arts Colleges:
What we are learning in predictive modeling in higher education
At Grinnell College
David Kil, Chief Data Geek, Civitas Learning
April 8, 2016
CIVITAS LEARNING, INC. – CONFIDENTIAL INFORMATION
My Life: Quantified Self  3-year NIH grant on spreading good health
behaviors through social network and nudging  20+ papers and 5 PhD’s awarded
Min-by-min Activity Pattern: Co pa raw 20
10 mph
½ marathon
No
device
3:20
6:40
10:00
Seoul
13:20
Spain
16:40
20:00
23:20
160
155
0 mph
Steps per day (K)
Weight (lbs)
40
Marathons
20
150
145
0
# friends
40
20
0
100
50
0
Wellness meter
Social reputation score
10/05/13
10/07/02
10/08/21
10/10/10
10/11/29
11/01/18
CIVITAS LEARNING, INC. – CONFIDENTIAL INFORMATION
11/03/09
11/04/28
11/06/17
Common Thread?: Relevance to students thriving at
Liberal Arts colleges?
CIVITAS LEARNING, INC. – CONFIDENTIAL INFORMATION
Motivation: Help students get maximum benefit out of college
Connect to
lifelong friends
& mentors
Maximize
graduation/job
rates in right
major
Develop soft
skills for
networked
economy
Lifelong
student
success
CIVITAS LEARNING, INC. – CONFIDENTIAL INFORMATION
“Business and engineering schools do
a pretty good job when it comes to
teaching hard skills. But they don’t do
so well with the softer competencies
companies are also looking for. Finding
and retaining talented individuals with
this mix of capabilities is a challenge
regardless of geography or industry.” –
WSJ, 3/4/2016
Predicting Risk Alone Is Insufficient – Connected Analytics:
Predicting success/risk to engagement to impact
Multi-level
linked-event
feature
extraction (why)
Lifecycle
management
Impact results
packaging for
insertion into
evidence-based
knowledge DB
Micro-pathway
student success
prediction
(Who)
Student
engagement
prediction
(When, where)
Action: Microintervention
delivery
Evidence-based
action knowledge DB
Program impact
analysis using
PPSM on termlevel student
success metrics
CIVITAS LEARNING, INC. – CONFIDENTIAL INFORMATION
Student impact
prediction &
prioritization
(What, how)
Treatment
impact analysis
using dynamic
PPSM on
exposure to
nudge
Tier-1 impact
analysis using
engagement
rule – KPI
mapping
Tier-2 Example with Real Data
Targeted emails to middle 60% of students by engagement score
(approx. SES range is 0.4-0.6)
Day -1 to +6 pilot vs. control lift in SES = 0.015
p value = 0.00016
p value on
pilot engagement score
diff of diff
0.505
Control engagement score
Engagement Score
0.5
0.495
0.49
0.485
0.48
-8
-6
-4
-2
0
2
days
before/after email
CIVITAS LEARNING, INC. – CONFIDENTIAL
INFORMATION
4
6
8
10
Conditional Probability Table (CPT) Analysis Extended to Faculty
for RT Coaching: The higher N, the greater hierarchical CPT depth (Example on student type,
engagement score bucket, experience, days into section)
Need a better
strategy for
dealing with
students with
low predictions
The CPT view gives a
comprehensive dashboard
to help the faculty optimize
outreach strategies.
High-engagement
students respond better
in general
CIVITAS LEARNING, INC. – CONFIDENTIAL INFORMATION
Aha! I need to
start my section
the right way!!!
Targeted emails
are about twice
more effective for
new students
Refining High-Risk Student Outreach with CPT
Analysis
Part of
product
roadmap
Divide bottom
1/3 students
into 3 groups
Divide students
into risk buckets
1.
2.
Develop
outreach
strategies for
pilots 1 and 2
Pilot 1 – mindset
Pilot 2 – faculty
mentorship
Control
3.
Key findings
• Moderate-risk students
• Special outreach for high-risk students
• Contextual, personalized triggers
• Short, NABC nudges with empathy
• Micro-pathway prediction
• Earlier term, new students
CIVITAS LEARNING, INC. – CONFIDENTIAL INFORMATION
Predictions over time
Check pilot
status using
prediction score
trend
Perform formal
outcomes
analysis at
census
Strong agreement
Decomposing Pathways: Nudging, Impact, Results
Sort courses by
enrollment per
major – top 60120 courses
Group similar
courses based on
course features &
grade correlation
between courses
Create
concurrent &
forward course
adjacency tables
CIVITAS LEARNING, INC. – CONFIDENTIAL INFORMATION
Create coursesequence Markov
chain
Perform 3-tier
PPSM on feasible
course sequences
& pathway
nudges
Estimate datadriven requisite
graphs & forecast
course demand
Design nudge
rules &
personalized
content with NLP
Degree
requirement
business rules
Evidencebased action
knowledge
Time-Series Data, Privacy, and Small N
•
•
•
•
Micro-pathway prediction
More context-aware nudging
Improved student experience
Near real-time impact analysis 
faster impact insights
• Greater N for model robustness
• Improved ROI
Examples
1. HRA design
2. Social network/
nudging – NIH
3. Happy Body
Human
values
Data
science
Design
thinking
CIVITAS LEARNING, INC. – CONFIDENTIAL INFORMATION
• Greater burden on
stakeholders  data and
action (Person of Interest)
• Creepiness factor (Minority
Report)
• Data security and student
privacy
Conclusion
• Connecting the last mile in data science
• Accelerate pathways from Data to Insight to Action to
Learning
• Importance of design thinking and science to
provide greater value to user while respecting user
privacy
• People analytics = an emerging field
• Opt-in and informed consent for passive data collection
• Data control
• Data aggregation, anonymization, and differential
privacy for research
• Trust and transparency, focusing on human values
CIVITAS LEARNING, INC. – CONFIDENTIAL INFORMATION