Transcript Metrics and Money: - Information technology

Metrics and Money:
The Process and Politics of
Accountability
Stephen Daigle, Ph.D, California State University
Michael Large, Ph.D, Social and Behavioral
Research Institute, CSU San Marcos
Patricia Cuocco, California State University
1
Overview
• ITS and the Measures of Success (MOS)
• MOS Research Agenda
• Findings
• Questions (and, we hope, answers)
2
Integrated Technology Strategy
• Early 1990s – Presidential Interest in Using
Technology As a Strategic Tool Driven By:
–
–
–
–
Immense Growth in Technology,
Antiquated Legacy Systems,
Increasing Demand for Access
Fiscal Constraints
• Mid 1990s – Develop Integrated Technology
Strategy Framework
– Iterative Process, Constituent Input and Approval
3
ITS – The Process
• 1996 Accepted by Board of Trustees
• Living Framework – Not Written Plan
• 2003 – Technology Decisions Still
Governed by ITS
4
What Is ITS
• Outcome Driven
– Excellence in Learning and Teaching
– Quality of the Student Experience
– Administrative Quality and Productivity
– Personal Productivity
5
ITS – the Icon
6
ITS – How It Fits
• Personal Productivity - Attained Through
Infrastructure Initiative – Minimum
Baseline at All CSU Campuses
• The Infrastructure Is Critical If ITS Benefits
Are to Extend to All Students, Faculty and
Staff
• The Infrastructure Requires Money
($250M) = PROBLEM
7
Getting the Infrastructure
• Legislature Challenged CSU to Be
Creative
• CSU Was Too Creative
• Public/private Partnership Made
Legislature Uncomfortable
• Californians Passed a Bond
• Presidents Decided Infrastructure Had
Priority Over New Buildings
8
Getting to “Yes” *
• Bond Expenditures Require Legislative Approval
• Legislature Wanted to Tie Investment in
Infrastructure to “Learning Outcomes”
• Not What ITS Is About
• Infrastructure Enables ITS Initiatives Which, in
Turn, Enable Outcomes
Apologies to Fisher, Ury and Patton
9
Negotiating Accountability
• Worked With Legislature – Agreed on
What Could Be Measured
• Developed 10 Year Process for Measures
of Success
• Framework
• Baseline
• Changes to the Baseline
10
Types Of Data Collected
•
•
•
•
System
Campus
Individual (Student, Faculty, Staff)
External
11
Scope Of Data Collection
•
•
•
•
•
•
23 Campuses; 1,000 Miles
400,000 Students
35,000 Faculty And Staff
10 Year Commitment
Institutional Comparisons Across Time
S, F, S Comparisons Across Time
12
Technology User Survey
Samples
• Students: Campus, Class Level, Ethnicity;
23 X 5 X 9 Cells
• Faculty: Campus, Rank, Discipline; 23 X
4 X 8 Cells
• Staff: Campus, Job Classification; 23 X 7
Cells
13
CATI LOGISTICS
(COMPUTER ASSISTED TELEPHONE INTERVIEWING)
•
•
•
•
No Self-select As With Mail
About 100 Questions; 20-30 Minutes
Importance Of Skip Facility
Instant Database Creation
14
CATI (Continued)
• Standardized Context Provided (E.G.,
High Speed Network)
• Trained Interview Staff, Monitoring,
Evaluation
• Refusals Less Than 2 Percent; Still Over
Sample
• $75 To $100k Per Survey, But Systemwide
15
User Outcomes
Hardware
Software
Network
Support
Training
ACCESS
USE
SATISFACTION
16
Examples of Metrics
• Institutional
–
–
–
–
Library Cost Avoidance
Smart Classrooms
Data Center Savings
Training and Support Spending
• Individual
–
–
–
–
Computer Ownership
Network Use
Help Desk Satisfaction
Use of Administrative Systems
17
Findings and Data
18
Findings
Hardware Software Network Support Training
Access
Use
Satisfaction
19
Hardware
• Access
– Students’ access to hardware
• Use
– Percent of Faculty Requiring Computer Use
• Satisfaction
– Faculty satisfaction with Teaching in Computer
Labs
20
Software
• Access
– Faculty access to software
• Use
– Percent of Students Using Components of Their
Student Information System
• Satisfaction
– Staff Satisfaction with Software
21
Network
• Access
– Students’ access to wireless networking
• Use
– Staff use of network from off-campus
• Satisfaction
– Staff satisfaction with e-mail
22
Student Access to a University-provided Computer
100
94
94
80
60
40
Percent
Administration
20
0
2001
6
2003
6
No
Yes
Access to a University-Provided Workstation
23
Percent of Faculty Requiring Students to Use a Computer
100
85
80
60
64
40
Percent
36
ADMINISTRATION
20
2000 AY
15
0
2002 AY
No
Yes
Required Students to Use Computer
24
Faculty Satisfaction with Giving Instruction in a Computer Lab
10.0
9.0
8.0
7.8
7.0
6.0
7.4
6.4
6.7
5.0
4.0
3.0
Technical Support
2.0
Pedagogical
1.0
0.0
Effectiveness
2000 AY
2002 AY
SURVEY ADMINISTRATION
25
Faculty Access to University-provided Software Needed
100
96
94
80
60
40
Percent
ADMINISTRATION
20
2000 AY
0
2002 AY
6
No
Yes
Access to University-Provided Software
26
Percent of Students Using Their Campus Student Information
System
Information about
100
Registration
86
80
60
67
88
Information about
Grades
71
Information about
Financial Aid
40
41 40
Information about
Billing
29
20
37
20
22
Degree Progress
0
Information
2001
2003
Administration
27
Staff Satisfaction with University-provided Software Available
40
30
29
25
21
10
12
Co
d
ie
d
ie
f
tis
Sa
9
f
tis
Satisfaction with Software
8
7
5
6
S4a
ll
tA 3
2
1
5
5
a
ot
0
y
el
et
pl
m
N
Percent
20
28
Percent of Students with Laptops and Wireless Capability
100
90
80
70
60
50
40
30
36
Percent
20
17
10
0
Owns a Lap-Top
Wireless Networking
29
Staff Use of University Network from Off-campus
70
60
60
50
50
50
40
40
30
Year of Study
Percent
20
10
2000
0
2002
No
Yes
Accessed Network from Off-Campus
30
Staff Satisfaction with E-mail Services
Q4 B1 B Satisfaction with Camp us E-mail Serv ices
ADMIN Year of Stud y
Mean
N
Std. Dev iatio n
1 2 00 0
8 .74
2 22 4
1 .47 6
2 2 00 2
8 .86
2 12 3
1 .34 4
Total
8 .80
4 34 7
1 .41 5
31
Importance of Providing Electronic Access to Course Instruction
Any Time and Place
10.0
9.0
8.0
7.0
Mean Importance
6.0
5.0
5.0
5.1
5.0
4.0
3.0
2.0
1.0
0.0
Professor
Assistant Professor
Associate Professor
Academic Rank
32
Comparison between Faculty and Students in Perceived
Importance of Providing “Any-time, Any-place” Instruction
Q1 A3 Impo rtan ce of Prov id ing Electron ic Access to Co u rse In stru ctio n
fo r Stu den ts at Any Time an d Place
Gro u p
Mean
N
Std. Dev iatio n
Facu lty - 20 0 2
5 .07
3 16 7
3 .04 0
Stud ents - 2 0 03
8 .11
3 08 6
2 .15 6
33
Uses of the Data
•
•
•
•
•
Accountability
Description of the Population of Interest
Change Over Time
Subgroup Comparisons
Cross-group Comparisons
34
Conclusion
• Expensive – but You Get What You Pay For:
Valid
• Reliable
• High Level of Confidence
• Negotiate – Don’t Be Passive – Be Proactive
• Accountability Provides Cover
• Infrastructure Is a Utility and a Prerequisite
• Strategic Planning Is Dynamic – Change/add
35
Look For Yourself
http://its.calstate.edu/documents/Data_
Collection/I_Reports_MOS/Measure_of
_Success.shtml
36
Copies of the Presentation
http://its.calstate.edu
37