Transcript Slide 1

QUALITY ASSURANCE FOR
ENHANCEMENT
An Exploration of Key Performance Indicators
for Academic Quality
Dr Jan Cameron
Assistant Vice-Chancellor (Academic)
University of Canterbury
Christchurch, New Zealand
New Zealand Institutional Quality
Assurance Framework
Ministry of Education
Education Review Office
(ERO)
(TEC)
New Zealand Qualifications Authority
(NZQA)
Schools
Polytechnics
Tertiary Education Commission
PTEs
New Zealand Vice-Chancellors’ Committee
(NZVCC)
New Zealand Universities Academic Audit
Unit (NZUAAU)
Committee for University Academic
Programmes (CUAP)
Universities
Introduction of Academic Quality
Measures in New Zealand
2003
TEC proposals for performance element related to teaching (PBTF)

completions; retentions; student satisfaction

intended “to enhance the provision of quality in terms of educational gains to learners”

objections by universities to retention and completion measures which “bear no relation
to the evaluation criteria of teaching performance”.
2033
NZVCC reiterates common practice academic quality assurance processes
2007-2008
Sector focus on developing acceptable quality assurance protocols

Joint TEC/NZVCC/NZUAAU working party to establish “evaluation indicators” for
universities – failed to agree on quality measures.

NZQA develops and pilots evaluation processes for non-universities.
Quality Measures: [Some of] the
Problems
 Validity – eg; retention and completion measures
 Reliability – eg; data quality
 Stability – eg; definitions of “part-time” status
 Utility – eg; measures of aspects which cannot be influenced by the
institution (eg; proportion of time spent in study); potential for
perverse effects
 Compatibility – between accountability and enhancement objectives
 Conceptual confusion – collapsing “quality” into “performance”
Specific New Zealand Concerns
TEC Objective
in conflict with
Proposed Performance Measure
Access
retention and completion
Staircasing
programme completion
Portability
retention and completion
Lifelong learning
retention and completion
Also
Open entry 20+
retention, completion, achievement
Quality of What?
Quality Input
Quality Processes
Quality Output
Quality Outcome
Who are the Users?
The Challenge
How to develop a new set of performance indicators which:
 Are relevant, valid and useful, to both the institution and to staff;
 Can be influenced directly;
 Can be used for benchmarking within the institution, both through
time and across academic units, as well as across institutions;
 Will spur enhancement initiatives;
 Will reflect achievements resulting from these, and
 Apply across and upwards through the institution
Development of Performance Indicators
Which are Useful for Academic Quality
Objective
Start from the input and quality assurance processes
Evaluate the outcome, as well as the output
Determine the appropriate measures
Closing the loop: using the quality outcome to inform the input
and/or quality processes
Input
Quality Assurance process
Outcome
Output
Example:
University objective: Excellent Learning
What student characteristics correlate with excellent learning?
(quality input)
What activities facilitate excellent learning? (quality processes)
How might excellent learning be identified? (quality outcome)
How might excellent learning be measured? (quality output)
What are the potential performance indicators? (quality measure)
Input
Students in specific
admission category
Potential PIs
Quality Assurance process
Identify students at
risk of poor
achievement
Intervention
(eg: provide learning
skills support)
Outcome
Output
Improvement in
group achievement
category over time
Mean GPA
Proportion of first years in “at risk” admission categories (indicates risk of low achievement, ie:
not achieving excellence – but might not be in university’s control).
GPAs or Pass rates for “at risk” categories of students over time (indicates likely impact of
identification and intervention – the quality assurance process).
Input
Students with poor
achievement,
defined in terms of
GPAs or pass rates
Potential PIs
Quality Assurance process
Academic progress
review identifies
students at risk
Academic progress
interventions
(eg: limit workload)
Outcome
Output
Change in
achievement
Actual GPAs or
pass rates
Proportion of students identified in year 1 who take advice or seek support to improve their
study (demonstrates the process delivers intended intervention – quality of the process).
Proportion of students identified in year 1 who continue study, who demonstrate positive
changes in year 2 (demonstrated interventions work – quality of the effect).
This performance indicator could be further disaggregated to differentiate the groups
involved and the nature of the intervention.
Example:
University objective: Excellent Teaching
What course characteristics correlate with excellent teaching?
(quality input)
What activities facilitate excellent teaching? (quality processes)
How might excellent teaching be identified? (quality outcome)
How might excellent teaching be measured? (quality output)
What are the potential performance indicators? (quality measure)
Input
Courses surveyed
on assessment
feedback
Potential PIs
Quality Assurance process
Mean score on
assessment
feedback
Intervention
(ie: workshops on
assessment
feedback)
Outcome
Output
Improvement in
mean score on
assessment
feedback
Mean score
Proportion of courses scoring lower than 4.0 on course survey assessment.
Mean score on assessment items in course surveys.
Development of Performance Indicators
from Quality Assurance Data
Some Cautions
Variability of outcome and output correlates with variability of input
Leads to potential for composition effects in aggregate scores
(the performance indicators)
Does composition matter?
 completion reflects full-time/part-time status
 retention reflects equity group composition
 achievement reflects prior school achievement
Example
Student achievement varies by admission category – so aggregate achievement will reflect
admission profile
Table 1: Proportion of admission groups by degree programme
Whole university
BA
BCom
BSc
LLB
A Bursary
0.23
0.20
0.21
0.35
0.37
B Bursary
0.26
0.22
0.27
0.31
0.28
Minimum Entrance
0.25
0.25
0.3
0.22
0.19
Adult Admission
0.23
0.33
0.22
0.12
0.15
Total Number
2,306
755
506
785
260
Mean GPA
3.722
3.731
3.401
4.042
3.835
Solution
Standardise for admission profile
Table 2: Degree First Year GPAs Standardised to Whole University
Number at
University
University
GPA
BA
BCom
BSc
LLB
A Bursary
598
5.7
5.6
5.8
5.8
5.5
B Bursary
594
3.9
4.4
3.9
3.6
3.9
Minimum Entrance
583
2.9
3.1
2.6
2.9
2.4
Adult Admission
531
2.2
2.6
1.7
2.2
1.3
Total
2,306
3.722
3.731
3
3.401
4
4.042
1
3.835
2
Standardised GPA
3.968
3.557
3.671
3.337
Rank on standardised GPA
1
3
2
4
Crude mean GPA
Rank on crude GPA
Example
Teaching performance, as measured by teaching evaluation scores, varies by discipline
area.
Mean score on Q “overall this was a very good course” 1 - 5
Humanities and Social Sciences, Music and Fine Arts
4.0
Science
3.8
Commerce
3.9
Law
4.0
Engineering and Forestry
3.6
Problems
If change the proportion of courses within each category, but have no
change in course scores, the aggregate mean changes – even though
there has been no actual change in scores
Year 1, Base Year Data
Number of
Courses
Mean
Score
BA
900
4.2
BSc
300
4.1
BE
350
LLB
Year 2, Number of high-scoring Education
courses halved
Number of
Courses
Mean
Score
BA
900
4.2
3.5
BSc
300
4.1
120
3.8
BE
350
3.5
BCom
250
4.0
LLB
120
3.8
BEd
1,200
4.3
BCom
250
4.0
BFA
30
3.2
BEd
600
4.3
BFA
30
3.2
Crude mean score
4.11
Crude mean score
4.06
Solution
Standardise the scores to the original course profile composition
Table 3: Impact of profile composition changes on overall teaching scores
Year 1, Base Year Data
Number of
Courses
Mean
Score
BA
900
4.2
BSc
300
BE
Year 2, Number of high-scoring Education
courses halved
Number of
Courses
Mean
Score
BA
900
4.2
4.1
BSc
300
4.1
350
3.5
BE
350
3.5
LLB
120
3.8
LLB
120
3.8
BCom
250
4.0
BCom
250
4.0
BEd
1,200
4.3
BEd
600
4.3
BFA
30
3.2
BFA
30
3.2
4.11
Crude mean score
Crude mean score
Standardised mean score
4.06
4.11
The decrease in the crude mean score is spurious. The overall standard actually stayed the
same.
Year 1, Base Year Data
Year 2, Number of high-scoring Education
courses halved, scores for BSc and BE
improve
Number of
Courses
Mean
Score
BA
900
4.2
BSc
300
4.1
BE
350
3.5
LLB
120
3.8
BCom
250
4.0
BEd
1,200
4.3
BFA
30
3.2
Crude mean score
Standardised mean score
4.11
Number of
Courses
Mean
Score
BA
900
4.2
BSc
300
4.2
BE
350
3.6
LLB
120
3.8
BCom
250
4.0
BEd
600
4.3
BFA
30
3.2
Crude mean score
4.06
4.13
The standardised mean shows that decrease in crude mean score is spurious: even though
there has been a reduction in the number of high scoring courses the overall standard has
actually improved.
Differing actual (or crude) figures reflect different forces at work in
producing change in overall score.
Comparison of aggregate
measures across institutions or across time is robust only so long as
composition remains stable. If composition varies either through time
or across institutions then this must be controlled for.
Direct
standardisation is a simple of way of exerting this control.
Summary

Performance indicators should guide and facilitate enhancement.

Developing performance indicators out of the quality processes (not vice versa).
a)
identify correlates of variation in the “quality” to be measured;
b)
identify how the quality variable might be measured (check validity and
reliability);
c)
check for composition difference on the correlate variable;
d)
if differences emerge, standardise for these;
e)
if there are composition differences, calculate aggregate KPI on basis of
standardised measure;
f)
use variation in measures on the correlates to identify potential points of
intervention for enhancement (closes the loop).
The process of disaggregating and re-calculating summary measures
opens up the possibility of using the same data set as a quality
assurance measure (ie: identifying points of variation for possible
intervention) and as a more robust performance indicator (controlling
for some variation which has a direct influence on performance). In
this way performance indicators can be used to facilitate the quality
agenda, as well as reflect it.
Thank you!

Questions?