Transcript Slide 1

31st Annual EAIR Forum
Vilnius, Lithuania
23 - 26 August 2009
Y R Mc Nicoll
A R Luff
Is Quality Improvement fit for
purpose? Comparing quality
systems within a major Australian
University
Background
• No universal accreditation of degree programs
• In 2002 Australian Universities’ Quality Agency
adopted ‘fitness for purpose’, QI approach:
– “does not impose an externally prescribed set of
standards upon auditees [our emphasis]. AUQA
considers the extent to which institutions are
meeting these objectives, and how institutions
monitor and improve their performance.
• Bradley review (2008) advocates
Standards
2
More Background
• Who are we?
• Society and HE student body are
significantly multicultural
– in 2007, 455,000 international students
predominantly from South East Asia
• Monash University
• Faculty of Medicine, Nursing & Health Sciences
• Quality process trial
> 3 participating faculties
3
The problem
• This paper presents observations about the effects
of a quality improvement based system,
– attempts to identify the value added, and to
– determine whether the time, effort and money invested in
the process offer a good return.
• How effective are quality reviews at identifying the
real problems?
• What problems can they create?
• What value do they add?
• Do they necessitate management reviews to
address specific problems?
• Are the two kinds of reviews complementary?
4
For whose benefit do we “do quality”?
• Universal suffrage: interests of society
are synonymous with those of
government
– Education debate is limited
• University interests:
– Students
– Academics
– Leadership
5
Methodology
• Significance
– Monash places a high priority on excellence
in teaching and learning and of the overall
“student experience”
– clear implications for Australia’s education
export market
• Data
– 3 quality reviews: 2 courses and 1 faculty
– 1 ad hoc management review
6
Principles for Quality: What sort of
Quality?
•
•
•
•
•
•
•
fitness for purpose;
quality is the professional responsibility of each individual and work
group;
the best way to effect quality assurance and accountability is through
continuous quality improvement via collaboration and organisational
learning;
policy should assure comparable treatment in all areas of the
university, while leaving room for different areas to develop
implementation suitable to their contexts;
an open, thoughtful and complementary approach to quality informed
by international research and scholarship;
a planned and systematic approach to quality including ensuring that
the results of monitoring and evaluation are fed back in order to effect
improvement;
external points of reference provide valuable perspectives for further
7
reflection and action.
Process 1: Quality review
• Broad-sweep, ‘helicopter view’
• Improvement-oriented process
1. Self-review followed by
2. External validation by a panel,
> Interviews stakeholders, site visits, and
document review
• Variations to process permitted
–
Refinements attempt to
> Balance student interests with FMNHS needs
> Align with university direction
8
Process 2: Management review
• Conventional, but unformalised
• At the Dean’s discretion
• Single- step review by a largely internal panel
with a brief to address specific problems.
• In the FMNHS such ad hoc reviews are not
uncommon.
• This paper refers to the FMNHS process,
drawing on the Biosciences teaching review
report
– Access to equivalent reports of other organisational
units is limited due to their sensitivity
9
Purpose & Scope: Quality review
• Terms of Reference (ToRs)
– Course: Structure, Management (including planning, QA,
development), Units and Major & Minor Sequences;
Student Profiles, plus
– in common with review of Faulty organisation:
> Teaching, Learning & Assessment; Human, Physical &
IT Resources, and Health & Safety issues; Professional
and Community Engagement.
• Concatenation with professional accreditation
requirements has been rare
• FMNHS guidance
– Elaboration of ToRs: for convenors
– External ToRs: for panels
10
Subtle differences of purpose
• BSc program review report
– “to test and validate (or otherwise) the findings of the Self
Review …;
– ... “to assess and test the quality and adequacy of the course in
meeting its objectives, and follows the guiding principles
outlined in the Course Review Policy”.
• BCom review does not identify a purpose,
• FMNHS identifies a two-fold purpose...
– reflection within the organisation,
– identifying strengths, weaknesses threats and
opportunities for improvement,
– [seeking] the perspective of outside experts (Williams and
11
et al 2007, p.12).
What value do quality reviews add?
• Accommodate discipline differences
– Speak the language of the discipline
• See further or more clearly than selfreview teams
• Reinforce each other’s findings with
common recommendations
• Serve both professional and generalist
degrees
12
How effective are these reviews at
identifying the real problems?
• Do the skeletons tumble out of the closet?
–
–
–
–
–
–
Absence of (effective) course management
Understaffing
Inadequate accountability for improvement
Poor rationale for course structure & content
Disadvantageous and draconian assessment
Confused marketing
• Bold, wide-ranging recommendations emerge
– Regardless of panel composition,
– Panel experience or guidance is key
13
What problems does quality review
create?
• Recommendations
–
–
–
–
That the evaluand has no power to address
Political recommendations to pressure the university
Conflicting recommendations
Multiple recommendations
• Compliance
– With recommendations
– Monitoring of quality review process
> Stricter enforcement by a central unit is required to
avoid escapees
14
Lessons and implications
• Quality improvement
– Pros
>
>
>
>
External panels overcome groupthink
Comparable treatment,
Scope for creative improvement,
Collegial approach
– Cons
> QI is ineffective in face of
– Unprofessional behaviour
– Poor accountability
> Does not resolve
– contradictions of applying managerial methods in the academic
environment
15
Lessons and implications
• Ad hoc management reviews
– Pros
> Can be generated at a Dean’s discretion
> Delve into serious problems
– Including those revealed by quality processes.
> Existence of two review processes has sometimes proven
necessary to overcome the abuse of trust to which universities
can be vulnerable.
– Cons
> Does a need for two processes suggest inefficiency?; or
ineffectiveness?
> Have we created an industry?
> What is the cost/benefit of conducting (quality) reviews in this
manner?
16
Conclusions
• Quality activities are obviously an industry
– Only a problem if the process drives down the
performance of Higher Education.
> The impact of quality activities is difficult to identify,
although INQAAHE leaders report many positive signs
(Harvey 2006).
• Debate regarding standards versus improvement
may be ill-posed.
– Quality reviews are quite effective at identifying real
problems, but they have limitations.
– They do not solve the long-standing problems of
accountability that form the obverse to academic freedom,
but we are unaware that any other process has resolved
this dilemma.
17
Conclusions
• Take care that the value added by QI is not lost in the
urgency to account for inputs by measuring outputs
• HEIs are complex, as is academic endeavour:
– a sophisticated approach to quality seems indicated.
• Should we enhance the complementary nature of the two
review processes to achieve a solution:
– Formalise the relationship as collegial, with option to police
– Rationalise their demands
– Continue to Invoke management review only as needed
• It is imperative to improve the student experience, and to
protect education exports:
– the effort seems warranted
18
Contact
Professor Tony Luff
Associate Dean (International)
Faculty of Medicine, Nursing & Health
Science
Monash University
AUSTRALIA
[email protected]
19