Transcript Document

Welcome
1
Paddy Maher
Emeritus Professor, University of Highlands and Islands and
QAA consultant
Approaches to Self-Evaluation in
Scottish Higher Education Institutions
2
Quality Enhancement Framework
1.
2.
3.
4.
5.
Institution-led quality reviews
QAA Enhancement-Led Institutional Review
Public information
Student engagement
Enhancement Themes
3
Institutional approaches to selfevaluation (IASE) project
A QAA Scotland project in Academic Year 2012-13 which aims to:
• Look at the mechanisms that HE Institutions use to evaluate
learning and teaching practices and identify effective features
• Encourage institutions to reflect on and share experience of
their approaches to self-evaluation
4
Scope: monitoring and review processes
Annual
Monitoring
Periodic
Review
Institutional
Review: ELIR
(or internal)
5
Methods
• Literature survey
• Consultation
• Identification of factors facilitating effectiveness
• Dissemination of findings
6
The wider evaluation literature
• Evaluation is a growth area but little about internal reviews
– ‘The limited number of papers on internal QA demonstrate how the
quality debate has been dominated by the activities of external agencies.’
[Harvey and Williams, 2010]
• Lack of trust in top-down, externally-driven QA models
– ’...20 years of QA has seen the systematic misalignment of quality culture
and academic culture. Furthermore the situation is getting worse ...
...(with notable exceptions, such as Scotland)...’ [Harvey, 2010]
7
Developing trust: recognising diversity
• ‘A modern university?
– ‘...a series of separate schools and departments held together by a
central heating system.’
[attributed to Robert Hutchins]
• Differing social practices
– ‘.... quality and evaluative approaches which ignore the social
practices of those involved risk alienating staff and introducing
unsustainable initiatives.’
[Bamber, 2011]
8
Insights from Sweden
[Eva Åkesson V-C, Uppsala University, HEIR Conference, 2012]
• Developing trust: ‘Staff and students are our university’s biggest asset, and
a deep trust in their willingness to be professional and committed should
be the corner stone of the quality system.’
• Aligning academic and quality cultures: ’Monitoring of the quality...should
be framed as a scholarly approach to professional learning and
development, not as a control system...internal evaluations should again
be framed as institutional research, i.e. efforts to understand our university
and learn to develop it.’
9
Consultation about evaluation processes
and their effectiveness
• Interviews
– Representatives of 8 Scottish HE Institutions
• Survey questionnaire
– Completed by 30 respondents from 13 Scottish HE Institutions
• Groups consulted:
–
–
–
–
–
ELIR Reviewers
Scottish HE Enhancement Committee
Universities Scotland Teaching Quality Forum
sparqs (student participation in quality scotland)
QAA Scotland
10
Annual Monitoring
• AM ‘routine’ but taking on greater significance: e.g., ‘A little
more conversation’
– A monitoring group (varying in composition) meets the subject team
to discuss an annual report (varying in size)
– Timing and nature of meetings vary but feedback is certain and rapid
• Data sets: improved quality and reliability
• Positive developments in student feedback: e.g.,
– Internal online surveys
– Strategic approach to external surveys, e.g., NSS
– Better loop closing: ‘You said, we did’
11
Periodic Review – staff engagement
• Engage staff by demonstrating the academic value of review
– e.g., staff contribute more to review agenda; better linkage with PSRB
accreditations; develop skills of reflection; review panel membership
as professional development; etc.
• Use annual monitoring reports more as the QA base to
reduce new paperwork and allow the review to concentrate
on evaluation and enhancement
• Make better use of externals’ subject/theme expertise in the
enhancement element of the review
12
Student engagement
• Quoted as the most positive and effective feature of quality
processes; for example:
–
–
–
–
Students’ enthusiastic involvement in reviews
Valued role of sparqs in training student reps
More effective Staff Student Liaison Committees,
etc., etc.
• Challenges?
– Low survey response rates
– How to engage students from harder-to reach categories: e.g. , DLs
• Several respondents would like to learn more from other
institutions’ approaches to student engagement
13
Preparing for ELIR
2013 cf 2008 (Good Practice in Reflective Analysis)
• More positive view of opportunity for institutional reflection
– ‘The ELIR provided an opportunity to step back and reflect on the many
activities and processes that underpin quality enhancement thereby
ensuring a holistic review of our approach’
• Characteristics of a ‘good’ RA: 2013 view very similar to 2008
– Use for updating commentary on Good Practice in Reflective Analysis
• Average length of RA ~doubled between ELIR 1 and ELIR 2
– Advance Information Set in ELIR 3 may reduce RA length again
• Similar preparation approaches across HEIs:
– ELIR steering group with one or a few authors and gathering evidence
and opinion from working groups
• Desire for greater sharing of experience of ELIR preparation
14
Initial conclusions
• Stability and change
– Evolving but relatively stable QEF gives HEIs reassurance to
develop processes that suit their requirements
• Vigour and rigour
– Dynamic sector with ‘continual improvement process’
– Monitoring and review rigorous and meeting SFC guidelines
• Assurance and enhancement appear to be well-balanced
– Respondents: institution-led evaluation leads to enhancement
• Effectiveness of self-evaluation processes?
15
Monitoring and review are more likely to
be effective when ...
TRUSTED

Giving confidence to and developing trust of external and
internal stakeholders: balancing assurance and enhancement

Answering the ‘so what?’ question

Based on teamwork and well supported by senior managers

Involving open discussions between reviewers and reviewed

Engaging a wide spectrum of students as reviewers and providers
of evidence, and fully utilising sparqs expertise and support
16
Monitoring and review are more likely to
be effective when ...
ALIGNING QUALITY AND ACADEMIC CULTURES


Flexible enough to recognise differences between subjects in
culture and practice
Framed as a scholarly approach to professional learning and
development and/or institutional research
KEPT SIMPLE AND FOLLOWED THROUGH



‘What’s working? What’s not? What needs to change?’
Leading to enhancement plans with SMART targets and closing
loops
Monitoring, review and ELIR preparation are linked, and are
themselves regularly reviewed
17
What’s next?
• Updating ‘Good practice in Reflective Analysis’
• Project report
• Dissemination workshop
18
19