Transcript Grown-ups like numbers…” The Role of Quality Assurance in

“Grown-ups like numbers…”
The Role of Quality Assurance in Higher Education
Stephan Neetens
Bologna Process Expert - Flemish Community of Belgium
SSU Conference, Rogaška Slatina, Slovenia
“Quality Higher Education: Here, There, Anywhere?”
30/11/07-01/12/07
1
1. Overview (1)
1. Overview
2. Introduction
3. History of quality assurance
4. QA concepts from quality management science
5. Societal context of quality assurance in higher education
6. Goals of quality assurance in higher education
7. Definition of quality
8. Structural approaches to quality assurance
9. Assessment levels
10. Assessment subjects
2
1. Overview (2)
11. Assessment organisation
12. Challenges
13. Conclusion
3
2. Introduction (1)

J. Huizinga, How Does History Determine the Present? A Discourse
that Was Never Held, Leiden, 1946: “And now I haven’t even touched
upon the most difficult word in our sentence: the concept how? How,
that means: in which manner. How poses the insoluble question
about the quality of things. The quality now, even of the smallest
thing, escapes the means of our logical thinking. Applied to our
theme how immediately inverts the question: how does history
determine presupposes the the more profound question: how is history
itself determined, and this means: by what is history determined, and
the by what immediately changes into a by whom and only this
question finds an answer, but outside the borders of the logical mind:
by God, and this answer reveals immediately that the how is
incomprehensible, because it is God’s business. With all our philosophy
and science we once more end up at the modest warning Dante
paraphrased Aristotle with: State contenti umana gente al quia. Be
satisfied, human race, with mere existence.”
4
2. Introduction (2)



If questions about quality are really insoluble and really defy
logical thinking, than the emphasis on quality and quality
assurance in HE policy poses a very big problem.
If questions about quality unavoidably lead us to God, than the
policy of quality and quality assurance are a specialty of the
philosophy of God or even worse of dogmatic theology.
Although questions about absolute quality are Byzantine, I
would like to treat them as a provocative challenge and say to
the HE community: “Non state mai contenti, gente
dell’accademia, al quia.”
5
3. History of quality assurance (1)


QA originated in world of manufacturing and industrial
production
before Industrial Revolution (<1780): quality =
straightforward problem
-> production centralised in 1 person’s hand who
performs and controls
-> standards for products with external inspection
system (guilds, city governments, customs
officials,…)
e.g. Book of the Eparch, Constantinople, 912
-> QA in the process sense poses no explicit problem
6
3. History of quality assurance (2)


Industrial Revolution (> 1780):
-> collectivisation of production process leads to mass
production whereby no longer 1 individual performs and controls
the production
-> structural introduction of foreman, but mainly for organisatory
and command purposes (military-hierarchical organisation of
labour process)
throughout 19th C.: mass production leads to low standard
products, widely diverging output, a lot of defective products
-> quality becomes an explicit problem linked to the organisation
of the production process
7
3. History of quality assurance (3)


end 19th C: in US 1st forms of explicit QA introduced
as part of larger industrial management reforms:
-> F.W. Taylor (1856-1915): Taylorism: scientific
management combined with quality departments to
oversee quality of production and rectify errors
-> H. Ford (1863-1947): Fordism: standardisation of
design and component standards ensure production
of standard products, combined with quality
departments and inspection of the output
1930s: 1st application of statistical controll as a QA
method
8
3. History of quality assurance (4)




WO II = very important impetus for QA in US and UK: need to
raise both quantity and quality of production for war effort
after WO II: quality and QA becomes both profession,
management process and scientific discipline
clear evolution in QA:
-> simple quality control: 40s-60s
-> quality engineering: 70s
-> quality systems engineering: 90s
no longer limited to manufacturing, but also widely applied in
practically all sectors of service provision (banking, health
care,…)
e.g. today 31% of companies with ISO 9001 certificate are
service providers
9
4. QA concepts from quality management science (1)



in quality management science throughout the last 50
years all kinds of concepts, methods, tools and
strategies conceived
e.g. SPC, Zero Defects, Six Sigma, quality circles,
Total Quality Management (TQM), Theory of
Constraints (TOC), Quality Management Systems
(QMS), continual improvement (Kaizen),…
mostly originated in manufacturing but also widely
applied in service provision -> also in education
3 concepts regularly encountered in QA world in HE:
PDCA-cycle, ISO 9001 standards, TQM
10
4. QA concepts from quality management science (2)
1) PDCA-cycle:
•
•
•
•
Plan-Do-Act-Check-cycle or Shawhart-cycle or Deming-cycle
developed by W.A. Shawhart in 1930s in Bell Telephone
Laboratories
made famous by W.E. Deming, father of modern quality control
4 elements:




PLAN: establish the objectives and processes necessary to deliver
results in concordance with specifications
DO: implement the processes
CHECK: monitor and evaluate the processes and results against
objectives and specifications and report the outcome
ACT: apply actions to the outcome for necessary improvement,
which means reviewing all steps (P-D-C-A) and modifying the
process to improve its next implementation
11
4. QA concepts from quality management science (3)
12
4. QA Concepts from quality management science
(4)
2) ISO 9001 standards and certificate:
• example of Quality Management System (QMS)
• QMS:
 set of policies, processes and procedures required for planning and
execution (production/development/services) in the core business area
of an organisation
 integrates the various internal processes within an organisation and
intends to provide a process approach for project execution
 enables an organisation to identify, measure, control and improve the
various core business processes that will ultimately lead to improved
business performance
• Quality Management System Standards of the International
Organisation of Standardisation (ISO)
• first issued in 1987, as the translation on international level of the
British industrial management standard BS 5750 developed during
WO II
13
4. QA concepts from quality management science (4)
• aims at certifying the processes and the system of an
organisation and not the product or service itself
• originated in manufacturing, but now employed across a
wide range of other types of organisations
• updated every odd year: last version = 2000, next version =
2008
• many HEI’s (especially in the non-university sector) have
received a certificate of compliance with the ISO 9001
standards
14
4. QA concepts from quality management science (5)
• version 2000 meant a radical change by going away from
product inspection and preventive paperwork to the concept
of process management -> examples of standards:
 set of procedures that cover all key processes in the business
 monitoring of processes to ensure that they are effective
 adequate record keeping
 checking of output for defects, with appropriate corrective
action where necessary
 regular review of individual processes and the system itself for
effectiveness
 facilitation of continual improvement
15
4. QA concepts from quality management science (6)
3) Total Quality Management (TQM):
• originates in the work of A. Feigenbaum, Total Quality
Control (1951):
 Total Quality Control defined as: “An effective system for coordinating the quality maintenance and quality improvement
efforts of the various groups in an organisation so as to enable
production at the most economical levels which allow for full
customer satisfaction.”
 quality control goes from a technical management process to a
business method
16
4. QA concepts from quality management science (7)
• TQM = management strategy aimed at embedding
awareness of quality in all organisational processes
• combination of 3 elements:
 total: involvement of the entire organisation, supply chain
and/or product life cylce
 quality: fitness for purpose
 management: system of managing with steps like plan,
organise, control, lead, staff, provision,…
• clear parallel with the quality culture discourse in HE (cf.
EUA-project)
17
5. Societal context of quality assurance in higher
education (1)

massification of HE since the beginning of 70s in WesternEurope and beginning of 90s in Eastern Europe:
• e.g. Flanders: 1958 -> 5000; 2008 -> 170000
• educational concepts did not adjust (e.g. Humboldian university
ideal)
• funding did not raise at same rhythm
• quality of education went down
• QA is instrument to raise quality based on reasoning that funds
must not only be spent but spent efficiently -> within current budget
efficiency gains can be made

change in relationship between government and HEI’s:
• growing legal and practical autonomy for HEI’s: freedom to
organise programs and determine content, to appoint staff, to
spend budget, to determine profile,…
• government demands post factum control via QA system
18
5. Societal context of quality assurance in higher
education (2)

bigger role for private provision of HE:
• private profit and non-profit HE providers(especially in Eastern
Europe)
• transnational education (especially in Eastern Europe)
• QA as an instrument for market entry regulation and consumer
protection

influence of corporate life and economic science:
• development of quality control as an applied economical science
• wide application of quality management strategies in profit and nonprofit service sector

new vision on relation between government and the
institutions/agencies it subsidises
• from inspection to assessment/audit
• from only controll to also consultancy
• from organised distrust to guaranteed trust
19
5. Societal context of quality assurance in higher
education (3)

European Union policy context:
• also in HE is Europe becoming an ever more closer union: art. 149150 EC, EC mobility programmes, Bologna Process, Copenhagen
Process,…
• integration of and exchange between nationale HE education
systems require mobility of students and staff and recognition of
degrees and study periods
• QA is an instrument to support mobility and create the institutional
trust necessary for smoother and quicker recognition

growing international competition between HE systems and
HEI’s -> QA is an instrument to make national HE systems and
its degrees more readable and understandable, and more
trustworthy abroad (quality labels)
20
5. Societal context of quality assurance in higher
education (4)

the fall of the Berlin Wall and the crumbling of the
communist bloque:
• disavowment of (neo)marxist ideology
• Undermining of belief in state planning and collective action
on state leve lto tackle collective action problems
• embracement of a retreating and at arms length government
concept
• reinforcement of the faith in self-regulation
21
6. Goals of quality assurance in higher education (1)

QA can have one or more of the following goals:
•
•
•
•
•
•
•
•
quality improvement
quality control/consumer protection
public accountability
information provision
international comparison/benchmarking
ranking
funding
recognition
22
6. Goals of quality assurance in higher education (2)


the last 2 goals can be seen as indirect goals: the first 6 is what
a QA system is explicitly geared towards, the last is what the
government or a government agency uses the outcome of QA
for
the goal or goals of QA determine the set-up and functioning of
the QA system, e.g.:
• the role of a self-evaluation report in QA for improvement >< QA for
control
• the use of reference points in QA for improvement >< the use of
standards in QA for control
• the publicity status of evaluation reports in QA for improvement ><
QA for information
• QA for ranking requires comparable criteria and standards and
provisions to facilitate transversal analysis
23
7. Definition of quality (1)

traditional description: something is of good quality when it does
what it is meant to do
-> poses the same problems as Aristotle’s well-known definitions
of justice:
• to give each person what is due to him
• to treat like cases alike and unlike cases unalike
BUT what is due to a person and when are cases alike and when
unalike, and above all who determines this?

-> what is something meant to do and who determines this?
2 major definitions of quality in HE possible:
• fitness for purpose
• fitness of purpose
24
7. Definition of quality (2)

fitness for purpose:
• relates to traditional description
• relates to world of economics and quality management
science
• something is effective and efficient: it does what it is meant
to do in the most profitable way possible
• = procedural or relative description
25
7. Definition of quality (3)

fitness of purpose:
• beyond traditional description
• beyond world of economics and quality management science
• asks the questions what is good education, what is good
programme
• raises pedagogical discussions: e.g. what is best: problem-based
learning? collaborative learning? ex cathedra learning?
• raises content discussions: e.g. what does a good legal education
entail? :
 next to technical legal courses also a broad introduction into other fields
of social sciences?
 next to traditional legal methodology also an active introduction into the
methodology of other fields of social sciences, so as to create an
interdisciplinary approach to law (e.g. Law & economics)?
 clinical legal education and/or moot court exercises?
• = absolute or material description
26
7. Definition of quality (4)
• possible to formulate general competences across fields of
study for HE graduates (cf. TUNING project)
• in some fields it is easier to formulate domain-specific
competences than in others (e.g. chemistry >< history)
• in the end the absolute description is also relative:
 peers sit together and determine what fitness of purpose is for
a specific programme at a specific point in time
 years later the fitness of purpose can and maybe must have
changed, taken into consideration scientific evolutions in the
field of study and evolutions in society
 M. Weber: “Ideas and concepts change when old men die.”
27
7. Description of quality (5)

definition of quality used in a QA system is important
for the set-up and methods of the system:
• fitness for purpose: focus on management evaluation
methods and techniques and the instruments and statistics
needed for these
• fitness of purpose: trustworthy development of a list of
standards or reference points necessary

good and balanced QA in HE pays attention to both
fitness for and of purpose
28
8. Structural approaches to quality assurance (1)

4 ways of organising QA:
•
•
•
•
evaluation
accreditation (certification)
benchmarking
audit
29
8. Structural approaches to quality assurance (2)

evaluation:
• traditional approach to QA in HE
• a group of peers with an open ended agenda comes in,
looks at aims, processes and output, judges and formulates
recommendations
• linked to quality improvement function

accreditation:
• an independent instance determines if a programme or
institution fulfils a list of standards that is fixed on beforehand
and the same for all programmes or institutions taken into
consideration
• linked to quality control and accountability functions
30
8. Structural approaches to quality assurance (3)

benchmarking:
• programmes or institutions are compared to each other on strong
and weak points
• this can be done on the basis of reference points fixed on
beforehand
• if the reference points are conceived of as standards this can lead
to ranking of programmes or institutions
• linked to (international) comparison (and ranking) function(s)

audit:
• integrated approach to quality and quality management
• focuses very strongly on internal processes and procedures for
policy steering, problem detection and problem solving
• usually combined with a focus on institution or faculty, not
programme
• linked to quality improvement function
31
8. Structural approaches to quality assurance (4)


every approach has its own specific consequences
for set-up and methodology
this analysis makes clear that the traditional
dichotomy and antinomy between QA and
accreditation fake is:
• QA and accreditation are not opposites, accreditation is one
of the forms of QA differing on the basis of aim and set-up
• when people oppose QA and accreditation, what they really
mean is the difference between evaluation and accreditation
• it has been observed during the past 10 years that
evaluation and accreditation are growing towards each
other:
32
8. Structural approaches to quality assurance (5)
 accreditation is paying more and more attention to quality
improvement and strategies to induce it
e.g. Hungarian accreditation system, big US debate on the
future of accreditation, the conception of the Dutch-Flemish
accreditation-system
 evaluation is making more and more use of reference points to
make judgements more objective, consistent and comparable
e.g. subject benchmarks in the UK QA system, indicator lists in
the Flemish QA system
33
9. Assessment levels (1)

4 possible levels for external QA:
•
•
•
•
programme
logical cluster of programmes/faculty
institution
HE system
34
9. Assessment levels (2)

programme:
• advantage: direct view and influence on quality of education
provision on the basic level
• disadvantage: for many quality issues a programme depends on
wider policy it cannot influence let alone control (e.g. funding, staff,
counseling, facilities, housing,…)
• example par excellence: the so-called Dutch model (1985)

institution:
• advantage: you have a systems approach, you can focus on how
an institution as a whole deals with quality issues, which processes
and procedures are in place, if there is a quality culture, and you
can assess the wider policy that influences the quality of individual
programmes
• disadvantage: no direct check of the real life quality of individual
programmes
• example par excellence: the French system (1984)
35
9. Assessment levels (3)

logical cluster of programmes/faculty:
• usually not considered to be a full-fledged option, but a
compromise alternative to overcome the disadvantages of
QA on programme and on institution level
• quite rare in practice

HE system:
• not really an alternative
• if done, on a meta-level: e.g. how do institutions tackle QA?
• mainly used as policy input for parliament and government
36
9. Assessment levels (4)

end 1990s fierce debate between proponents of
programme QA and proponents of institution QA:
• proponents of programme QA mainly students,
governments, parliaments -> quality improvement and
accountability
• proponents of institution QA mainly institutional leaders and
their umbrella organisations (e.g. EUA) -> stress on
institutional autonomy, trust and responsability
37
9. Assessment levels (5)

in practice the border lines are once more much more blurry:
• systems based on institutional QA are looking for ways to include
the programme level (e.g. France, UK post 2001); systems based
on programme QA are looking for ways to include the institutional
dimension (e.g. Sweden) -> there seems to be growing a
consensus that a good QA system combines programme and
institution evaluation
• evolutionary theory: in a first stage programme QA is
indispensable, after eliminating the rotten apples and through this
processing stimulating the HEI’s to build strong internal QA systems
external QA can shift focus from programme to institution (BUT
probably a rationalising apology for British policy failure)
• combination strategy: in the US system all HEI’s have to be
institutionnaly accredited, but next to the you have a lot of
programme accreditation in those fields of study where society has
an interest to guarantee the quality of individual programmes and
graduates (e.g. doctors, lawyers, nurses, teachers, engineers,…) 38
10. Assessment subjects (1)

QA in HE can assess the following subjects:
•
•
•
•

education
research
facilities (libraries, laboratories, counseling services,…)
Management
QA of education can assess the following subjects:
• input
• process
• output
39
10. Assessment subjects (2)

input:
•
•
•
•
•
•
•
aims and goals of the programme
content of the programme
number of staff
Staff qualifications
number of students
staff/student ratio
facilities (library, laboratories, ICTinfrastructure,…)
• …
40
10. Assessment subjects (3)

process:
•
•
•
•
•
•
•
•
educational concept
teaching method
work forms
forms of evaluation
studeability of the programme
complaints procedure
internal QA system
…
41
10. Assessment subjects (4)

output:
•
•
•
•
•
retention/success rate
level of master thesises
employment rate
time lapse before 1st employment after graduation
wage level of graduates at 1st employment and after 5 years
of experience
• satisfaction level of alumni
• …
42
10. Assessment subjects (5)

traditionally the focus in QA is on input, but this gives
a distorted image: process and output have to taken
into consideration, especially output, because this
indicates the level of attainment of a programme

focus on output in QA links up with the concept of
learning outcomes in curriculum development and
evaluation -> QA procedures should try to assess in
an objective and systematic manner if programme
graduates attain the learning outcomes their
programmes put forward for them
43
11. Assessment organisation

in the Berlin Communiqué (2003) the European ministers
responsible for HE agreed on a brief codification of best practice
for QA organisation: an national QA system must be based on
the following principles:
 main responsibility lies with HEI’s: they develop an internal QA
policy and system
 clear definition of responsabilities of all involved institutions and
organisations
 evaluation of programmes and/or institutions containing a.o.
internal self-assessment, external evaluation, student participation
and publication of results
 system of accreditation, certification or comparable procedures
 international participation, cooperation and networking

further developed in the European Standards and Guidelines for
QA in the Bergen Communiqué (2005)
44
12. Challenges

how to guarantee that QA also improves quality?
 how to avoid a compliance culture?
 how to avoid the stifling of educational experiment
and innovation?
 how to diminish the administrative burden?
 how to keep costs down?
 how to organise international cooperation and
participation?
 how to build a European roof on national systems so
that they can communicate and exchange?
45
13. Conclusion (1)
1)
2)
3)
QA is a necessary good because it guarantees that university
professors deal with education in a manner as professional as
they deal with research
QA is a necessary good because it gives students the lever to
become actively involved in the quality management of their
programmes while at the same time respecting the academic
freedom and content expertise of their teachers
the major challenge for the future is to gear QA systems more
towards quality improvement because only then will they be
worth their money and burden -> a.o. this means widening the
focus of QA: not only numerical checking but also content and
policy debate and recommendations
46
13. Conclusion (2)
A. de Saint-Exupéry, The Little Prince: “Grown
ups like numbers. When you have a new
friend they never ask you: ‘What is the colour
of his eyes? What sounds does he like? What
games does he like to play?’ No, instead they
ask you: ‘How old is he? How big is he? What
grades does he get in school? How much
does his father earn?’ By knowing numbers
they think they know the person. Ah, but you
shouldn’t hold it against them. You must be
very patient with grown-ups.”
47