Transcript 2005-RE- Existing evaluation instruments

Reflective Evaluation
About existing evaluation and selfevaluation instruments and reasons
for inventing something new for CVT
Irina Michel, Gerald Heidegger,
Wiebke Petersen
Reflective Evaluation
Transcultural aspects
 Copenhagen Process:
 No „unification“ of systems in VET
and CVT but subsidiarity principle
(contrast to Bologna process for
higher education)
 Reason: VET and CVT systems too
strongly embedded in cultural and
economic conditions
 Therefore reduction of the aims to the
following four fields:
Reflective Evaluation
 With respect to our project:
 European Dimension
 ---- transcultural mutual learning
 Accreditation of non-formal and informal
learning
 ---- accreditation of CVT which includes non-formal
learning
 Quality (of processes and outcomes)
 ---- our new tool aims at self reliant quality
assurance
 Transparency
 ---- the criteria for the self-evaluation should be
transculturally transparent
Reflective Evaluation
 Transcultural comparison:
Four main types of
- Welfare system
- VET and CVT systems
- Re-Integration schemes
See overhead transparency
Aspects of the history of quality
management and its tranfer to the
educational sector
-
The idea of quality management stems from
production industry (ISO 9000)
-
During the last decade the idea of quality
management has been broadly transfered to nonprofit organisations (EFQM – European Foundation for
Quality Management)
-
The first non-profit sector that has taken over quality
management was the care sector – soon stressing
self-evaluation (because a lack of „hard“ outcomes)
-
Today a lot of public schools go through evaluation
processes.
-
For disadvantaged young people: Our method QSED
(Quality through Self-Evaluation and Development)
Output
indicators or
personal
development
common
understanding of
the phenomenon
that should be
evaluated
Contexts of
evaluation
historical and
cultural
embeddedness of
the phenomenon
of evaluation
From ISO
9000 via
EFQM to
QSED
national annd
international
evaluation
discussions
Figure 4.1: Contexts of evaluation
different
evaluation
interests, fears
and, targets
Carried out by
OECD, CEDEFOP
Indicator centered
evaluation
strict, highly
structured
procedures
Carried out by
IDEA, IOCE
Evaluation trends
Measuring soft
outcomes
innovative
evaluation
approaches
targeting
empowerment or
participation
Theoretical
interests in
evaluation
and metaevaluation
Context
of
evaluatio
n
relevant
aspects of
evaluation
Expertise
of
evaluatio
n
Purpose
Evaluatio
n
practices
and
methods
and
meaning of
evaluation
Target/
Object of
evaluation
External evaluation
 Advantages
 Seemingly „Objective „ ,
Reliable Outcomes
 Easily transferable
results
 No self-deception
 No cheating(???)
 Should be combined
with (internal) selfevaluation
 In order to….
 Disadvantages
 To measure „soft“
outcomes is very
difficult/impossible
 Processes are aiming at
the targets of the
external evaluation
 In this way processes are
severely narrowed down
 Practitioners are objects
of external powers
 They become
disempowered
 They become passive
 avoid these
Existing European evaluation approaches:
example EFQM
leadership
(1)
politic and
strategy
(2)
company results
(9)
criteria of EFQM
staff orientation
(3)
societal
responsibility
and image (8)
staff
contentedness
(7)
customer
contentedness
/satisfaction (6)
resources
(4)
processes
(5)
Figure 6.1: Criteria of EFQM
European standard: the EFQM
Example for the criteria
 Processes

This criterion is defined as follows: How does the
institution recognise, identify, carry through tests
and improve its processes?

It is divided into five subsections which deal with the
following questions and for which several starting points are
given:
5a ”How are processes identified which are significant for
the company results?”
Starting points could be, how the institution ...
…defines core and support processes (1)
…identifies core and support processes(2)
…values repercussion on the business results(3)





European standard: the EFQM
 Using the EFQM as an evaluation tool


In the first step it takes place through self-evaluation
The second step includes the coming in of external
evaluators
 Underlying assumptions and targets of
evaluation according to EFQM

“Is the client content with the product which is
given?”.- Is it the “Customer”, that is the Labour
office???
 Advantages of the EFQM in comparison
with other evaluation tools:
 it tries to take care also of soft outcomes
Adaptation of the EFQM model to a re-integration institution for
disadvantaged youngsters
customer
orientation
client
orientation
Self
directing
JAW-stamp
of quality
continuous
improveme
nt
Figure 6.2: JAW-stamp of quality
Benchmark
ing
A Self-Evaluation instrument for
schools: A different idea of quality
 Basic ideas of Q2E : Quality though Evaluation and
Development („Entwicklung“) with a focus on selfevaluation
awareness of the
own quality
promise
optimizing
through error
recognition
aspects of the
new quality
understanding
Figure 6.3: Aspects of the new
quality understanding
client orientation
Example of a criterion in Q2E: “Shaping of teaching and
learning processes (methodical-didactical
arrangements)”
-
to be ticked from 1(weak) to 4 (strong) -
1)The teacher attaches importance to the goal that
aims and intentions of the lessons are understood
by the pupils. The learners see the importance of
learning aims and contents.
2)The teacher arrives at explaining complex learning
processes and difficult facts in relation to
experiences and knowledge of learners.
3)The teacher arrives at initiating the pupils´ interest
for the contents and engaging them for
participation.
Example of a criterion in Q2E: “Shaping of teaching and
learning processes(methodical-didactical
arrangements)”
-
to be ticked from 1(weak) to 4 (strong) -)
4)Space of active participation and self-directed learning
of the pupils in the lessons is offered
5)The teacher arrives at making the pupils aware of their
responsibility for their learning and he supports this
through adequate measures.
6)The teaching is shaped in a way that pupils could
create a strong relation between theory and their own
experiences.
basic instrument for
the development of a
quality model
…
...basic instrument for
the development of a
school specific
evaluation instrument
as basis for a quality
comparison between
schools
…
Q2E can be
used as ...
…
…as basis for the
development of a
quality handbook
help for the development
of a school specific profile
of strengths and
weaknesses
…
Figure 6.4: Options for using Q2E
basic instrument for
focus evaluation
Our new IT-Instrument:
QSED
 Centred on self-evaluation of
practitioners
 To be adapted to the specific
circumstances
 Culturally adaptable
 QSED („Quality through Self-Evaluation
and Development“) for support schemes
for disadvantaged young people
should now be adapted to
CVT
Reflective Evaluation
Target group of the instrument:
Trainers and practitioners for
 People at risk in the labour market:
 - Unemployed, often long term
 - People under threat of being fired
 - Women returners
 - But also: People in conventional
CVT courses
Some features of the QSED
(„old tool“)
Six dimensions
Three levels
Highly(!!!) interactive IT-Tool
Adaptation through the users
themselves
 The users can completely change the
instrument apart from the basic
structure (dimensions and levels)
 Next: the dimensions




Recognition of
Self-Evaluation
skills/Assessment
and Reflection
Collaborative
Situated
QSED
Learning
Funding/
Administrative
structures
networks
of actors
Inclusiveness
to the end
Transformation of QSED for the
new target group
 Some dimensions: the same,
although with new aspects and new
criteria
 Some dimensions: new, for the
specific needs of the target group
(both participants and practitioners)
 Next page: tentative outline of the
dimensions for the new tool:
„Developing quality in
CVT“
Validation and
accreditation
of competencies
New!
Didactical/
methodological
Concepts
New!
Funding/
Administrative
structures
Self-Evaluation
and Reflection
“DQCVT”
Collaborative
networks
of actors
New!
Life situation/
Experiences
New!
to the end
Transformation of QSED for the
new target group
 Starting from QSED:
 Three levels („Multi-level approach“):
Explanation: next page
 Macro: „Structural“
 Meso: „Institutional“
 Micro: „Individual“
 Also some other explanations
Structural-level:
on this level you find statements which point out power relations in
society.
Institutional level:
On this level you find questions which deal with the activities and the
processes within Reintegration programmes.
Individual level:
On this level you find questions which deal with the actions and
experiences of practitioners and learners.
Explanation:
Here you find further information regarding the statements and
questions.
Suggestions:
Here you can write in suggestions for influencing the societal,
economic and political conditions.
Answer and Change:
Here you can answer or change questions in order to design your
own tool.
Back to Index Using this button you get
Back
back to the cluster which
shows the dimensions.
Back to
content
Using this button you get
back to the content of the
dimension.
Developing Quality in CVT
 Now let´s try out and look at the
„old“ QSED in order to get some ideas
about the
 New
DQCVT