Transcript Presentation - Training Evaluation

Support the spread of “good practice” in generating, managing,
analysing and communicating spatial information
Training Evaluation
By: Rainer Zachmann
Unit: M03U05
Content
•
•
•
•
•
Introduction
Relevance of evaluation
Monitoring course climate
Pre- and post-tests
Partial evaluation
Content
•
•
•
•
Continuous activity evaluation
Evaluation of course coordination
Evaluation of course materials
Final course evaluation
Introduction
• Evaluation is an essential part of the
training curriculum.
• Evaluation helps in many aspects of
training.
• Evaluation shows whether objectives have
been accomplished.
Introduction
• Evaluation tools may range from objective
tests to subjective assessments.
• None of the evaluation procedures
described are intended to grade trainees
or resource people.
Support the spread of “good practice” in generating, managing,
analysing and communicating spatial information
Relevance of Evaluation
Relevance of evaluation
• Kirkpatrick’s levels of training evaluation:
– Level 1 = reaction
– Level 2 = learning
– Level 3 = behaviour
– Level 4 = results
Relevance of evaluation
• Level 1 (= reaction) measures participants’
satisfaction
– lowest level of measurement, nonetheless
important
– includes aspects before and during the course
Relevance of evaluation
• Level 2 (= learning) measures:
– acquisition of new knowledge;
– improvement of skills;
– change in attitude.
Relevance of evaluation
• Level 3 (= behaviour) follows the
application after training.
• Level 4 (= results) assesses impact at
trainees' organisations.
Relevance of evaluation
• This Unit addresses Levels 1 and 2.
• Levels 3 and 4 are follow-up activities.
Relevance of evaluation
• Evaluation is an essential and integral part
of the training curriculum, not just the end
of training.
Relevance of evaluation
Evaluation helps to:
• determine training needs;
• specify training objectives;
• adjust training methods and materials;
• delete unnecessary content.
Relevance of evaluation
Evaluation helps to:
• execute training;
• verify accomplishments;
• follow-up with trainees;
• reduce training costs.
Relevance of evaluation
• Evaluating whether objectives have been
accomplished is as important as specifying
training objectives.
Relevance of evaluation
Two types of evaluation:
• formative evaluation to monitor training
• summative evaluation to measure
accomplishments
Relevance of evaluation
• Teachers and organisers often evaluate
informally and unconsciously.
Relevance of evaluation
• Objective tests are based on specific
questions to be answered.
• Response tests give freedom for
subjective expression.
• Practical skills testing is important for
PGIS.
Relevance of evaluation
Formal evaluations include:
• monitoring course climate;
• pre- and post-tests;
• partial tests;
• continued activity evaluation;
• evaluation of course coordination and
course materials;
• final course evaluation.
Relevance of evaluation
• Evaluations are not intended to grade
people, but to monitor course progress.
Support the spread of “good practice” in generating, managing,
analysing and communicating spatial information
Monitoring Course Climate
Monitoring course climate
Many methods exist:
• “Democracy Wall”
• “Evaluation Dartboard” is useful for
monitoring participants’ feelings
Multilingual democracy wall deployed at the 2007 Web2forDev Conference in Rome, Italy
Photo: Anja Barth, CTA (From Unit M10U01)
Taylor, B. and Beniest, J. 2003.
Training in agroforestry
- A toolkit for trainers
Page 2.176; ISBN 92 9059 151 x
The World Agroforestry Centre
United Nations Avenue
PO Box 30677, GPO 00100
Nairobi, Kenya
www.worldagroforestrycentre.org
Support the spread of “good practice” in generating, managing,
analysing and communicating spatial information
Pre- and Post-Tests
Pre- and post-tests
• The pre-test assesses trainees' initial
knowledge.
• Multiple-choice questionnaires may be
used.
• Multiple-choice questions are difficult to
design, but analysis is easy and objective.
Pre- and post-tests
• Multiple-choice questions consist of a
"stem" and several options for answers.
• The stem should be a concise statement
that leads directly into the options.
Pre- and post-tests
• Include in the stem as much information
as possible about the item.
• Formulate the stem concisely.
• Each option should be a grammatically
correct completion of the stem.
Pre- and post-tests
• Multiple-choice questions consist of a
stem and:
– a definitely wrong distracter;
– one correct answer;
– (an)other definitely wrong distracter(s).
Pre- and post-tests
• Example: Multiple-choice questions are:
a)
b)
c)
d)
easy to develop
easy to answer
relatively objective
not widely applicable
distracter
distracter
answer
distracter
Pre- and post-tests
• Only one option should be the correct
answer.
• Correct answers must not be self-evident.
• All other options, i.e. distracters, should be
plausible but definitely wrong.
• The length of options should not provide
a clue to the answer.
Pre- and post-tests
• Coordinators should ask only relevant
questions.
• Avoid negatively formulated questions.
• Arrange the answers in random order.
• Coordinators should not use multiplechoice questions when other evaluations
are more appropriate.
Pre- and post-tests
• Coordinators should present the pre-test
at the beginning of the course.
• Ask trainees to be honest and not to
guess.
• Analyse the test immediately.
• Discuss results with trainees and resource
people.
• Coordinators should not give the
questionnaires back.
Pre- and post-tests
• Use the same questionnaire for the posttest.
• Present the post-test one day before the
course ends.
• Analyse the test overnight.
• Compare pre- and post-test results and
present them to trainees and resource
people.
Support the spread of “good practice” in generating, managing,
analysing and communicating spatial information
Partial Tests
Partial tests
• Partial tests help ensure that resource
people cover subject matter appropriately.
• Coordinators should not use partial tests
to grade trainees.
Partial tests
• Coordinators should use the first 15 to 20
minutes of each day for the partial tests.
• Ask four or five questions from topics of
the previous day.
• Ask open essay questions.
Partial tests
• Coordinators should design questions and
expected answers (both!) in collaboration
with resource people.
• The partial tests should only cover
relevant course content.
Partial tests
• Allow trainees to use open books, training
documents, etc. to place them in a real life
situation where they may consult
references.
Partial tests
• Coordinators should analyse answers
immediately with instructors.
• Return tests to trainees and discuss
results.
Support the spread of “good practice” in generating, managing,
analysing and communicating spatial information
Continuous Activity Evaluation
Continuous activity evaluation
• Trainees evaluate the course.
• At the beginning of the course, explain
how to use the evaluation.
• Ask trainees to be critical.
Continuous activity evaluation
• Every morning, coordinators should
distribute new evaluation forms.
• Ask trainees to evaluate immediately.
• At the end of each activity, write the title of
the activity on a board to remind trainees
to evaluate.
Continuous activity evaluation
• At the end of the day, coordinators should
collect the evaluation forms.
• Summarise results on the summary forms.
Continuous activity evaluation
• Coordinators should include trainees in
processing the evaluations.
• Ask trainees to draw conclusions and
report them back to the class the next
morning.
Continuous activity evaluation
• Coordinators should give the summaries
to the resource people and discuss the
results.
• Keep copies in the files for future planning.
Support the spread of “good practice” in generating, managing,
analysing and communicating spatial information
Evaluation of Course
Coordination
and Course Materials
Evaluation of course coordination
• Resource people evaluate the support
given by course coordinators.
• Coordinators should adapt the sample to
specific needs.
• Ask resource people to return the
evaluation after the course.
Evaluation of course materials
• Resource people evaluate materials
included in the PGIS Training Kit.
• Whenever you use some of the PGIS
materials (e.g. Unit Trainer Notes,
Exercises, Handouts, PowerPoint
Presentations, etc.), return your evaluation
to the course coordinator.
Support the spread of “good practice” in generating, managing,
analysing and communicating spatial information
Final Course Evaluation
Final course evaluation
• Ask trainees to assess the course’s overall
acceptability, strengths and weaknesses
and to provide suggestions.
• Coordinators should adapt the sample to
their needs.
Final course evaluation
• The first question - general opinion on the
course - is arranged so that ideal answers
result in a diagonal pattern from top left to
bottom right.
First question
Final course evaluation
• An important aspect is the balance
between theory and practice.
• Trainees usually criticise having too much
theory and not enough practice.
Question 7
Final course evaluation
• Questions like “What topics should have
been extended?” are not conclusive:
trainees wish to extend almost everything.
• A better question would be “What topics
should have been reduced or omitted?”
Final course evaluation
• Coordinators should not ask “What topics
have been most useful?”: trainees find
almost everything most useful.
• Instead, ask “What topics have been least
useful?”
• Coordinators may ask “What new topics
should be included in the future?”
Final course evaluation
• Coordinators should give questionnaires to
trainees several days before the course
ends.
• Collect questionnaires the day before the
course ends.
• Extract the most relevant conclusions
overnight.
• Discuss the conclusions with trainees the
next morning.
Final course evaluation
• After the course, coordinators should
summarise the questionnaires.
• Include the summary in the final course
report.
• Refer to the results when planning the
next course.
Last question