Transcript Slide 1

The challenge of assessing students’
professional development and
achievement
Mantz Yorke
[email protected]
SCEPTrE, University of Surrey
17 March 2011
Text for the day: 1
Reasons for the frequent failure of summative assessment
by teachers to live up to the expectations for it are most
often given in terms of their preparation, or lack of it, for
this part of their work. Many teachers have a narrow view
of assessment and do not know how to respond to freedom
to use evidence from students’ actions, projects and
processes. Merely being required to follow given criteria or
guidelines is not enough.
Harlen (2005, p.249)
Text for the day: 2
... there is a need to assess work-based learning through
methods that are adequate, valid, and avoid undermining
the nature of the learning, given that it will typically be
issue-based, driven by the learner, and transdisciplinary.
The aim of assessment is generally to assess learners’
progress as ‘map-makers’ or self-managing practitioners
... not to confirm their conformance as ‘mapreaders’.
Lester and Costley (2010, p.566)
1
Contexts
Proportion of student’s scheduled time
HE curricula and the development of professionalism
Programme
designed
around
workplace
engagement
Total
Work
placement
as separate
curricular
component
Some
None
Employment
divorced
from HE
curricula
None
Work
placement
as integrated
curricular
component
Extracurricular
award; selfselected
placement
Partial
Level of curricular integration
Complete
2
The challenges of assessing
professional achievements
The nature of the challenge
The development of professionalism is often situated
and contextualised in ways that differ considerably from
the development of learning in academic disciplines
It is the ‘situatedness’ of professional behaviour that
constitutes such a challenge for summative assessment
What should be assessed?
Work-based methods of assessment target the practice-based
elements of higher education study and collect information
that is not so much concerned with performance in their
normal practice[*] but is concerned with being able to think at
a higher level about practice and understand the context well
enough to be able to offer interventions into practices that
have been researched, analysed, critiqued and evaluated. [...]
[The assumption is] that assessments based on a practice that
has subsequently been theorized by the practitioner are a
better reflection of higher level learning than assessments
of routine performance.
Costley and Armsby (2007, p.29)
* Save in some areas of professional practice, as the authors note.
3
Theory and practice
The interplay between practice and theory
Value system
Practice
Theory
Kolb’s learning cycle (modified)
Concrete
experience
Active
experimentation
Reflective
observation
Abstract
conceptualisation
Note: Personal qualities and attributes are almost entirely obscured
A variation on Bruner’s ‘spiral curriculum’
Trajectory of learning
CE
AE
RO
AC
4
Summative assessment
What kinds of problem are the focus?
Problems set in higher education Problems faced by professionals
are quite often characterised by
are quite often characterised by





being deliberately formulated
being well defined
having a ‘right answer’ …
… and a preferred way of reaching it
the availability of most, if not all, of
the relevant information
 being of limited intrinsic interest
 their detachment from ordinary
experience
 adequate time
 being posed to the individual
 ‘happenstance’
 ‘messiness’
 multidisciplinarity
 incompleteness of information
 the pragmatic need to satisfice – i.e.
to reach a ‘good enough’ solution
within the time and resources
available, rather than a perfect
solution
 requiring the involvement of more
than the individual
‘Generic’
capabilities;
WIL/WBL/WRL
Academic
disciplines
Traditional
assessment
methods
Standards
broadly
understood
What are appropriate assessment methods?
How should the issue of standards be dealt with?
Note: In some subjects (e.g. Nursing and Teacher Education)
the relationship is closer than a grid like this suggests.
Educational objectives & their assessment
Type of objective
Problem
Solution
Assessment
Instructional
Specified
Specified
Prescriptive
Problem-solving
Specified/Open Open
‘Expressive’
Open
Open
Social
Open
Open
Responsive
NB: Some alleged problem-solving is essentially puzzle-solving,
where there is a ‘right answer’. This fits best in Row 1.
Assessing professional behaviour ...
... involves evidencing:
• what knowledge, understanding, values and capabilities
the person can demonstrate when engaged in their
professional work
• what s/he achieves in work situations
• how s/he goes about her/his work
Highest or normal level of performance?
Assessment methods for professional behaviour
Before engaging
• Behaviour in, and reflections on, simulations
• Analyses of case studies
• Assignments and examinations of various kinds
During
• Direct observation of individuals (which may involve checklists)
• Direct observation of group activities
After
• Reflective analyses or commentaries of various kinds (e.g. logs or essays)
• Reports on the professional activity (e.g. work placement)
• Portfolios of experience (consider ethics and assessor time commitment)
• An analytical digest of the portfolio content might be more practicable.
• Presentations relating to the professional activity (various modes)
• Interviews
• Evaluation of artefacts produced during the professional activity
Who is doing the assessing?
• Educational provider
• Employer
• Student
Do they have sufficient expertise?
Not in all cases
Are there any sociological pressures?
Conflict of interest
Reluctance to fail
Some awkward questions
How many academics can truly claim to have ‘recent and
relevant’ workplace experience when they come to assess
professional performance?
Whereas academics possess disciplinary expertise, do they have
the capability to assess the ‘soft skills’ valued by workplaces?
Do academics see enough of an assessee’s performance to
reach a valid judgement?
Do workplace assessors have sufficient expertise as regards
assessment?
How should assessments from the workplace and academic
aspects of programmes be combined?
5(a)
Some philosophical considerations:
measurement v judgement
Assessing professional performance
Assessing professional capability and performance
requires post hoc judgement of standards reached against
general expected learning outcomes
It requires the (summative) assessor to have sufficient
expertise to make the judgement
Notes
Totting up marks for different aspects of performance is inadequate and
inappropriate
Being successful in some aspects of professional performance is
mandatory (e.g. safety; classroom management), whereas a more
developmental approach may be adopted in respect of other aspects of
performance
Some achievements are measurable…
… with tolerable reliability, and some not (unless cost
is not an issue)
Some intended learning outcomes (Knight’s ‘wicked
competences’) are inherently fuzzy (even if apparently
specified in tight terms), and resist measurement
Areas of difficulty include
• the exercise of ‘soft skills’
• solving novel, or unbounded, problems
Judgement is difficult to fit into grade-based systems
(‘mapping’ or profiling is preferable to averaging)
Gold at the end of the rainbow?
Introducing standardised and understood methods of
assessing and grading these [generic] attributes, at the
level of difficulty appropriate to the stage of the learning
process, ensures that students better understand why they
must learn particular things and also provides meaningful
evidence to use as part of their future career activities.
Australian Universities Quality Agency (2009, p.12)
A strong echo here of the Australian ‘Mayer Report’ (1992)
Gold at the end of the rainbow?
No.
… the more general and generic standards descriptions
become, the less useful they are for achieving the goals
of ‘greater confidence’ and a ‘clearer picture’ of the
meaning of final achievement grades.
Krause (2009, p.2)
The optimisation issue
Utility
Tight .................................................................. Loose
Specified learning outcomes
5(b)
Some philosophical considerations:
realism v relativism
Realist v Relativist approaches (‘Ideal types’)
(In practice, it is never as cut and dried as this)
Realist
Relativist
Standards are objectively defined
Standards are normative and consensual
Performances can be measured against
these standards
The assessor is objective and detached
Performances are assessed with reference
to these standards
The assessor interprets the extent to which
the performance relates to the standards
Values play no part in the assessment
Value positions are embedded in the
norming of standards
The context of the assessment is taken
into account
There are broad statements of
expectations
The situation of the student is not taken
into account
Explicit criteria and rubrics are invoked
Measurements are taken as true and
reliable representations of achievement
Assessments are judgements of the extent
to which achievements match expectations
Tasks are set by assessors
Tasks may be selected by students to suit
their strengths and interests
Assessment: two contrasting models
Measurement model
Judgement model
• Validity
• Credibility
• Reliability
• Dependability
• Objectivity
• Confirmability
• Generalisability
• Transferability
• Low ‘reactivity’
• ? ‘Reactivity’
• Fairness
• Fairness
• Efficiency
• Efficiency
• ‘Cheat-proofness’
• ‘Cheat-proofness’
• Utility
• Utility
• Intelligibility
• Intelligibility
6
Shifting the paradigm?
Assessment: a ‘paradigm shift’? (after Kuhn, 1962)
1. An academic community operates according to received beliefs.
2. Normal assessment is based on such received beliefs.
3. Normal assessment attempts to force student achievements
into the currently-used conceptual boxes.
4. Normal assessment often suppresses novel approaches
because they tend to subvert its basic tenets.
5. A shift (a ‘scientific revolution’) in professional commitments
to shared assumptions takes place when an anomaly subverts
the existing tradition of assessment practice.
6. New assumptions (paradigms) require the reconstruction of
prior assumptions. This is difficult and time consuming.
It is also strongly resisted by the established community.
Assessment: how the paradigm might shift?
(Again, after Kuhn, 1962)
1. Initially, a new potential paradigm may have few supporters.
2. Proponents should work to improve it; explore it; and show
how it can be effective in practice.
3. The potential paradigm has to convince, and to gather support.
4. Activities based on the paradigm multiply.
5. Eventually, the new paradigm supplants the old.
Assessment: why the paradigm might not shift
Any or all of the following
might happen:
1. ‘It ain’t that broke;
let’s stick with it.’
Less
stable
Energy
barrier
2. ‘It’s too difficult to
change.’
Now
3. ‘It’s impractical.’
4. The messenger is
discredited
or, metaphorically,
shot.
More
stable
Possible future
Time
7
Some challenges
Some challenges for assessment practice
HEI’s involvement: standards, criteria and grading methodology
• The sub-heading understates the inherent complexity!
• Dealing (inter alia) with achievement within, and by, groups
• Staff development in respect of assessment
Employer involvement in assessment of professional achievement
• Training
• Commitment to full engagement in the assessment process
• Potential role conflict of mentor and assessor
• Quality assurance re assessment of professional achievement
These challenges have no easy answers