Slide 1

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 2

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 3

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 4

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 5

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 6

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 7

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 8

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 9

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 10

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 11

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 12

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 13

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 14

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 15

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 16

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 17

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 18

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 19

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 20

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 21

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 22

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 23

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 24

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 25

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 26

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 27

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 28

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 29

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 30

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 31

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 32

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 33

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 34

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 35

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 36

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 37

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 38

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 39

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 40

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 41

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 42

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 43

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 44

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 45

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 46

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 47

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care


Slide 48

Can We Measure & Track the
Acquisition of Competency?
Evaluating learning in relation to the NOS
competencies using mixed methods

Dr Sharon Vitali
Evaluating the Outcomes of Social Work
Education Symposium
15 May 2009

School of Health and Social Care

What do we want to know?
Is this student academically eligible for a BA
(Hons) social work degree?
Is this student competent to practice as a novice
social worker?
Are our assessment (academic & practice)
mechanisms reliable?

School of Health and Social Care

What do we want to know?
Where do our students acquire the knowledge
skills and values that are required of them to
qualify?

Is our curriculum fit for purpose?
What are the weaknesses in our curriculum ?
(content and or timing of delivery)

School of Health and Social Care

Competency Based Education &
Training
 From the Latin word ‘competens’ to “…be fit, proper
or qualified”
(The Oxford Library of Words and Phrases,1998)
 a competent learner is "…one who has reached a
defined standard of performance" (Freeman & Lewis 1998:314)
 outcomes = workplace expectations of specific
knowledge and skills, which are believed to be
essential and the required level of performance
 what a person can do; rather than length of type of
training (allows APEL)
School of Health and Social Care

Competency Based Education &
Training
 refers to skills and knowledge that can be transferred
and applied to new situations and environments
 results in a recognisable, consistent and portable
qualification
(Toohey, Ryan, McLean & Hughes, 1995)
 market and government driven - attempting to ensure
ability and accountability thereby enabling efficient
and global recruiting of crucial segments of the
workforce

School of Health and Social Care

Competency Based Education &
Training
 an amalgam of knowledge, skills and values that
must be demonstrated and measured against a
benchmark of standards that have been agreed
upon.
 No penalty for barely reaching adequacy, neither is
there any reward in excelling.
 Licensed to legally perform a specific activity in
society.
 License holder subject to disciplinary measures upon
infractions, and the license may be revoked should
the misdemeanour be significant enough
School of Health and Social Care

Challenges to Evaluating Outcomes
of Educational Programmes:
 Potential for bias in ‘satisfaction outcomes’
 Potential for bias in ‘ipsative’ (self appraisal)
evaluations
 Competency Based Education vs. Competency
Based Assessment

 Methodology Strategy:
= Mixed Methods & Triangulation of Measures

School of Health and Social Care

What did we do?
 Cohort Sample
(n=38 at beginning n=33 at completion = 87%)

 Longitudinal - Baseline to Graduation
 Repeated Measures
 Triangulation of Measurements
 Mapping Against Learning Activities

School of Health and Social Care

What measurement methods?
 Fixed Outcomes
(National Occupational Standards of Competency for SW)

 RICET
(Reflective Ipsative/self Competency Evaluation Tool)

 Pre-placement interview performance


(Voices of Experience & Independent Researcher)

 Portfolios inclusive of written reflective performance
 Academic Performance
 social work & non-social work modules
 coursework, computer based tests, presentations
School of Health and Social Care

Evaluation Sources & Schedule:
Y1-S1

RICET
Self

LEARNING PLAN
Tutor

RICET
Self

Y1-S2

INTERVIEW
Voices of
Experience

RICET
Self

PORTFOLIO
Tutor

Y2-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y2-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

Grades SW
& Other

Y3-S1

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CASE
VIVA

Grades SW
& Other

Y3-S2

RICET
Self

RICET &
PORTFOLIO
P/Teacher

PORTFOLIO
P/Teacher
& Tutor

CRITICAL
CAREER
REVIEW

Grades SW
& Other

School of Health and Social Care

Grades
SW & Other
INTERVIEW
Researcher

Grades SW
& Other

Curriculum of Learning Activities
Y1-S1

Fit for Practice One

Human Growth &
Development One

Professional
Communication

Partnerships in
Practice One

Y1-S2

Fit for Practice Two
& Shadowing
a Social Worker

Human Growth &
Development Two

Social Policy

Social Work Law

Y2-S1

Placement in Voluntary
Independent or Private
Sector

Critical Evaluation of
Research Literature

SW with Children +
SW with Adults
+
Mental Health for
Social Workers

Partnerships in
Practice Two

Y2-S2

Placement in Voluntary
Independent or Private
Sector

Placement in
Voluntary
Independent or
Private Sector

SW with Adults

Mental Health for
Social Workers

Y3-S1

Placement in Statutory
Sector

Placement in
Statutory
Sector

Practice Learning
Interventions

Dissertation

Y3-S2

Placement in Statutory
Sector

Alternative
(LA Children or...)

Professional
Consolidation

Dissertation

School of Health and Social Care

What else did/could we collect ?
 Demographic features of Students
Age, gender, country of origin, first language,
previous education, previous work / volunteer
experience, mode of entry, dyslexia/disability,
childcare responsibilities, distance travelled to attend
 Practice Setting
Type, experience level, student audit (PLEAT)
 Demographics of Practice Teachers
Age, gender, country of origin, first language,
previous education, previous work experience,
practice teaching experience

School of Health and Social Care

Caution about Findings
 Previous study had larger sample and found
statistically significant changes
BUT…
 This sample too small to run any tests of
significance…so just descriptive findings
 TODAY …some trends and demonstrate to you how
the methodology works to answer your questions

School of Health and Social Care

What did we find?
 Inflation of Ipsative Measurement:
Inflation at ipsative baseline measure = “don’t
know what they don’t know”
Correction of inflated measure except in one
subset = cultural artefact

Students over-rated themselves compared to
tutors & practice teachers throughout study

School of Health and Social Care

School of Health and Social Care

What did we find?
 Performance by Mode of Entry :
 Widening participation
 Lowering of age at entry requirement
 “Added value”
Slightly higher average graduation grades in
traditional mode of entry students
Grade range very similar
School of Health and Social Care

Average Graduation Grade by Mode of Entry
66

65.06

65
64
63
62

61.17

61
60
59
Non-Traditional (n=13)
51.8% to 75.5%

School of Health and Social Care

Traditional (n=17)
53.3% to 77%

What did we find?
 Performance Trajectories
 Academically Strong
 ‘Natural’ in Practice
 Confidence

 3 trajectories (L to L) (H to H) (L to M)

School of Health and Social Care

School of Health and Social Care

What did we find?
 Measurement of acquisition of competencies possible
 Mapping of acquisition of competencies possible
 Identification of gaps & weaknesses possible
OBSERVATIONS:
 Incremental over the 3 years
 Practice teachers and esp. tutors tended to score
lower than students’ ipsative scoring
 Regression in measurement when shifting into
second year (statutory) placement
School of Health and Social Care

School of Health and Social Care

NOS 1: Prepare for social work involvement
6

5

Mean Score

4

F4P Test of
Knowledge

F4P Test of
Knowledge
& Skill

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 2: Work with individuals, families, groups and communities to help them make
informed decisions
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 3: Assess needs & options to recommend a course of action.
5
4.5
4
ENHANCED:
Assessments

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 4: Respond in a crisis situation
5
4.5
4

3

ENHANCED:

2.5

Crisis
Intervention

2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 5: Interact with individuals, families carers, groups & communities to achieve
change & development & to improve life opportunties.
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 6: Prepare, produce, implement & evaluate plans with individuals, families, carers,
groups, communities & professional collegues.
5
4.5
4

3

F4P2
Theory to
Practice

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 7: Support the development of networks to meet the assessed needs & planned
outcomes.
5
4.5
4

3

F4P2 Theory to
Practice, Social
Policy & Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 8: Work with groups to promote individual growth & development
5
4.5
4
F4P2 Theory to
Practice
HG&D 2

3

Group Work

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 9: Address behaviour which represents a risk to individuals, families, carers,
groups & communities.
5
4.5
MOVED:

4

Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 10: Advocate with, and on behalf of individuals, families, carers, groups &
communities.
5
4.5
4

Social Policy &
Law

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 11: Prepare for, & participate in decision-making forums.
INCREASED:

5

Individual
Presentations &
Posters

4.5
4

3

Social Policy &
Law

2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 12: Assess & manage risks to individuals, families, carers, groups & communities
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 13: Assess, minimise and manage risk to self & others
5
4.5
4

MOVED:
Mental Health
for Social
Workers

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 14: Manage & be accountable for your work
6

ENHANCED IN:
Practice
Learning
Interventions

5

Mean Score

4

3

2

1

0
PRE

S_ES1 T_ES1 S_ES2

S_MP1 PT_MP1 S_EP1 PT_EP1
Measurement

School of Health and Social Care

S_MP2 PT_MP2 S_EP2 PT_EP2

NOS 15: Contribute to the management of resources & services
ENHANCED IN:

5

Practice
Learning
Interventions

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 16: Manage, present & share records & reports
5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 17: Work within multi-disciplinary & multi-organisational teams, networks &
systems.
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 18: Research, analyse, evaluate & use current knowledge of best social work
practice
ENHANCED IN:

5

Practice
Learning
Interventions

Critical
Evaluation of
Research
Literature

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 19: Work within agreed standards of social work practice & ensure own personal
development
Professional
Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 20: Manage compex ethical issues, dilemmas & conflicts
5

Professional
Consolidation

4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

NOS 21: Contribute to the promotion of best social work practice
ENHANCED IN: Practice
Learning Interventions &
Professional Consolidation

5
4.5
4

3
2.5
2
1.5
1
0.5

Measurement

School of Health and Social Care

PT
_E
P2

S_
EP
2

PT
_M
P2

S_
M
P2

PT
_E
P1

S_
EP
1

PT
_M
P1

S_
M
P1

S_
ES
2

S1
T_
E

S_
ES
1

0
PR
E

Mean Score

3.5

What have we done?
 Analysis of non-completers:
 This cohort = 8
 Delayed (Temporary Withdrawal) =2
 Unsuitable = 2
 Academically Incapable = 1
 Analysis of Added Value & Exit Velocity Trajectories

School of Health and Social Care

What have we done?
 Redesigned curriculum and timing of module delivery
 Redesigned module content & assessment methods

 Enhanced support for ESL & non-traditional mode of
entry students
 Enhanced support for students in first year followed
by proactive ‘counseling out’ for unsuitable/incapable

School of Health and Social Care

What have we done?
 Increased specific learning activities
Theory to practice
Interviewing skills

 Improved quality of PLO – workshops training, etc
 Increased participation of Practice Educators on PAP

 Placement transition tutorials & workshops by
Practice Leads
School of Health and Social Care

What might be of use to you?
 A bit about the RICET
Free instrument (just ask for request first &
possible later collaboration)
Valid & reliable
Cheap to administer, easy, quick
Can be administered via V L Environments /email
Can be modified to suit any competency based
evaluations

School of Health and Social Care

What might be of use to you?
 A bit more about the RICET
Useful for developing learning contracts
Useful for mid-placement evaluations
Useful for negotiating differences of opinion
Useful aide to self-reflection and critical
awareness
Identifies ‘rogue’ students and/or practice teachers

School of Health and Social Care

What might be of use to you?
 Audit process so no need for:
research budget
extensive ethics applications
data analysis
 Excellent for rolling curriculum reviews & refinements
 Excellent for Validations & Quality Assurance
Reviews
 MY HOPE:…..collate larger sample for analysis
School of Health and Social Care