Transcript Evaluating learning about working in partnership with

Evaluating learning about
partnership with service users
using Concept Mapping
Roxana Anghel
[email protected]
Joanna Fox
[email protected]

Level 2b outcome (Kirkpatrick, 1997; Barr, 2000) – Acquisition of
knowledge – concepts, procedures and principles of working with service
users and carers (in Carpenter, 2005:6)

Focus:
 Evaluate the progress in learning about working in partnership with
service users throughout the first two years of the BA course
Rationale




The BA (Hons) SW degree - strong emphasis on training focused on
partnership with service users (DoH, 2002)
 ‘All social workers will learn and be assessed on partnership’ (p16)
Core expectation that defines ‘good practice’ (NOS, GSCC, QAA)
Service users and carers consider ‘very important’ for students to learn
how to involve them in assessing their needs, and how to treat them
with respect (Barnes et al, 2000; Anghel & Ramon, 2009)
However a complex concept with varying definitions, in practice and
education often implied rather than made explicit (Taylor & Le Riche,
2006)

Generally taught in an embedded or discrete way – ‘no defined partnership
curriculum’, not explicitly defined as an outcome (Taylor & Le Riche,
2006).

Anglia Ruskin - embedded and discrete approaches to learning
(involvement of service users in curriculum delivery, and teaching about
aspects of partnership on various modules).

As learning about partnership is spread across various learning
opportunities on the BA it is important to assess the students' gradual
understanding of the concept and acquisition of a meaningful framework
before becoming practitioners.


David Ausubel (1968, 2000) – Meaningful Learning (opposed
to rote learning) is achieved when

the learner assimilates new concepts into a pre-existing
conceptual framework stored in the long-term memory

the learner deliberately seeks the logical connections between
new and old learning thus choosing a meaningful learning
mindset

the concepts progress in acquiring meaning as new concepts
are added

retention depends on the strength of the meaningful
connections between the concepts stored

meaningful learning is personal
Joseph Novak (1984) – Concept Mapping
Concept Mapping (CM)


A visual map of the summary of an individual's understanding of a
knowledge domain (Novak & Canas, 2006)
Concepts – perceived regularities in events or objects, or records of
events or objects, designated by a label.
 Events, objects, even emotions or feelings (Freeman & Jessup, 2004)

Differs from brainstorming and mind maps by making explicit the
concept-links

Initially designed as a research instrument used mainly in science
education, but increasingly being used in a variety of teaching and
research fields including social work (Anghel & Fox, 2008; Webber et
al., unpublished)
T1 map before
MA module
Cross link
Concept link
Research Design

Before and After quasi-experimental longitudinal design - three
measurements - two academic years
 CM and questionnaire at T1 (induction week BA); T2 (end year 1);
T3 (end year 2)

Questionnaire
 Demographic details
 Students’ previous experience of social work and
 Rating possible impacting factors
 Non-academic (media, workplace, personal experience)
 Classroom-based (various modules, lecturer teaching style,
discussing with colleagues)
 Non-classroom-based academic factors (practice/ observational
placement)
 ‘no impact’, ‘little impact’, ‘some impact’, ‘great impact’.
Research Process

University colleagues consulted on the idea of outcomes evaluation, on
the research design, and later on using CM.

Some allies but initially difficult to propose an outcomes evaluation
culture

Advisory group – mixed – working mostly on the development of the
CM analysis method
 2 service users
 2 year three SW students
 3 researchers (one familiar with SPSS analysis)
 3 senior lecturers
 Students and service users paid for their time and travel (additional
£5000 secured from University fund)
Research Procedure

During induction week BA students received one hour instruction on
CM, a handout with the instructions, and the participant information
sheet

Two days later they were invited to participate by signing a consent
form – participants given 30 mins (negotiation with lecturers) to
complete handwritten CMs individually. Student with dyslexia
supported.

Instruction: complete the conceptual map of ‘working in
partnership with service users’

Questionnaire completed after each CM

Before the subsequent measurements the students received 15 mins of
instruction as reminder
Sample



At T1 – 13 (17% of BA cohort)
At T2 – 6 – attrition 53%
T2 sample
 Females
 4 White British
 1 student with dyslexia
 3 students with previous experience of social work
(employment, personal experience)
 1 Asian – previous experience of social work (volunteering)
 1 Black Caribbean – no previous experience
 Age ranges: 36-45
Analysis

Novak (1984) – structural method – scoring the maps by giving specific
values to hierarchies, concept-links and cross-links.




Arbitrary values
Mechanical counting of correct links
Hierarchies may not be relevant to all types of knowledge (Freeman,
2004)
West el al. (2000) – relational method – based on degree of
foundational or core relationship to the main concept.
Scoring instructions developed by the team

Combination of structural and relational methods

Acknowledges the importance of the theoretical basis of conceptlinks and cross links, but rejects the value of hierarchies and the
mechanical counting of concepts

Acknowledges the importance of the degree of core relevance of the
ideas included in the map to the main concept
Description of link
False connections, whether indicated through concept links or only by ‘???’
Tautologies
Relevant concepts linked by ‘????’ (no concept link)
* Where the ‘???’ cannot be assumed with certainty by the scorer, as there could be
more than one alternative explanation to ‘???’
* Where the scorer can assume with certainty the meaning of ‘???’ as the only
possible alternative, please score depending on the relevance of the concept to the
main concept/question
Score
0
1
2 or 3
Debatable links - Propositions have sense, but are true only in some circumstances
2
Contextual – propositions that are relevant to social work but are far from the core of
the main concept
2
Lists – where a number of concepts are attached to one concept link but individually
do not add theoretical value to the map – score once the entire bunch
2
Relevant – propositions that are relevant to the concept (the degree of relevance
should be captured in the qualitative analysis)
3
Cross-links
The links that connect two concepts belonging to two different branches that helps the
reader see how concepts in two different domain of knowledge in the map are related
and indicate meaningful learning
- where the cross-link is not specifically relevant
to the main subject – score as contextual link
- where the cross-link appears as ‘???’ but the meaning is obvious
and it’s close to the main concept – give full cross-link value
6
2
6
Ex.
Developing the Scoring instructions







Advisory group – service user involvement
‘Blind’ scoring
2 raters – scoring independently, meeting to discuss were wide
differences in scoring (only a few)
Instructions altered three times and scoring repeated with three different
sets of raters
The final version of the scoring method was then applied to the entire
set of maps in one go by two new raters: the lead researcher and a senior
lecturer
Pearson (0.96) and Spearman (0.87) – Strong inter-rater reliability
Wilcoxon (r=0.854) – Temporal stability
Findings


Increased knowledge demonstrated
Significant limitation – only 6 pairs of maps, then 4
160
Map Scores
140
120
1
2
3
4
5
6
100
80
60
40
20
0
T1
T2
T3

Most impacting sources of learning at T1: Ethics and Values; observational practice;
Poverty, Social Exclusion and Social Work, and academic reading.

At T2: Social Work with Children and Families, Social Work with Adults, practice
placement; Principles and Skills of Social Work; academic reading; and discussing with
colleagues
Student 4 profile





Female 36-40
White British
No disability
Previous contact with social work through employment in NHS –
generally good appraisal of her experience
Scores
 T1: 35.5
 T2: 80.5
 T3: 136.5
T1: 35.5
T2: 80
T3: 136.5
Student 3 profile




Age range – 36-40
Asian ethnic origin
Previous volunteering work – appraised as ‘excellent in opening my eyes
to a lot of issues concerning social services’.
Scores
 T1: 29.5
 T2: 26.5
 T3: 48
T1: 29.5
T2: 26.5
T3: 48
Conclusions





Some evidence of significant increase in knowledge – however,
inconclusive due to small number of maps – thus CM as method for
measuring social work education outcomes needs to be explored further
In particular the balance between quantitative and qualitative evaluation
of maps has to be strengthen.
CM – has potential for being used routinely as evaluation method –
validity depends on familiarity with the tool and adequate time given to
learn it and complete it.
Being based on the theory of meaningful learning CM pertains to a wide
range of subject matters in social work and other fields.
Workshop: CM as formative evaluation method, learning and teaching aid,
and empowering tool in social work practice
Selected Bibliography











Anghel R. & Ramon S. (2009) Service users and carers’ involvement in social work education:
lessons from an English case study. European Journal of Social Work, 12 (2), p. 1 – 15 .
Ausubel D. (1968) Educational psychology: a cognitive view. New York: Holt, Rinehart and
Winston
Carpenter J. (2005) Evaluating Outcomes in Social Work Education, SCIE, SWAP and SIESWE,
London
DoH (2002) Requirements for social work training [online].
Cmap software FREE download - http://cmap.ihmc.us/download/
Freeman L.A. & Jessup L.M. (2004), The power and benefits of concept mapping: measuring
use, usefulness, ease of use, and satisfaction, International Journal of Science Education, 26(2),
151-169
Novak J.D. & Gowin D.B. (1984), Learning how to learn, Cambridge University Press,
Cambridge
Novak J. & Canas, A.J. (2008) The Theory Underlying Concept Maps and How to Construct
and Use Them,
http://cmap.ihmc.us/Publications/ResearchPapers/TheoryCmaps/TheoryUnderlyingConceptMap
s.htm
Taylor I. & Le Riche P. (2006) What do we know about partnership with service users and carers
in social work education and how robust is the evidence base?, Heath and Social Care in the
Community, 14(5), 418-425
West D.C., Pomery J.R., Park J.K., Gerstenberger E.A., & Sandoval J. (2000), Critical thinking
in graduate medical education – a role for concept mapping assessment?, American Medical
Association, September 6, 284(9), p 1105-1110
West D.C., Park J.K., Pomeroy J.R. & Sandoval J. (2002) Concept mapping assessment in
medical education: a comparison of two scoring systems, Medical Education, 36, 820-826