Transcript Slide 1

Mobile Assessment of Work-Based
Learning: Evaluation in Progress.
Chris Dearnley – Learning and Teaching Fellow,
University of Bradford
Julie Laxton- ALPS Teaching Fellow, University of
Leeds
Shupikai Rinomhota – Lecturer, University of Leeds
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
ALPS CETL
Who we are?
A centre for excellence in teaching and learning (CETL) focussing on
assessment and learning in practice settings. 5 Universities (Leeds, Leeds
Metropolitan, Bradford, Huddersfield and York St John) crossing 16 health
and social care professions
What is the aim of this programme?
•To achieve excellence in assessment and learning based on
interprofessional Common Competency Maps, which can be delivered by
mobile technology
•To ensure that students graduating from courses in health and social care
are fully equipped to perform confidently and competently at the start of
their professional careers
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
ALPS CETL
This programme is developing 3 approaches to improving
confidence and competence in pre-registration health and
social care students
• Promoting interprofessional assessment of core
competences
• Testing the viability of the use of mobile devices to deliver
assessment and learning in the practice setting
• Encouraging service user and carer involvement in student
feedback
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
ALPS Mobile
Assessment processes
• 3 competency maps developed;
• 5 generic assessment scenarios;
•
•
•
•
•
Working Interprofessionally
Gaining Consent
Providing Information
When to Consult and Refer
Demonstrating respect for Service User
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
ALPS Mobile
Assessment processes
• These facilitated formative feedback from a range of
stakeholders
• Delivered onto the mobile devices, linked with e-portfolio
system for tutor feedback - T-mobile Vario I with unlimited
internet access
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
E-Valuation working
group
What was the purpose of this group?
Initially - to research the learning and teaching aspects
related to mobile assessment processes
Eventually - incorporated evaluation of the ALPS
assessment tools in all formats
Our key motivation/question –
DOES IT MAKE A DIFFERENCE?
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
E-Valuation working
group
•Representatives from all five sites
•3 clearly defined work streams
Work Stream 1 – evaluation of devices and tools
with students (two stages)
Work Stream 2 – evaluation of mobile
assessment tool with staff (lecturers & mentors)
Work Stream 3 – reliability and usability
questionnaire
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
Evaluation Challenges
Please read the extract taken from the ALPS
Evaluation Strategy and consider the practical
implementation of this plan, particularly taking into
account the following:
•The size and complexity of the cohorts involved
•The type of delivery of the tool
•Obtaining ethical clearance
•The data collection methods employed
What issues might be encountered?
Would you have used the same approach?
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
Evaluation Challenges
•Diversity of HEIs
•Different ethical approval processes
•Uni-professional or multi-professional focus groups?
•How do we ensure consistency?
•Mobile vv paper based tools delivery
•The challenges of collaboration!
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
Mobile device
implementation processes
• Selected cohorts across the
5 sites
•All professions included
•Standard training delivered
•Placement areas prepared
•Confidentiality and security
ensured
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
Evaluation processes
Work Stream 1 – evaluation of mobile
devices and assessment tools with
students (stage 1)
• 1 Cohort specific focus groups; 7 professions &
79 students
• ALPS Research Officer attended all to ensure
consistency across sites
• Thematic analysis of focus group sessions
• Students invited to complete on-line diary/blog
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
Findings
• Two extremes of students – those who
use it a lot and those who don’t use it at
all.
• When no alternative and it offers
something of value they will use it –e.g. in
community settings (OTs, SWs) or where
they have no computer access.
• Perseverance by those who find it
useful/have a reason e.g. dyslexic
students
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
Findings
• There appears to be a clear differences
across the professions due to culture,
client group and context in which they
operate; e.g. audiologist as clear
technicians as opposed to SW; medics
• Huge training needs so that students
understand potential uses, e.g. audio
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
Student Quotes – the good...
• Yes I think it’s good in that respect ……(student referring
to reflection)
• …………I found it great on a weekend for me because I
could go on to the library and access um a PDF and it’s
actually really quite clear on the screen……..
•
….You see it’s nice getting the comments from (Tutor)
because I don’t get to see (Tutor) a lot in clinic …. and I
can see if we’d been doing that from the start that would
have been good because he’d have been able to see
how I’d progressed…….
Assessment and Learning in Practice Settings (ALPS) © September 2008
http://www.alps-cetl.ac.uk
Student Quotes – the bad...
• We need a device with better functionality so that the
software just goes on it
• If I could use if for personal purposes as well like phone
calls, I would use it more
• I think the idea of having something where you can look
stuff up when you’re with somebody is, is great, but
realistically with the, the size and quality of the internet
pages you get on here and the speed of it, it doesn’t,
doesn’t really make it very, very workable.
Assessment and Learning in Practice Settings (ALPS) © September 2008
http://www.alps-cetl.ac.uk
Student quotes – the ugly...
• our deputy manager has asked me not to use it
with clients because he thinks people associate
them playing – little gizmos, with playing games,
even though I explained what it was
• …. and it just takes up more time than it saves
I’m afraid
Assessment and Learning in Practice Settings (ALPS) © September 2008
http://www.alps-cetl.ac.uk
Work Stream 2 – evaluation of
mobile assessment tool with
staff (lecturers & mentors)
There are barely
enough computers
in practice settings,
so students need
there own, but this
also means there
they are not
technically minded
enough to
understand student
use
The HPC
ask for a
“wide variety
of evidence”
I don't see why
we shouldn't
have shared
assessment
tools given the
similarities
(OT/physio)
It skills are part of
Band 5
requirements and
we should expect
an electronic
portfolio
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
Work Stream 3 – reliability and
usability questionnaire – 112
responses
•Whilst most participants used computers for a variety
of purposes only 51.4% participants used their mobile
devices and then it was only 1-2x per month
•Difficulties for their use – most commonly cited was;
42% no connection and 36% device lost their charge.
•96% received training before using their device.
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
Overview of Outcomes
•Mobile learning & assessment offers huge
potential for increasing flexibility and choice
• Many students and staff like the idea
•The technology has to be user friendly –
students don't want an extra device –they want
to use their own
•This presents security challenges
•There are cost and accessibility implications
•Mobile assessment of practice settings
requires a shift in culture underpinned by
training and awareness raising
Assessment and Learning in Practice Settings (ALPS) ©
http://www.alps-cetl.ac.uk
ALPS Contact details
Chris Dearnley – [email protected]
Julie Laxton- [email protected]
Shupikai Rinomhota- [email protected]
Email: [email protected]
Tel: 0113 343 6352
General Website: www.alps-cetl.ac.uk
Assessment and Learning in Practice Settings (ALPS) © September 2008
http://www.alps-cetl.ac.uk