Transcript Slide 1

Psychometric Properties of the Level of
Knowledge Use Survey (LOKUS) Tool
(Shreya Telang, Machiko Tomita, Vathsala Stone)
Presenter:
Vathsala I. Stone
[email protected]
University at Buffalo/Center on Knowledge Translation for Technology Transfer
http://kt4tt.buffalo.edu
NARRTC Annual Meeting, Apr. 27, 2011
Background
Overall Context:
Knowledge Translation (KT) (CIHR, 2009; Sudsawad, 2007).
Knowledge (Evidence)  Practice  Impact on beneficiaries
Specific Context:
Technology based Research - Sub-optimal level of demonstrated impact from
R&D investment
KT4TT Center:
• Develop KT models, methods & metrics for technology based R&D
• Conducting “end-of-grant” KT interventions in 3 technology areas –
Augmentative and Alternative Communication (AAC), Environmental Access
& Wheeled Mobility.
• LOKUS measures Knowledge Use as indicator of intervention effectiveness.
Background (contd.)
KT Intervention Project in AAC
• Selected AAC Study (new knowledge):
End-of-grant; NIDRR funded; Innovative
• Intervention strategy:
Contextualized Knowledge Package (CKP) + Training
(webinars) + Technical Assistance
• Effect on 6 types of Knowledge Users (stakeholders):
• Manufacturers; Clinicians; Transition Brokers; Researchers;
Policy makers; Consumers with disabilities
1. Level of Knowledge Use Survey (LOKUS) tool seeks to:
• identify the level of use (and the corresponding categories)
• of new knowledge generated by technology-based
research
• (attained) by stakeholders (users) of that knowledge.
2. Developed by the Knowledge Translation on Technology
Transfer (KT4TT) Center
• Based on Hall et al (2006)
3. Psychometric study of LOKUS conducted by Shreya
Telang (2011).
Development of LOKUS: Content Validation
• Tool based on Hall’s (2006) framework - Levels and
Categories - educational setting.
• Expert testing of items – 3 KT scholars and 4
Technology Transfer (TT) experts.
• Tool showed 100% Face Validity.
• Tool altered: added/changed levels, eliminated
irrelevant categories; closer to KT4TT context.
• Web version pilot tested by 6 individual stakeholders.
• Resulted in LOKUS, subject of this study.
Framework for Item distribution in LOKUS (Based on Halls et al, 2006)
CATEGORIES
Being
Aware
Getting
Information
Sharing
Assessing
Planning
Implementing
0 - NON-AWARENESS
1- AWARENESS:
2 – ORIENTATION:
LEVELS
3 – PREPARATION:
x
x
x
x
x
x
4 - INITIAL USE:
5 - ROUTINE USE:
6 – EXPANSION:
7 –COLLABORATION:
8 – INTEGRATION:
9 – MODIFICATION:
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
Focus of the Psychometric Study
• Reliability
• Test-Retest
• Alternate Assessment method (Web based Vs. Paper-andPencil method)
• Responsiveness to change
• Ability to detect changes in knowledge use over time;
• Examination of Developmental nature of levels and
categories.
R
R
Web
based
group
Paperandpencil
Group
R
e
s
e
a
T3
Study
T1 r
T2
Intervention (at 4
(NK) in
(Baseline)c (at 1 week)
(CKP)
weeks)
LOKUS
h
A
O D
O
X
O
B
O e
O
O
s
C
O i
O
O
A
O g
O
X
O
n
B
O
O
O
C
O
O
O
Method: Participants
Represent one of the 5 stakeholder types in the KT4TT Center
intervention project – the “Clinicians”.
Inclusion: College Students / faculty members from allied health
disciplines (Occupational therapy, nursing…..) and clinicians
experienced in AAC; 18 years or older;
Exclusion: participation in similar survey/focus group within past 6
months.
Sample Size: Based on power analysis. large effect size d =.94 (Colbert,
1977); N = 64 (needed); 72 recruited, pre-screened and randomly
assigned to two groups; Final N = 69 (35, paper-and-pencil, 34 web
based).
Method: Intervention
LOKUS items asked about New Knowledge from 3 NIDRR
funded published studies in AAC - Study A, Study B and
Study C.
To identify Responsiveness to changes, a simulated
Intervention condition was introduced between T2 and T3
by providing
• a CKP on Study A (treatment) only to both groups.
• and no CKP on the other two studies (controls).
Method: Data Analysis
Focus
Data
Equivalence of
Demographics on age,
participant groups (WB gender, marital status,
and P&P)
highest educational level,
experience in AAC
Test-Retest Reliability LOKUS scores at T1 and
T2, on levels and
categories.
Compare Alternate
LOKUS scores at T1 and
Assessment methods T3, on levels and
categories.
Responsiveness to
change
LOKUS scores at T1 and
T3, on levels.
Developmental
nature of levels
Participant changes in
level status.
Analysis
Independent t-tests and chi-square tests.
T1 - T2 comparison using ICC (3,1)
Mann-Whitney U-tests, for difference
between groups regarding (1) change
(dichotomous) in levels and (2) number
that changed their level status.
Identify significant change (i.e., # of levels
changed >Std Error of Measurement);
Compare Studies A , B and C.
Developmental, if majority of changes
move up a level.
Results: Demographics
There was no significant difference between the
two groups regarding:
•
•
•
•
•
Age ( t = .757; p = .452);
AAC experience (t = .880; p = .382);
Gender (chi sq = 1.292; p = .256);
Marital status (chi sq = 4.872; p = .301) and
Education (chi sq = 1.003; p = .793)
Levels
StudyA 0- Non Awareness
Corr. =1.0) 1-Awareness
2-Orientation
7-Collaboration
0- Non Awareness
1-Awareness
StudyB
(Corr.=1.0 2-Orientation
)
3-Preparation
4- Initial Use
7-Collaboration
Study C 0- Non Awareness
(Corr.=1.0 1-Awareness
)
2-Orientation
Note: Identical responses at T1 & T2
t
R
e
t
Number who maintained
Levels at e
T2
s
50 t
12
R
6
1 e
49 l
7
i
10
a
2
1 b
1 i
58
l
8
3 i
t
y
Categories
1
2
3
4
5
6
n/a
n/a
4
1
n/a
n/a
4
1
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
10
7
1
3
3
3
n/a
1
n/a
n/a
5
2
n/a
n/a
n/a
1
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
Study
(NK)
Study
A
Study
B
Study
C
Change from T1 to T3
f
%
Total
P &P
66
33
95.7
100
Web
31
91.2
Total
5
7.2
P &P
2
5.7
Web
3
8.8
Total
8
11.6
P &P
4
11.4
Web
4
11.8
p
o
n
s
i
T1v
Mean
e
n
0.43
e
s
0.65s
t
o
C
0.17h
a
n
g
e
T3 Mean
Mean
Diff.
SD
SEM
1.88
1.45
1.36
0.71
0.86
0.21
1.13
0.44
0.33
0.16
0.60
0.25
e
n
c
e
b
Study
Change between T1 and T3
e
(NK)
t
f
%
w
Total
66
95.7
e
Study A
P &P
33
100
e
Web
31
91.2
n
Total
5
7.2
t
P &P
2
5.7
Study B
h
Web
3
8.8
e
Total
8
11.6
M
Study C
P &P
4
11.4
e
Web
4
11.8
t
Note: * Significant; p < 0.05
h
Difference between Methods
(p-level)
Changed/Not
Changed
Number of levels
0.038*
0.194
0.147
0.125
0.961
0.100
m
o
v
e
d
Levels at T1
Number
who moved up levels at T3
u
1 2p 3 4 5 6 7 8 Tot
l 4 1 0 1 0 0 62
0- Non Awareness (n=97) 37 19
1-Awareness (n=45)
8e 6 0 0 0 0 0 14
2-Orientation (n=41)
-v 6 0 0 0 0 0 6
3-Preparation (n=18)
-e 0 0 0 0 0 0
4- Initial Use (n=3)
-l
- 1 0 0 0 1
5-Routine Use (n=1)
-s - - 0 0 0 0
6-Expansion (n=1)
- -a 0 1 1
7-Collaboration (n=0)
- -t - 0 0
8- Integration (n=1)
- -T - 0 0
3
Summary and Conclusions
LOKUS demonstrated:
• Good Face Validity.
• Excellent Test-Retest Reliability for both levels &
categories.
• Good responsiveness to detect change regarding use
of New Knowledge.
• Developmental nature for lower levels.
LOKUS is a usable tool at least for AAC field. Longitudinal
study needed to extend conclusion for higher levels and
for categories.
Key References
1. CIHR. About knowledge translation. Retrieved October 25, 2009, from http://www.cihrirsc.gc.ca/e/29418.html
2. Hall, G.E., Dirksen, D.J., and George, A.A. (2006). Measuring Implementation in Schools: Levels of
Use. Austin, TX: Southwest Educational Development Laboratory (SEDL).
3. Lane, J.P., Stone, V.I., Bauer, S. M., Leahy, J.A., and Tomita, M.R. (2008). Center on Knowledge
Translation for Technology Transfer. Proposal submitted to National Institute for Disability and
Rehabilitation Research (NIDRR)’s Disability and Rehabilitation Research (DRRP) Program (84.133A-7).
4. Sudsawad, P 2007. Knowledge Translation: Introduction to Models, Strategies, and Measures. Austin:
Southwest Educational Development Laboratory, National Center for the Dissemination of Disability
Research. (p.4; 21-22)
5. Telang, S.R. Establishing Psychometric Properties of the Level of Knowledge Use Survey (LOKUS)
Questionnaire for Knowledge Translation for Technology Transfer. (Unpublished Masters Thesis).
University at Buffalo, State University of New York.
Acknowledgement
This is a presentation based on Shreya Telang’s (2011) work
for her masters’ thesis which was partially supported by the
KT4TT Center under funding by the National Institute on
Disability and Rehabilitation Research of the U.S. Department
of Education, under grant number H133E030025. The
opinions contained in this presentation are those of the
grantee and do not necessarily reflect those of the U.S.
Department of Education.
Thank you!!
Contact: [email protected]
http://kt4tt.buffalo.edu
Questions?
Appendix
Study A
Light, J. and Drager, K. (2007). AAC technologies for young children with complex
communication needs. State of the science and future directions. Augmentative &
Alternative Communication, 23 (3), 204-16.
Study B
Light, J., McNaughton, D., Weyer, M. & Karg, L.(2008). Evidence-based literacy instruction for
individuals who requireAugmentative and Alternative Communication: A case study of a
student with multiple disabilities. Semin Speech Lang, 29 (2), 120-132.
Study C
Quach, W.(2007). Facilitating children’s learning of Augmentative and Alternative
Communication systems. Retrieved from Proquest Digital Dissertations. (AAT 3275080).
LOKUS and the LoU Scale: Differences
LoU Scale
To measure Use of
Innovations
Educational setting part of Concerns Based
Context
Adoption Model
(CBAM)
8 Levels linked by
decision points; 7
Framework
Categories in each level
Systematic interviews
Observation
Method
Purpose
LOKUS
To measure Use of Innovations
Broader, social setting – focus
on knowledge from technology
based research
10 Levels, with 3-6 Categories
under levels
Web based, branched items
eliciting self reported responses
- large scale survey capability