Transcript Using Evaluation to Optimize the Responsiveness of HIV Clinical Training and Capacity-Building Moderator: Janet Myers, PhD, MPH Presenters: Mi-Suk Kang-Dufour, PhD, MPH Kevin Khamarko, MA 2012 Ryan White.

Using Evaluation to Optimize the
Responsiveness of HIV Clinical
Training and Capacity-Building
Moderator:
Janet Myers, PhD, MPH
Presenters:
Mi-Suk Kang-Dufour, PhD, MPH
Kevin Khamarko, MA
2012 Ryan White All Grantees Meeting
Workshop Overview
•
•
•
•
Brief description of the NEC
Overview of AETC process & cross-site data sources
How to use data to target training
Examples of evaluations from specific initiatives:
– HIV Testing Initiative
– Minority AIDS Initiative
– US/Mexico Border Initiative
• Future directions for cross region evaluation
Background on the
AETC National Evaluation Center
AETC NEC Focus
• To provide leadership in the development, design,
testing, and dissemination of effective evaluation
models with emphasis on outcomes.
• To determine the outcomes of AETC clinical
education and training programs with respect to
changes in provider behavior and clinical practice.
• We evaluate all aspects of the training process, from
pre-training conditions to training outcomes.
Training Evaluation Framework
Before Training
Pre-Training
Conditions:
Tailoring Training:
Individual
Characteristics,
Training Motivation,
Pre-training
Environment
(Barriers and
Facilitators Project
Measures)
Curriculum Adequacy
Competency
Assessment
(Curriculum Adequacy
Checklist, Professional
Gap Assessment)
UCSF NEC 2
2007 - 2012
UCSF NEC 3
2012 - 2016
During Training
Training Process
Evaluation:
Number and Type of
Activities
(Event Record) and
Trainees
(Participant
Information Form)
Columbia NEC
1999 - 2004
After Training
Training Outcome
Evaluation:
Training Impact
Evaluation:
Trainee Reaction
(ACRE
Immediate- Post)
Learning and
Behavior Change
(ACRE Follow-Up)
Patient Outcomes
(Chart Review)
UCSF NEC 1, 2 & 3
2004-2007
UCSF NEC 2
2007 - 2012
AETC NEC Services
 Serves as resource to AETC regions and centers:
 Evaluation training: web-based modular curriculum
 Evaluation planning & design
 IRB (human subjects) TA
 Web-based data collection system
 Survey creation
 Analysis
 Dissemination
AETC Standardized Process and
Cross-Regional Evaluation Data
Sources
AETC Process Data Forms
• OMB approved forms required by HRSA
– Event record (ER) collects information about the trainings
– Participant Information Form (PIF) collects demographic
and practice characteristics of training participants
PIF and ER Datasets
Total # of
trainings
Total # of
trainees
•
•
•
FY 2008/09
FY 2009/10
FY 2010/11
18,070
17,190
17,357
139,496
141,751
132,692
There are PIFs without unique IDs, particularly from some regions
In FY 10/11, there were 2.7% repeat PIFs
Across 3 years, about 22% of trainees attended more than one
training
AETC Cross-Region Evaluation
(ACRE)
• Purpose & Significance:
• To implement a select list of standardized evaluation
questions across the AETC network to assess the
effectiveness of training programs
• Standardization Process:
• Questions were vetted and piloted over the course of three
years
• Full implementation was required in fourth year (FY 10/11)
Immediate Post: ACRE IP
• Goal: To implement a standardized survey across
the AETC network to assess trainees’:
• knowledge change
• intent to apply training content
• Four standard questions developed in collaboration
with the AETC evaluation community in late 2007
and finalized in July 2008.
ACRE IP Questions
1. How would you rate your level of knowledge about this
content before the training program? (Novice to Expert)
2. How would you rate your level of knowledge about this
content after the training program? (Novice to Expert)
3. How would you rate the overall quality of the training
program? (Poor to Excellent)
4. I can apply the information learned in my
practice/service setting. (Disagree Strongly to Agree
Strongly)
6 week Follow-up: ACRE FUP
• Goal: to implement a standard quantitative follow
up instrument measuring changes in knowledge and
behavior at 6 weeks after a training
• Qual pilot site findings & a draft quantitative survey
were shared with AETCs in early 2010
• All regions began implementing the follow-up survey
by April 1st, 2011
ACRE FUP Pilot Questions
• The ACRE Follow-up pilot questionnaire focused on 4
main areas:
1. Information sharing (Yes/No)
2. Changes in abilities and/or skills (Scale, 1-5)
3. Changes in practice behavior (Scale, 1-5)
4. Implementation barriers encountered (Yes/No)
ACRE Protocol
AETC trainings are categorized into 5 levels:
– Level 1: Didactic presentations
– Level 2: Skills building
– Level 3: Clinical Training
– Level 4: Clinical consultations (group or individual)
– Level 5: Technical Assistance
ACRE Protocol
• ACRE IP questions are collected immediately
following the event for all Level 1, 2, and 3 trainings
• ACRE FU questions are collected through web-based
survey 6 weeks following all level 2 trainings that
include clinical topics
• Some exceptions made for joint trainings with other agencies
or CME courses
ACRE IP Response Rates**
FY 08/09 FY 09/10 FY 10/11
Total PIFs collected*
(levels 1-3)
Total ACRE IP surveys
completed
% PIFs from (levels 1-3) with
a matched ACRE IP
* Not able to exclude exceptions for this analysis
42,467
88,765
89,550
14,430
20,725
40,537
28%
24%
48%
** FYs 08/09 and 09/10 included pilot regions only
Number of IP records collected by
quarter
14000
12000
10000
8000
6000
4000
2000
08/09 Fiscal Year
09/10 Fiscal year
quarter 4
quarter 3
quarter 2
quarter 1
quarter 4
quarter 3
quarter 2
quarter 1
quarter 4
quarter 3
quarter 2
quarter 1
0
10/11 Fiscal year
ACRE FU Pilot Implementation
(1/1/2011 – 6/30/2011)
Records
ACRE Follow-up
Event Records (ER)
Participant Information Forms (PIF)
N
1501
1363
866
Number of FUP records collected
by quarter FY 2010/2011
600
500
400
300
200
100
0
quarter 1
quarter 2
quarter 3
09/10 Fiscal year
quarter 4
General Findings from ACRE
Evaluations
Overall Means For Each IP Question
5.00
+0.98
4.00
+0.99
+0.76
3.00
08/09
09/10
10/11
2.00
1.00
0.00
Overall quality Ability to apply Knowledge
the information before training
Knowledge
after training
Means by Topic of Training (FY 10/11)
Clinical
Prevention
Organization &
Targeted
management/
& behavior Psycho-social
delivery
population
treatment
change
Overall quality
Ability to apply the
information
Knowledge before
training
Knowledge after
training
Change in
knowledge
4.18
4.24
4.20
4.23
4.32
4.20
4.26
4.23
4.27
4.32
3.22
3.27
3.30
3.26
3.26
3.94
4.03
4.04
4.01
4.09
0.72
0.76
0.74
0.75
0.83
Regression Findings: 09/10 Trainings
• Compared to didactic trainings, trainings that
incorporated skills building or technical assistance
significantly increased perceived ability to apply the
training content (both p<0.05).
• Technical assistance was about twice as likely as
skills building to improve perceived ability.
Regression Findings: 09/10 Trainees
• Direct HIV service providers were more likely to
report higher gains in knowledge and ability to apply
knowledge (p<0.05).
• Compared to 08/09, in 09/10, direct service
providers reported a larger gain in knowledge
(p<0.05).
Regression Findings: 09/10
Trainees cont.
• Compared to white providers, non-white providers
were more likely to report larger increases in
knowledge (p<0.01).
• In 09/10, in particular, African American providers
demonstrated the greatest gains in knowledge
among all groups.
Using Evaluation Data to Target
Training
Example: Responding to National AIDS
Strategy Goals
NHAS has 4 overall goals:
1. Reducing new HIV infections
2. Increasing access to care and improving health
outcomes for people living with HIV/AIDS
3. Reducing HIV-related disparities and health inequities
4. Achieving a more coordinated national response to the
HIV epidemic
Example: Responding to National AIDS
Strategy Goals
Achieving the NHAS goals depends on having a well
trained cadre of health providers – a key function of the
AETCs
Training Providers is explicitly included in the NHAS plan
(Goal 2 step 2):
Take deliberate steps to increase the number and
diversity of available providers of clinical care and
related services for people living with HIV.
Example: Responding to National AIDS
Strategy Goals
FY 08/09
FY 09/10
FY10/11
Novice HIV providers (<1 year)
11%
10%
8%
New HIV providers (<2 years)
12%
12%
13%
Providers in rural settings
19%
20%
20%
Providers who are racial/ethnic
minorities themselves
44%
43%
44%
Providers who serve HIV infected
minorities
54%
55%
55%
Providers who serve mostly (>50%) HIV
infected minorities
30%
31%
29%
Evaluating Specific Initiatives
8000
Number of Training Events for specific
initiatives, by funding year
7000
6000
5000
08/09
09/10
10/11
4000
3000
2000
1000
0
HIV testing (by topic)
MAI
UMBAST
HIV Testing
Myers JJ, Bradley-Springer L, Kang Dufour MS, Koester KA,
Beane S, Warren N, Beal J, Frank LR. Supporting the
integration of HIV testing into primary care settings. Am J
Public Health. 2012 Jun;102(6):e25-32..
Routine HIV testing
In 2008, the CDC provided supplemental funding
to the AETC program to enhance delivery of
intensive, clinic-based education, training, and
technical assistance activities to support the
integration of HIV testing into primary care
settings
5
Routine HIV Testing Trainings:
ACRE IP Outcomes FY 10/11
4
+0.97
3
2
1
0
Overall quality
Ability to apply
Knowledge Before Knowledge After
HIV Testing Trainings: Findings
• Compared with other AETC training, HIV
testing training was longer and used a broader
variety of strategies to educate more providers
per training.
• During education, providers were able to
understand their primary care responsibility to
address public health concerns through HIV
testing.
HIV Testing Case Strategies
• Intensive long-term trainings focused on
developing organization-level systems to
help health professionals deliver testing.
• Technical assistance concentrated on
establishing or revising policies and
procedures for testing and linkage to care
for newly diagnosed patients.
Minority AIDS Initiative
Shade SB, Sackett N, Khamarko K, Koester KA, Bie J, Newberry J, Beal
J, Culyba R, Jacobson K, Kinder A, Nusser J, Myers JJ. Quality of
Comprehensive HIV Care in Underserved Communities: Does
Clinical Training Lead to Improvement. Am J Med Qual. 2012 Aug 14.
[Epub ahead of print]
MAI Chart Review Project
• Chart abstraction and feedback (34 clinics;
N=530) was used to assess adherence to clinical
practice guidelines.
• We identified training needs and assessed change
in clinical practice (14 clinics, N=271).
MAI Chart Review Project
• 49% (95% CI=44, 53) adherence to clinical
practice guidelines at baseline
• Gave feedback associated with chart review and
provided clinical trainings
• 11% increase (95% CI=7, 16) in adherence to
clinical practice guidelines at follow up
re
e
...
n
*
ca
na
tio
g/
Ed
u
cc
i
tin
g
Monitoring
ni
n
Va
es
g
*
Sc
ST
IT
ha
rt
in
g
t
*
C
ito
r in
Vi
si
in
g
Treatment
M
on
tia
l
Te
st
xi
s
*
In
i
b
hy
la
RT
l
20
La
ro
p
A
ra
l
10
O
IP
O
ve
% Change in
Adherence to Guidelines
25
Results
Prevention
*
*
15
*
*
5
0
* p<0.01
MAI Funded Trainings: findings from
process and ACRE data
• Among those providing direct services to HIV
infected clients
– Most provider services to minorities
– Providers who are themselves minorities are more
likely to have a practice that serves predominantly
minority clients
60%
Participants characteristics
MAI Funded vs. Other trainings
50%
40%
30%
20%
10%
0%
Minority providers
Provide direct HIV care Provide direct HIV care
to minority clients
to mostly (>50%)
minority clients
MAI Funded
Other funding
Percent of providers reporting that they serve minority
populations with HIV, by race/ethnicity of provider
70.0%
60.0%
50.0%
40.0%
White
Black
Hispanic
Other
30.0%
20.0%
10.0%
0.0%
No Minority patients
Less than Half of
HIV patients are
minorities
More than half of
HIV patients are
minorities
Based on PIF data from providers of direct services to HIV infected patients,
aggregated across funding years 08/09 to 10/11
MAI Funded Trainings: findings from
process and ACRE data
• MAI trainings were well received across all years
• Knowledge increased a similar amount across
all funding years
ACRE IP results for MAI funded trainings
5
+0.68
4
+0.67
+0.69
3
FY 08/09
FY 09/10
FY 10/11
2
1
0
Overall quality Ability to apply
Knowledge
Before
Knowledge
After
MAI Funded Trainings: findings from
process and ACRE data
• Participants reported increased skills and
abilities and implementing those skills at 6weeks post training
• Skills increases and implementation were seen
across training modes
MAI Training Outcomes by Mode
Chart/Case review
Computer based
Conference call/telephone
Lecture/Workshop
Preceptorship/mini-residency
Role play/simulation
Self-study
Telemedicine
0
Implemented Skills and Abilities
1
2
3
4
Improved Skills and Abilities
5
U.S.-Mexico Border AETC
Steering Team (UMBAST)
UMBAST Background
• 3 regional AETCs (Mountain-Plains,
Pacific, and Texas/Oklahoma) provide
targeted training and capacity building
assistance to health care providers serving
border communities
– Border is defined as U.S. communities 61.5
miles north of the Mexican border
UMBAST Background
UMBAST Training Trends
FY 2008 2009
FY 2009 2010
FY 2010 2011
FY 2011-2012
UMBAST Other UMBAST Other UMBAST Other UMBAST Other
Total
Events
104
2985
104
3389
68
3808
66
3509
Trainees
with PIFs
782
31999
1002
35492
1241
31564
1078
25973
Mean #
of PIFs
9.41
9.33
11.28
8.57
22.75
7.75
15.71
7.48
UMBAST Training Topics
Health Care
Clinical
Organization
Management
and Delivery
Prevention
and
Behavior
Change
Psycho - Targeted Immigrant/ Racial/Ethnic
social Populations Border
Minorities
UMBAST
FY
08/09 Other
78.85*
85.57*
41.35
20.19
27.88*
77.88
43.26*
89.78*
42.48*
41.41
24.49
16.34*
14.97
30.01*
UMBAST
FY
09/10 Other
70.19
79.81
46.15
25
24.03
79.81
50.96
84.63
41.13
38.09
27.91
15.23
16.4
33.17
UMBAST
FY
10/11 Other
85.29
86.76*
70.59
30.88
32.35*
76.47
64.71*
87.73
33.64*
31.28
22.53
10.19*
11.84
26.65*
UMBAST
FY
11/12 Other
56.06
90.91
48.48
12.12
83.33
65.15
40.91
83.67
38.33
36.65
23.94
47.34
12.65
27.61
UMBAST Trends by Level
FY 2008 2009
FY 2009 2010
FY 2010 2011
FY 2011-2012
UMBAST Other UMBAST Other UMBAST Other UMBAST Other
Level 1- Didactic
32.69
24.79
24.03
17.17
42.64
14.29
34.85
11.91
Level 2- Skills Building
9.61
23.01
26.92
21.74
22.05
21.03
36.36
25.71
Level 3- Clinical
Training
0.96*
11.29*
8.65
14.48
1.47
17.01
0
13.48
Level 4- Group Consult
7.69
8.07
11.53
8.76
48.53
8.95
30.3
9.52
Level 4- Clinical Consult 3.84*
23.55*
3.84
26.61
5.88
28.23
0
29.1
Level 5- TA
18.26*
66.35
17.41
20.59
14.62
31.82
13.57
65.38*
UMBAST Training Trends
FY 2008 2009
FY 2009 2010
FY 2010 2011
FY 2011-2012
UMBAST Other UMBAST Other UMBAST Other UMBAST Other
% of Clinical
Trainees
61.3
75.18
63.86
75.1
68.76 74.84
65.2
70.06
% >1 Year
Experience
80.24
78.06
85.5
85.18
72.75 77.55 82.31
81.42
% Urban
53.64
69.22
56.99
69.07
63.89 72.32
62.81
67.9
Capacity Building Evaluation
and
Future Directions
What is Capacity Building?
Expert HIV
Educator and
Provider
Longitudinal Training
and Mentoring Model
(LTMM)*
Advanced HIV
Clinical Care
Intermediate HIV
Clinical Care
Basic HIV Clinical Care
HIV Screening & Diagnosis
Pre-HIV Screening, Diagnosis and Care
*Adapted with permission from materials created by the Northwest AETC
• CBA definition from the MAI project:
“a program of multiple and distinct efforts aiming to improve the
capacity of MAI-targeted clinics and providers to serve HIVinfected patients.”
• As an example, an AETC might work with a clinic to conduct
needs assessment and then plan a multi-month effort of
engagement with a clinic including:
– several didactic trainings for all clinic staff;
– preceptorship opportunities for primary care providers;
– one-on-one assistance in implementing a new patient flow system; and
assistance developing a referral network for complex HIV care.
SUPPORTED HIV CARE
Stage 2
COLLABORATIVE HIV
CARE
Stage 1
EXTERNAL HIV CARE
Foundation
•Provide HIV testing
(routine or
targeted) but no
medical
management for
patients with
positive test results
•Onsite HIV care
integrated with
primary care
through extensive
consultation from
clinical mentor for
initiation of ARV
regimen and for
management of
more complex
patients
•Onsite HIV care
integrated with
primary care,
supported by
consultation with
clinical mentor as
needed for
management of
treatment failure
and/or co-morbid
conditions
complicated by HIV
disease and/or
treatment
COMPREHENSIVE HIV CARE
MANAGEMENt
Stage 3
•Onsite integrated
HIV care with
primary care,
including
management of
complex comorbid conditions,
PMTCT, and
complex OI's with
access to
consultation from
clinical mentors as
needed
NCHCMC’s Creating the Community Care Continuum
Practice-based questions in the newest version of
ACRE:
From the list below, check the types of HIV clinical care services you
provide to your patients/clients:
I conduct HIV testing
I provide primary care for HIV-infected patients
I monitor HIV-specific lab tests
I initiate antiretroviral therapy
I conduct adherence counseling and monitor adherence
I provide prophylaxis and treatment for opportunistic infections
I manage treatment when drug resistance is present
I initiate care to prevent and treat co-morbid conditions
I provide clinical consultation to other clinicians regarding HIV care
Practice-based questions in the newest version of
ACRE:
From the list below, check the types of situations in which you refer HIVinfected patients to other care settings:
I refer patients for all HIV-related care and treatment after diagnosis
I refer when I think the patient needs to start on antiretroviral therapy
I refer when antiretroviral treatment fails
I refer patients with co-infections/co-morbidities
I refer patients with complex treatment issues
I do not refer , but I do consult with HIV care specialists when I have questions
I do not refer, I am an HIV care specialist
I refer patients for all HIV-related care and treatment after diagnosis
I refer when I think the patient needs to start on antiretroviral therapy
Practice-based questions in the newest version of
ACRE:
How knowledgeable are you about where to refer an HIV-infected patient for
care?
I do not have any referral sources for HIV-specific patient care issues
I have limited referral sources for HIV-specific patient care issues
I have referral sources for some HIV-specific patient care issues
I have referral sources for most HIV-specific patient care issues
I have referral sources for all HIV-specific patient care issues
Practice-based questions in the newest version of
ACRE:
In your practice, how often do you provide HIV testing to patients with
unknown HIV status?
To all existing patients
To all new patients at intake
To patients based on risk factors
When a patient requests it
I refer patients elsewhere for HIV testing
I am not involved in HIV testing in any capacity
To all existing patients
To all new patients at intake
To patients based on risk factors
THANK YOU!
Questions?