Transcript Document

Presentation for the 4th Annual Center for AIDS Prevention Studies
Conference
San Francisco, April 16, 2004
Client Level Data Collection:
The California Experience
Jeffrey Bernstein, MS
Center for AIDS Prevention Studies, California Technical Assistance
Program
Shanna Livermore, BS
Steven R. Truax, PhD
California Department of Health Services, Office of AIDS,
HIV Education and Prevention Branch, Sacramento
1
Today’s Presentation
• Overview of ELI
• What challenges have been experienced at
the state and local levels?
• How does ELI work successfully in
California at the state and local levels?
• How has this technology been used to
inform public health decisions?
2
ELI Background
Overview
Development
3
What is ELI?
• Evaluating Local Interventions (ELI) is a webbased evaluation system that enables California’s
prevention providers to collect and access
information critical for the planning and
implementation of effective HIV prevention
programs.
4
Background
• Evaluation is important for planning
and enhancing programs and for
securing resources.
• Office of AIDS (OA) has long
recognized the need for evaluation and
technical resources required to support
it.
5
Background
• CDC Guidance on evaluation released
in 1999 added additional motivation.
• CDC’s and OA’s shift in new directions
6
Shift happens!!
7
Benefits
• Systematic data collection
– Gives the client a voice
• Consistent reporting requirements
8
Benefits
• There is a strong relationship between data
collection and program planning
• Helps in grantwriting
• Helps structure your services/conversations
9
Benefits
• Immediate feedback to providers
– Over 50 “cannned reports”
– Raw data available
10
Prevention Activities Required
to be Evaluated
• Individual-level interventions
• Group-level interventions
• Outreach interventions
• Prevention case management
• Health Communication
11
ELI Development Partners
• Coordination & Development
– California State Office of AIDS
– Universitywide AIDS Research Program/UCOP
• Program Evaluation Technical Assistance
– Center for AIDS Prevention Studies, UCSF
• Guidance & Input
– Local Health Departments
– CDC
12
Strategic Planning Process for
Development of ELI
• Sought input from stakeholders
• Assessed
– existing evaluation efforts
– state of the art measures, methods
– local feasibility and utility
13
Pilot Phase
Fall 2001
14
Pilot Process for ELI
• ELI piloted in 5 counties
• Feedback collected on
–
–
–
–
data collection forms & protocols
web system
resource issues
technical assistance needs
• Trainings and TA developed
• ELI revised
15
Pilot Feedback: Key Concerns
• Use of individual-level forms in outreach and
group settings
• Need for additional fields, data elements, &
reports
• Ability to download data
• Integration of existing data collection efforts into
ELI system
• Need for training and technical assistance on data
collection, system use, and use of data
• Resources
16
OA Response to Pilot Feedback
• Revised protocol for data collection in outreach and group
settings
• Created additional data fields, optional fields & reports
• Developed downloading capability
• Designed & implemented statewide trainings
• Developed partnership with CAPS to provide evaluation TA
to support use of forms & data
• OA is available for
– system technical assistance
– consultation for integration of ELI forms into existing data
collection efforts
17
Explanation of forms
18
Prevention Interventions to be
Evaluated Using ELI
• Intervention: A specific activity intended to bring
about HIV risk reduction
–
–
–
–
–
1 Individual-Level Interventions (ILI)
2 Group-Level Interventions (GLI)
3 Outreach
1 Health Communication
1 Prevention Case Management (PCM)
19
Individual Level Interventions
20
Group Check Sheet (front)
21
Group Check Sheet (back)
22
Overview of ELI Elements
• Agency Information
• Intervention Info.
–Intervention setting
–Intervention type
–Referrals
–Risk reduction materials
• Behavioral Info.
– Sexual Behavior
– Hepatitis & STD History
– Substance Use History
• Supplemental Info.
• Demographics
–Gender
–Race/ethnicity
–Sexual orientation
–Matching criteria
23
What are the Matching Criteria &
Why are They Important?
•
•
•
•
•
Date of Birth
First Letter of Last Name
Gender
Race/Ethnicity
Zip code
24
Selecting Forms
• Discuss form selection
• Forms should be decided on by the time
intervention is set-up
• Some degree of form flexibility upon Office of
AIDS approval
25
ELI Data Collection Forms
• Not a script!
– Provide a framework for a conversation with a
client
– Optional word use and question order
– Use terms familiar to and appropriate for the
client
26
Explanation of Reports
• Over 50 “canned” ELI reports available
–
–
–
–
Age by Race by Gender
Missing Fields
Intervention Effectiveness
Referrals by Risk
Raw data available
27
Reports
28
Reports
•
•
•
•
•
•
•
•
•
•
Budget Report
Referrals by Risk
Gender by Race by Age Group
Sexual Orientation by Race by Age Group
STD by Race by Gender
STD Type by Race by Gender
Drugs by Race by Age Group
Materials Distributed by Risk
Missing Fields
Intervention List
29
Reports
30
Reports
• Garbage in – garbage out
• Reports are as rich as the data you enter
31
Challenges
32
Challenges
•
•
•
•
Time
Staffing
Changing CDC requirements
Funding
– Program cuts
33
Challenges
• Individual client data requirements in
certain settings
• Federal vs. State vs. Local requirements
• Information Technology is not always a
super highway
• Different levels of knowledge
• Requires new scope of work format
• Intervention set-up: LHJ vs. CBO
34
Challenges
• Data Collection
– New forms/form integration
– Matching Criteria
• BIG challenge with some populations (e.g., clients
with immigration issues)
35
Successes
36
Successes
•
•
•
•
Increased communication
Increased collaboration
Increased technical assistance
Increased training opportunities
37
Successes
• Accurate timely information about HIV
prevention interventions
• Efficiency increase scope of work review
• Intervention Working Group (IWG)
38
Increased focus on quality
interventions!
NOT quantity.
39
Characteristics of Successful
Interventions
•
•
•
•
Science based
Highly targeted
Clearly defined goals and objectives
Realistic goals, objectives and
implementation timeline
• Focused on reducing specific risk behaviors
• Adequately trained staff
40
Characteristics of Successful
Interventions
•
•
•
•
•
•
Sensitive to target population
Sustainable resources and support
Flexible
Part of community plan
Culturally, developmentally competent
Gender specific
41
Lessons Learned
• Secure funding ahead of time
• Lengthen and expand the pilot phase
• Be prepared for major delays
42
Lessons Learned
• Need for forms in Spanish
• Be prepared for varied knowledge levels
– Including data issues
• Promote data utilization
43
Lessons Learned:
Agency level data use
• Clearly identify the intended users of the
data
• Develop buy-in among data stakeholders
• Identify evaluation questions meaningful to
the intended users
44
Lessons Learned: TA and
Training Issues
• Regional trainings require out of county
travel
• Training design must support new and
experienced users of ELI
• SOW vs ELI vs Real Life
• Site visits often perceived as punitive
45
Next Steps
• Outcome evaluation
• Continue on site agency level TA on
intervention set up
• Data Use
– More training required
– Report manual
46
Next Steps
• Suggestions for next steps
– What should be the focus of future ELI
trainings?
– How can we make ELI more agency friendly?
47
Contact info
• Shanna Livermore, Office of AIDS
[email protected]
916-449-5823
• Jeffrey Bernstein, CAPS/CTAP
[email protected]
1-800-983-5550
48