Transcript Slide 1

National Child
Traumatic Stress Initiative
(NCTSI)
Data and Evaluation Webinar
Webinar Overview

Overview of data and evaluation for the NCTSI:
 National Center for Child Traumatic Stress
(NCCTS) Data and Evaluation Program/Core
Data Set (CDS)
 Cross-site Evaluation
 Transformation Accountability (TRAC)
NCCTS Data and Evaluation Program

Range of quality improvement, data
collection, and evaluation initiatives

CDS:




Captures quantitative data on Network-supported
treatments and services
Types of traumatic events
Treatments delivered
Functioning of youth over time
Cross-site Evaluation

Eight study components:


Child and family outcomes and satisfaction with
services
Network functioning with regard to trainings,
collaboration, product development, and product
adoption and dissemination
TRAC

TRAC:


Uniform tool for collecting data on mental health
outcomes across programs
Grantees are required to collect TRAC data as
part of SAMHSA’s reporting obligations
NCCTS Data and Evaluation Program:
Overview and Next Steps
Ernestine Briggs-King, Ph.D.
National Center for Child Traumatic Stress
Duke University School of Medicine
Objective: How Do All the NCTSI Data
Collection Initiatives Fit Together?
SAMHSA
initiatives
QPR
TRAC
MACRO
International
Initiatives
Cross-site Evaluation
NCTSN Member
sponsored
initiatives
Surveys
Data & Evaluation
Program Initiatives
Core Data Set
eSUF, Metrics,
Product
Evaluation
Data and Evaluation Program
Ernestine Briggs-King, Ph.D.
919-682-1552, ext. 254
[email protected]
Bart Evans, B.S.
Project Leader
919-668-8981
[email protected]
Rebecca Wilgus, RN, M.S.N.
919-668-8897
[email protected]
Data and Evaluation Program: What Do We Do?
The Data and Evaluation Program is broadly
concerned with data collection, analysis, and
dissemination.
The Duke Clinical Research Institute (DCRI) provides
the following services:
●Data operations (includes project management,
regulatory, contracts, finance)
●Data management
●Data coordinating activities
The NCTSN Data Repository is housed at the DCRI.
Data Operations Committee includes key members
from NCTSN Centers, SAMHSA, Macro, UCLA, and
Duke.
Data and Evaluation Program: What Do We Do?
Regulatory Issues
Data Collection Initiatives
● Electronic Service Utilization Form (eSUF)
•Aggregate data on clients, services, and trainings
● NCTSN member-driven data collection initiatives
•Surveys (e.g., New Grantee Surveys)
•Product evaluation
•Collaborative group initiatives
● Evaluation
•Training, implementation, and dissemination efforts
● Core Data Set (CDS)
•Client-level data about youth and families served
Introduction: CDS
 Quality improvement initiative
 Network-wide data collection that will provide
answers to central Network questions
 Essential for ensuring that the work done in the
Network is systematically measured, disseminated,
and recognized
 Numerous benefits for participating clinicians,
individual sites, and the Network as a whole
● Clinical improvements and utility
● Real-time reports
● Sustainability
NCTSN Clinical Summary Report
What is in the CDS?
Demographic and living situation information
Trauma history and detail
Indicators of severity
Treatment services and interventions
Standardized assessment measures
Behavioral and Emotional Difficulties
● Child Behavior Checklist
● UCLA PTSD Reaction Index
● Trauma Symptom Checklist for Children-Alternate
 CDS measures: administered at treatment entry, end of
treatment (if short-term) or every 3 months





Bidirectional Data Flow
● All centers will have continuous
(interactive) access to client-level
reports (i.e., scoring of measures,
summaries of client history)
● All centers will have continuous
(interactive) access to data status
reports that describe data entered
by their center
● Each center can receive from
Data Repository a ready-toanalyze data set of its own data
Next Steps: Data Collection
Data collection and data entry resource requirements
are determined by your implementation strategy
●Real-time—Clinicians collect and enter data online
during assessment of client and family
●Ex post facto—Clinicians collect data on paper
during their assessment, and data entry staff enter
data into InForm at a later date
●Hybrid model—Combination of above models
What will work best at your center??
Next Steps: Regulatory Processes
Start the regulatory process soon!!
•
•
•
•
•
Assign a “point person”
Obtain documentation from your IRB
Submit a Public Health Information Letter
Signed Data Use Agreement
Templates, roadmaps and technical assistance
are available
- Web site: www.nctsn.org
- Data and Evaluation staff: Just a phone call away!
If your center is not affiliated with a local IRB, the DUHS IRB may serve as your IRB of record.
Contact Bart Evans at the Data and Evaluation Program for more information.
Next Steps: Training
Training—Technology
 How to use the InForm system
Training—Clinical Content
 How to complete the case report forms
 How to administer the standardized
assessments
NCTSN Data and Evaluation Program’s
Commitment to You
•
Customer support
DCRI EDC Help Desk: [email protected]
Phone: 888-372-7743
NCTSN Data and Evaluation Team: [email protected]
Phone: 919-668-8182
•
Technical consulting and training
•
Clinical expertise and guidance
Data and Evaluation Webinar
Overview: Data Collection and Evaluation Activities
Introduction to Macro
 Macro International
• Management and consulting research
organization
– More than 1,000 employees
– Multiple offices in the United States and overseas
– Many contracts with local, State, and Federal governments
(80–90 percent of Macro’s work)
• More than 40 years as a government
contractor
–
–
–
–
Applied health services research
Information technology
Organizational development
Communications
20
The Common Goal
To raise the standard of care and improve
access to services throughout the United
States for children and adolescents and their
families who have experienced trauma
21
Monitoring and Evaluating the NCTSN
 Evaluation has always played a large role
in the NCTSN
 Initial congressional authorization required
a “rigorous evaluation plan” for each
center (P.L. 106-310, 2001)
 Evaluation plans were to include methods
for assessing the effectiveness of:
• Processes
• Treatment/intervention outcomes
22
What is Cross-site Evaluation (CSE)?
NETWORK LEVEL
Network
Collaboration
Adoption and
Implementation
Product
Development
and
Dissemination
Study
NREPP
Descriptive and
Clinical Outcome
Study
Satisfaction
Study
CLIENT & FAMILY
LEVEL
National
Impact
TraumaInformed
Services
23
History of NCTSN Growth and
CSE Participation
 10 new grantees join NCTSN
 5 grantees continue on
consecutive awards
 44 grantees currently funded
 80 grantees ever funded
 Macro wins competitive bid to
conduct CSE
 Macro begins year of evaluation
design, development, and review




53 grantees
currently funded
OMB approves CSE design
Macro begins implementing CSE
44 grantees currently funded
70 grantees ever funded
36 grantees
currently funded
 16 new grantees join
NCTSN
 12 grantees continue on
consecutive awards
 44 grantees currently
funded
 70 grantees ever funded
20 grantees
currently funded
NCTSI program authorized
under Children’s Health
Act of 2000
2000
2001
2002
2003
2004
2005
2006
 CSE implementation ongoing
 44 grantees currently funded
 70 grantees ever funded
 5 new centers join NCTSN
 3 grantees continue on
consecutive awards
 54 grantees currently
funded
 85 grantees ever funded
2007
2008
Fiscal Years
Note: Counts of “currently funded” grantees exclude the National Center for Child Traumatic Stress
24
Continuum of Training and
Technical Assistance
EVALUATION LIAISON SUPPORT
MONTHLY UPDATES
Regional
Training
Reviewing
Details of the
Cross-site
Evaluation
Individual
Training and
Technical
Assistance
Creating Centerbased Strategies
for Implementation
Cross-site
Procedures
Manual
Componentspecific
Updates by
E-mail/Ongoing
Consultation
Steps Toward Implementation
Ongoing Technical Assistance
25
Getting Started: Practical
Help Tips
Getting Started
Engaging Stakeholders in the Evaluation
• Staff members
• Community members
• Other relevant stakeholders
Ensuring Organizational Readiness
• Staff hiring plan and roles
• Staff orientation to grant activities
• Communication with current grantees
Beginning Institutional Review Board (IRB) Process
• Local IRB plans
• Local IRB application process and deadlines
27
Getting Started (cont.)
Study Component
Data Sources
Who Collects
Data
Who
Responds
Timeframe for
Implementing
Local IRB
Center
Family consumers
FY 2006 and
ongoing thereafter
Involves local
IRB submission
Macro
Family consumers
FY 2006 and
ongoing thereafter
Core Clinical Characteristics Forms*
1. Descriptive and Clinical
Outcomes of Children
Receiving Direct Clinical
Mental Health Services
TSCC-A*
UCLA PTSD Index for DSM-IV*
CBCL (1.5-5; 6-18)*
2. Consumer Satisfaction
With Direct Clinical
Mental Health Services
3. Provider Knowledge and
Use of Trauma-informed
Services
4. Product/Innovation
Development and
Dissemination
YSS-F
Key informant interviews and focus
groups
Macro
TIS Survey
Centers’ quarterly progress reports
and annual progress reports
Collaborative group leader
interviews
Providers
Administrators
Macro
Case studies
5. Adoption of Methods and
Practices
GAAS
Macro
AIFI
Network Survey
6. Network Collaboration
8. NREPP
Collaborative group
leaders
Network staff, nonNetwork staff
Administrators,
providers, evaluators
Administrators,
providers
Administrators
Macro
CTPT survey
7. National Impact of the
NCTSI
Center-defined
experts
National Impact Survey
Macro
Incorporated into regular Network
reporting mechanisms
Macro
Collaborative group
members
Professional
organization
members
Not applicable
FY 2006 and early
FY 2007
FY 2008 and
ongoing thereafter
FY 2006 and
ongoing thereafter
FY 2006 and FY 2008
FY 2007 and FY 2009
Depends
somewhat on
local IRB;
however, likely
no implications
FY 2006 and
annually thereafter
FY 2008 and
annually thereafter
FY 2006 and FY 2008
FY 2007 and FY 2009
FY 2006 and
annually thereafter
No implications
Ongoing monitoring
process
No implications
28
Contact Information
• If you have questions or comments or need more
information, feel free to contact the following
members of the CSE team:
–Christine Walrath, Officer in Charge
E-mail: [email protected]
–John Gilford, Project Director
E-mail: [email protected]
–Bhuvana Sukumar, Deputy Project Director
E-mail:
[email protected]
29
Center for Mental Health Services
(CMHS)
and
Performance Management
NCTSI
CMHS Transformation
Accountability (TRAC)
Agenda
• Performance management and CMHS
• TRAC and NCTSI
• Next steps
CMHS Performance Management
NCTSI
Eval
CMHS
TRAC
National Outcome
Measures (NOMs)
PART Review
GPRA
SAMHSA
Federal/4 yrs
Federal/Annual
TRAC Overview
CMHS-TRAC is a Web-based, centralized,
data platform that collects and reports
outcome measures on CMHS programs for
the Government Performance and Results
Act (GPRA).
TRAC: A New Approach to
Accountability
• TRAC is a strategic imperative for CMHS
• Driven by:
– Government-wide requirements
– SAMHSA data strategy
– Center commitment to performance management
• Outcome measures targeted at:
– Consumer services
– Technical assistance
– Infrastructure development
– Prevention
SAMHSA-wide Domains:
NOMs
SAMHSA is obtaining OMB approval to use NOMs for GPRA
reporting purposes:
–
–
–
–
–
–
–
–
–
–
Functioning
Stability in Housing
Employment and Education
Crime and Criminal Justice Status
Perception of Care
Social Connectedness
Access/Capacity
Retention
*Cost-effectiveness
*Use of Evidence-based Practices
*Indicates that measures have not yet been developed for these domains
CMHS GPRA Data Collection:
Current Status
 Most programs were collecting GPRA data
in some form
 Lack of standardized outcome data:
– Limits the ability to demonstrate what CMHS
has achieved
– Creates difficulties in aggregating data to
summarize program performance
 NOMs tool is now the standard data
collection method for all service programs
TRAC and NCTSI
Service Treatment Grants
• NOMs tool used to collect interviews on all consumers
enrolled in services:
– Paper NOMs tool/electronic NOMs tool
• Data collection points—baseline, reassessment, and
discharge
• Reassessment is collected every 3 months while consumer
is receiving services
• No followup after discharge or if consumer is lost to
contact for 90 days or more
What Can TRAC Do for You?
 Reports:
– Enrollment and reassessment rates
– NOMs Outcome Report
– Cross Tabs and Frequencies
 Data download:
– Grantees can download data through
CMHS-TRAC system and run analysis
How Can Grantees Prepare?
• Visit the TRAC Web site for more information at
https://www.samhsa-gpra.samhsa.gov/home/ index.htm
(Click CMHS TRAC  General Information  Service
Programs  Related Links )
• Contact TRAC Help Desk with your questions at
888-219-0238 or [email protected]
• If necessary, seek IRB approval immediately
• Determine processes for data collection
Q&A