Transcript The Value of Moving From Data Collection To Thoughtful

MOHealthWINS:
A Transformative Opportunity
for Missouri
Data Collection and Performance Measures
MOHealthWINS Summit March 13-14, 2012
J. Cosgrove, Cosgrove & Associates
MOHealthWINS Vision for Missouri
Connecting target populations to employment
opportunities in the State’s growing health
care industry
MOHealthWINS’ four priorities:
– Accelerate Progress for Low-Skilled and Other Workers
– Improve Retention & Achievement Rates To Reduce
Time To Completion
– Build Programs That Meet Industry Needs, Including
Developing Career Pathways.
– Strengthen Online & Technology-Enabled Learning
Cosgrove & Associates Data Collection and Evaluation Vision:
Ensure the systematic and timely collection of data required to meet
DOL reporting and accountability requirements, and Consortium
and individual college’s research and evaluation information
needs.
– Cosgrove & Associates will partner with individual
colleges to ensure the collection and analysis of all
data required to meet U.S. Department of Labor
accountability requirements
– Cosgrove & Associates will help the Consortium and
individual colleges move from ideas and opinions to
an understanding of program effectiveness that is
precise, predictive, valid and reliable.
Survey Feedback
What You Told Us:
•
•
•
•
•
•
•
•
•
A list of EVERYTHING we need to track and report on.
We would like to see a uniform plan for collecting the data .
A plan for the uses of it after it is collected.
An organized system for sending the collected data in.
Be clear about what data we need to collect and in what format so we can determine
the extent to which we can use our ERP systems or must develop new data collection
and reporting systems.
It must satisfy not only the reporting requirements of DOL around "trainees" but also
track how well the new instructional strategies are working.
We are looking forward to having real-time data that we can analyze to support our
continuous improvement throughout the grant. We also appreciate the flexibility that
this will support as we make changes in and enhancements to our programming as
the work progresses.
Provide us the technical assistance and recommendations that will help us
continuously improve.
Support our endeavors and provide guidance.
MOHealthWINS Data Collection & Performance Measures:
What Will I Take Away From This Session?
•
Identification of DOL Primary Research and Outcome
Questions
•
Description of Data Collection, Reporting, Evaluation
Workplan and Timeline
•
Required Data Elements and Data Collection Process
•
Value of IDID
•
DOL Evaluation Framework: Participant and
Comparison Cohorts
•
You Are NOT ALONE: Next Steps and Who To Call For
Help
•
How To Fill Out Your Bracket (MIZ….ZOU) and Get
Free Drinks At The Hotel Bar!!!
Conceptual Framework For Implementation & Outcome Evaluation
Primary Questions:
•
Are individual college strategies and statewide strategies being implemented as designed in a
timely manner?
•
Are program participants, who enrolled with low academic skills improving such skills in a timely
manner? How does the progress of grant participants compare to the progress of similar
students who are not involved in the grant strategies?
•
Are grant participants being retained at the appropriate targeted rate? How does the grant
participant retention rate compare to similar students who are not involved in the grant
strategies?
•
Are grant participants completing the desired degree/certificate award in a timely manner?
How does the grant participant completion rate compare to similar students who are not
involved in the grant strategies?
•
Are grant program completers securing employment in occupations targeted by the
Consortium? How does the grant participant employment rate compare to similar students
who are not involved in the grant strategies?
•
Are employers satisfied with overall employment preparation of grant program completers?
•
Are individual colleges demonstrating the capacity to use performance tracking and program
evaluation data to continuously improve grant designated strategies and programs?
Data Collection, Reporting and Program Evaluation
Project Phases and Steps:
Research, Reporting & Evaluation Accomplished in 3 Phases
Phase 1 Timeframe: 2/20/12 to 9/15/12
– Review DOL and Consortium reporting needs and research/evaluation questions
– Examine data/information needs *****We Are Here****
– Train campus users in regard to DOL required data elements and student cohorts
– Develop statewide data system
– Pilot data collection process and tracking system
– Train campus users on data system
– Begin data collection process
Phase 2 Initial timeframe: 8/1/12 to 12/1/12, however phase 2 will be ongoing for the life of the project
– Data collection start-up and program implementation monitoring
– Participant and cohort development and description
– DOL reporting and Consortium research/evaluation information sharing
Phase 3 Timeframe: is 1/1/13 through 9/30/14
– Continuous sharing of research/evaluation information with individual colleges and the
Consortium. The goal of Phase 3 is to build the capacity of individual colleges and the Consortium
to use research/evaluation for continuous improvement processes.
TWO SIDES TO ALL DATA: THE IDID MODEL
IDID Framework:
• INQUIRE: What do we need to report and
what do we want to know?
• DISCOVER: Data collection
• INTERPRET: Which programs/strategies
worked and why did they work?
• DEVELOP ACTIONS: Based on thoughtful
interpretation of data, what actions need to
occur to ensure continuous program
improvement?
Behind Every Number Is A
Student and A Life To Be
Changed!
The Value of Moving From Data Collection To Thoughtful
Interpretation of Information
Source: Mark Milliron, President Catalytic Conversations
Data Collection Model: Statewide Integration
Multiple Points of
Participant Entry
Colleges identify
participants and
submit student
unit record data
to C&A
Data Returned To
Campuses For
Program Improvement
C&A cleans
data & builds
State data file
C&A submits State
data file to MERIC
for multiple
system matching
C&A provides
data analysis by
program
(Consortium &
campus level)
C&A completes
DOL quarterly &
annual reporting
requirements
DOL Participant &
Performance
Outcomes
MERIC returns
State file to C&A
•
Colleges must be able
to identify grant
participants by
program (credit and
non-credit)
•
The Statewide
Consortium is the
reporting unit for DOL
Performance &
Accountability
Measures
•
Data will be used at
the college level for
continuous
improvement
Data Collection Process: USDOL Data Requirements
I. Initial Point of Contact
•
•
•
•
•
•
•
•
•
•
•
Social Security Number
Name and Contact Information
U.S. Citizenship
Employment Status At Initial Enrollment
Wages At Initial Enrollment
Gender
Ethnicity/Race
Age
Disability Status
Veteran Status
TAA Eligible
II. College Student System (Credit and Non-Credit)
•
Social Security Number
•
TAA Eligible and Grant Participant
•
Name and Contact Information
•
U.S. Citizenship
•
Employment Status At Initial Enrollment
•
Wages At Initial Enrollment
•
Gender
•
Ethnicity/Race
•
Age
•
Disability Status
•
Veteran Status
•
Pell Eligibility
•
Student Status (full or part-time)
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Highest Education Level Completed Upon Entry
Educational Goal (degree or certificate)
Basic Skill Deficiency Reading-Assessment Score
Developmental Reading Level---One Level Below
College Level, Two Levels Below College Level, or
Three or More Levels Below College Level.
Basic Skill Deficiency English-Assessment Score
Developmental English Level---One Level Below
College Level, Two Levels Below College Level, or
Three or More Levels Below College Level.
Basic Skill Deficiency Mathematics-Assessment Score
Developmental Mathematics Level---One Level Below
College Level, Two Levels Below College Level, or
Three or More Levels Below College Level.
Campus Code
Program Code
Term Code and Start Date
Credit or Non-Credit Code
Entering Student Status
Term Credit Hours Attempted
Term GPA
Term Credit Hours Completed
Data Collection Process: USDOL Data Requirements Continued
III. Tracking and Performance Outcome
Data
• Each college will track
implementation of programs and
related strategies
• Developmental Skill Improvement
(course grades, re-test on
assessment instrument, etc.)
• Term to term retention
• Program (degree or certificate
completion)
• Non-Credit to Credit program
movement
• Transfer or Continuing Education
upon program completion
• Employment Status upon program
completion (1st, 2nd, and 3rd quarters)
• Wage data upon program completion
(1st, 2nd and 3rd quarters)
IV. Action
Use Data &
Thoughtful
Interpretation For
Continuous
Improvement
Implementation progress and
performance measures will be
shared with the Consortium and
colleges on a quarterly and
annual basis.
Based on thoughtful
interpretation of data, colleges
will undertake steps to ensure
continuous program
improvement?
USDOL Evaluation Framework: Participant & Comparison Cohorts
GRANT PARTICIPANT
UNIVERSE:
NON-TAACCCT-FUNDED
UNIVERSE
Participant
Cohort---Subset
of Total Grant
Participant
Universe :
Comparison Cohort--Subset of Non-Grant
Participant Universe :
Matched
Participant Cohort and Comparison Cohorts Are
Matched On Key Demographic Variables---Age and
Gender At A Minimum
Why is DOL Requiring Participant and Comparison Cohorts?
• Meets the goal of “Continuous Improvement”
• DOL hopes to learn from the cohort information:
– Did the TAACCCT program design/ updates/changes
that were proposed and implemented have a
positive effect on students who went through the
new/updated/revised program, as compared with
students who did not?
• Education retention and completion
• Job placement, retention and earnings
Participant Cohort To Comparison Cohort Analysis Will Focus
On Key Tracking and Outcome Variables
•
Program Completion
•
Retained in Program
•
Retained in Other Program
•
Credit Hours Completed
•
Developmental Skill Improvement
•
Earned Credentials
•
Further Education After Graduation
•
Employment After Graduation
•
Employment Retention
•
Earnings
Defining and Building Participant & Comparison Cohorts
•
Remember, the DOL reporting unit is the Consortium
•
Each program of study should have its own Participant Cohort and be matched
to an appropriate Comparison Cohort (non-grant participants).
•
Programs of study may be combined (resulting in only one Participant Cohort)
if occupational outlook and/or educational requirements are similar.
•
Combining programs may be a way to meet the matching requirements for a
Comparison Cohort or avoid some of the problems with having an invalid
Comparison Cohort.
•
The number of students in the Comparison Cohort must be the same as the
number of students in the Participant Cohort.
•
College student systems will be used to capture demographic and educational
characteristics for the Participant and Comparison Cohorts.
YIKES….THERE IS MORE TO THIS THAN I IMAGINED!!!!
• DON’T FREAK OUT!
• REMEMBER A SIGNIFICANT PORTION
OF THIS GRANT FOCUSES ON THE USE
OF DATA AND INFORMATION TO
CONTINUOUSLY IMPROVE OUR
PROGRAMS AND STRATEGIES!
• WE ARE HERE TO HELP YOU AND WILL
BE WORKING WITH EACH COLLEGE TO
BUILD DATA SETS AND WHEN
APPROPRIATE PARTICIPANT AND
COMPARISION COHORTS.
Next Steps: Campus Visits, Statewide Meetings,
Database Construction, and Initial Data Entry
Phase 1 Timeframe: 2/20/12 to 9/15/12
•
Review DOL and Consortium reporting needs and research/evaluation questions
 Meet with individual campus MOHealthWINS teams.
 Develop Statewide Data Collection Task Force. Task Force will examine data/information
needs and Campus and Agency steps needed to provide required data.
 RESPONSIBILITY: Cosgrove & Associates, Campus Teams, and Task Force Representatives
•
Train campus users in regard to DOL required data elements, and participant and
comparison cohorts.
 RESPONSIBILITY: Cosgrove & Associates
•
Develop statewide data system and data element dictionary.
 RESPONSIBILITY: Cosgrove & Associates and State Agency Data Partners
•
Pilot data collection process and tracking system with three campuses.
 RESPONSIBILITY: Cosgrove & Associates, State Agency Data Partners and Pilot Institutions
•
Train campus users on data system and outline data reporting deadlines.
 RESPONSIBILITY: Cosgrove & Associates
•
Begin data collection process.
 RESPONSIBILITY: Individual Campuses
Einstein and Berra: Final Thoughts On
Data Collection, Evaluation and Program Improvement
• Data Don’t Drive.
• The important thing is not to stop
questioning.
• Data collection and research are 90%
mental and the other half is physical.
• I wish I had an answer for that
because I am tired of answering that
question.
• Finally, you don’t make the pig fatter
by weighing it everyday.
Questions and Contact Information
Ensure the systematic and timely collection of data required to meet
DOL reporting and accountability requirements, and Consortium and
individual college’s research and evaluation information needs.
Contact Information
John Cosgrove
[email protected]
314.913.6159
Maggie Cosgrove
[email protected]
314.610.2799