Continuous Improvement

Collecting, Analyzing, and Sharing Data

At the end of this presentation, you will know:

1) 2) 3) 4) 5) 6) All the wonderful ways that you can use data for important purposes What “formative evaluation” is, and how it relates to continuous improvement How to organize yourselves for formative evaluation How to collect data and use it to fuel continuous improvement Different approaches to analyzing, reporting on, reflecting on, and sharing your data How to work with C-PAD to accomplish data related tasks

The Joys of Data!

Data are not just for grant requirements anymore!

Continuous Improvement Positioning your Program for Funding Marketing your Program

By using data for continuous improvement, you will increase:

 Participation  Satisfaction  Tailoring  Effectiveness

Using Formative Evaluation for Continuous Improvement

What is formative evaluation?

• Evaluation = using data for assessment and learning • Formative = for the purposes of creating, developing, and improving •

Formative (program) evaluation

= Using data to inform how you can continually pursue program excellence

Formative Evaluation as a Tool for Continuous Improvement

•

Formative evaluation asks:

Bullet Points Is the program reaching its participation targets?

Are participants satisfied with the offerings?

So that your team can further investigate:

How can we attract more people to the program?

How can we increase satisfaction levels?

Does the design of services meet the needs of its participants?

How can we better align service offerings with participant needs?



In order to achieve:

 Increased participation  Increased satisfaction Increased tailoring Do participants succeed in desired outcomes?

How can we increase the rate of positive outcomes?

 Increased effectiveness

We are busy delivering a program – how can we be evaluators too?

•

Forget: “evaluation”

•

Remember: “leveraging data for program excellence!”

Story from the Field: Leveraging data for program excellence

•

Julie Seeley, Spoon River College

The Formative Evaluation Toolkit: A Guided Tour

What you will find in the FET:

• Many pages (but not to worry!) • • • • • • Step-by-step guidance through the phases of evaluation Tools and templates for conducting your evaluation Templates for working meetings to accomplish evaluation tasks Checklists at the end of each chapter to keep you on track Blank pages for notes, questions, and insights Support for: – Completing deliverables due to Champion Colleges – Fulfilling grantee reporting requirements

Step-by-step guidance for:

• Data collection (chapters 3 & 4) • Organizing for evaluation (chapter 2) • Reporting on findings (chapter 5) • Reflecting on findings (chapter 5)

Data Collection (Chapters 3 and 4)

Data Types

• Process data • Outcome data • Stakeholder feedback

•

Process Data (Chapter 3)

Counts of program participants – Students completing workforce program courses – Students participating in support services • Information about program components, describing: – Workforce programs and courses – Math, English, and computer courses

Outcome Data (Chapter 3)

• Number of students who have: – Received credits for prior learning – Received a degree – Received a certificate – Received a non-credit certificate – Become employed

Important Considerations for Collecting Process & Outcome Data

• • • • • “Extractable” v. “Real-Time” Data Make a data collection plan – Identify the data you need to collect – Figure out how you’ll collect it Work with an IR Partner Plan early for real-time data collection Stay organized with data storage

FET Tools for Process and Outcome Data

• Guidance for a working meeting to develop process and outcome data collection plans • Tools: – Tool 2: Data Choices Table (p. 42) – Tool 3: Process Data Collection Plan Worksheet (p. 45) – Tool 4: Outcome Data Collection Plan Worksheet (p. 54) – Tool 5: Options for Collecting Employment Data (p. 57) – Data Storage Excel Workbook (URL on p. 17)

Stakeholder Feedback (Chapter 4)

• Hearing from students, internal partners, and community partners • Looking for feedback that will help you to: – Attract more students to the program – Tailor services better – Raise satisfaction levels – Support students better for completion and employment

Important Considerations for Collecting Stakeholder Feedback

• Decide what you want to learn • Make a data collection plan – Decide which stakeholders you want to hear from – Identify the data collection methods you want to use – Identify people responsible for collecting feedback • Make data collection instruments • Don’t collect too much data!

FET Tools for Stakeholder Feedback

• • Guidance for two working meetings to develop: – – A stakeholder feedback data collection plan Data collection instruments Tools: – Tool 6: Stakeholder Feedback Evaluation Questions Worksheet (p. 61) – – – Tool 7: Stakeholder Feedback Data Collection Plan Worksheet (p. 63) Tools 8 & 9: Survey, Interview, and Focus Group Protocol Templates (pp. 66, 82) Tools 10-12: Guidance for Conducting Interviews and Focus Groups (pp. 88, 90, 91)

Organizing for Evaluation (Chapter 2)

Assembling an Evaluation Team

• Evaluation Team Manager • IR Partner • Process and Outcome (P&O) Data Lead • Stakeholder Feedback Lead

Convene the Team for an Evaluation Launch Meeting

• Review how the Plus 50 data tasks fit together • Customize your evaluation timeline

Reflection and Reporting (Chapter 5)

o

Analysis, reflection, and reporting as an iterative process

Preliminary reporting: pulling together data with initial analysis o o   Reflecting on findings Writing up memos: Findings/Learnings Recommendations (based on learnings and reflection)

• •

FET Tools for Reflecting on and Writing up Evaluation Findings

Guidance for working meetings to reflect: – With the Plus 50 Team on key findings from the process and outcome data – – With the Plus 50 Team on key findings from the stakeholder feedback With the Plus 50 Advisory Committee on all key evaluation findings Tools: – Tool 13: Evaluation Questions Tool (p. 92) – – Tool 14: Qualitative Data Analysis Template (p. 95) Tool 15: Memo Write-up Tool (including an memo outline) (p. 99)

Working with C-PAD to Support Continuous Improvement

Phase 5

1.

Collect Process and Outcome Data 2.

Gather Feedback from Program Participants and Partners 3.

Share Results with Key Stakeholders 4.

Plan for Program Improvement

Feeding two birds with one seed: Process and Outcome Data in C-PAD P&O data used for continuous improvement = P&O data used for grant-required reporting to AACC

Questions?