Transcript Slide 1

Planning with Data
May 25, 2011- Louisville, KY
Jenell Holstead
Megan Weikel
Danielle Neukam
A joint venture between the Forum and High/Scope.
Workshop Objectives:
 Consider aspects of change and ways to stimulate a
successful change initiative.
 Read and interpret program quality data.
 Create an effective improvement plan for your program,
based on data.
 Develop strategies for taking the plan back to your
organization.
© David P. Weikart Center for Youth Program Quality
2
Youth Program Quality Intervention (YPQI)
ASSESS
PLAN
IMPROVE
Create
improvement plan
Carry out
improvement plan
Conduct SA
(Local Self-Assessment Team)
Conduct EA
(External assessor)
(Local Self-Assessment Team)
(repeat)
(Program Director, site
managers, program staff)
Center Profile
(CEEP)
© David P. Weikart Center for Youth Program Quality
3
Workshop Agenda:
9:15-10:00
The change process and leading
change
10:00-10:15
BREAK
10:15-11:30
Review aggregate and site-level data
11:30-1:00
LUNCH
1:00-2:00
Developing Program Improvement
Plans
2:00-2:30
Implementation of Plans / Reporting
2:30
Closing/Evaluations
© David P. Weikart Center for Youth Program Quality
4
Activity:
Leading Change
A joint venture between the Forum and High/Scope.
Stages of Change:
Where are you?
Termination
Maintenance
Action
Preparation
Contemplation
Precontemplation
© David P. Weikart Center for Youth Program Quality
6
6
Where does change happen?
What does this have to do with Program Quality?
Kentucky Department of
Education
Policy
Context
District Coordinators
Organizational
Setting
Program Managers/Staff
Action
Instructional
Setting
Preparation
Youth
© David P. Weikart Center for Youth Program Quality
Contemplation
7
Break
A joint venture between the Forum and High/Scope.
Reading the Data
1. Self Assessments
2. External Assessments
3. Site Visits (CEEP)
4. Center Profiles
A joint venture between the Forum and High/Scope.
1. Self Assessment Report
All sites that entered YPQA results into Scores Reporter
will have self assessment reports. If you did both a pink YPQA and a
purple School-Age PQA, you will have both results.
Samplple Self Assessment
Samplple Self Assessment
Detailed Report
All Sites Self N=29
Item/Scale/Domain
5.00
I. Safe Environment
4.00
I-E2
A. Psychological and emotional safety is promoted.
1. Emotional Climate
2. Mutual respect (religion, ethnicity,etc.)
B. The physical environment is safe and free of health hazards.
1. Health and safety
2. Sanitation
3. Ventilation and lighting
4. Temperature
C. Appropriate emergency procedures and supplies are present.
1. Emergency procedures
2. Fire extinguisher
3. First aid kit
4. Other safety equipment
5. Supervised entrances
6. Supervised access to outdoor space
D. Program space and furniture accommodate the activities offered.
1. Sufficient space
2. Suitable space
3. Furniture
4. Appropriate furniture
E. Healthy food and drinks are provided.
1. Drinking water
2. Available food and drinks
I-E3
3. Healthy food and drinks
I-A1
I-A2
3.00
I-B1
I-B2
I-B3
2.00
I-B4
I-C1
I-C2
1.00
I-C3
I-C4
I-C5
0.00
I. Safe Environment
II. Supportive
Environment
III. Interaction
I-C6
IV. Engagement
I-D1
Item/Scale/Domain
I. Safe Environment
A. Psychological and emotional safety is promoted.
B. The physical environment is safe and free of health hazards.
C. Appropriate emergency procedures and supplies are present.
D. Program space and furniture accommodate the activities offered.
E. Healthy food and drinks are provided.
II. Supportive Environment
F. Staff provide a welcoming atmosphere.
G. Session flow is planned, presented, and paced for Children.
H. Activities support active engagement.
I. Staff support Children in building new skills.
J. Staff support Children with encouragement.
K. The physical environment is flexible and child-centered
L. Materials are plentiful, appropriate and accessible
III. Interaction
M. Children have opportunities to develop a sense of belonging.
N Adults engage with children in a positive way
O. Children have opportunities to practice leadership skills
P. Adults encourage children to manage feelings and resolve conflicts appropriately
IV. Engagement
Q. Children have opportunities to set goals and make plans
R. Children have opportunities to make choices based on their interests
S. Children are encouraged to take responsibility
T. Children have opportunities to reflect.
Total Scores
© David P. Weikart Center for Youth Program Quality
All Sites Self
N=29
I-D2
4.69
I-D4
4.80
4.80
4.30
4.76
4.74
I-D3
I-E1
II. Supportive Environment
4.15
4.65
4.53
4.02
4.43
3.61
3.55
2.60
II-F1
4.16
II-G4
4.05
4.70
3.57
4.29
II-F2
II-F3
II-G1
II-G2
II-G3
II-G5
II-H1
II-H2
II-H3
3.68
2.68
3.88
4.61
3.43
4.17
II-I1
II-I2
II-J1
II-J2
F. Staff provide a welcoming atmosphere.
1. Staff greet Children
2. Staff tone of voice and language
3. Staff smile, use friendly gestures, make eye contact
G. Session flow is planned, presented, and paced for Children.
1. Start and end on time
2. Materials and supplies ready
3. Enough materials and supplies for all children
4. Staff explain activities clearly
5. Appropriate time for activities
H. Activities support active engagement.
1. Children engage with materials or ideas
2. Children talk about what they are doing
3. Connections with prior experiences
I. Staff support Children in building new skills.
1. Children encouraged to try new skills
2. Mistakes allowed
J. Staff support children with encouragement.
1. Staff use specific, nonevaluative language
2. Open-ended questions
All Sites Self
N=29
4.69
4.80
4.60
5.00
4.80
4.77
4.73
4.89
4.81
4.30
4.43
4.07
3.53
4.81
4.70
4.96
4.76
4.83
4.62
4.79
4.79
4.74
4.79
5.00
4.57
4.15
4.65
4.40
4.75
4.80
4.53
4.62
4.52
4.69
4.65
4.26
4.02
4.53
3.84
3.65
4.43
4.30
4.60
3.61
3.87
3.39
10
YPQA: The Pyramid of Program Quality
Plan
Make choices
Engagement
Reflect
Lead and mentor
Be in small groups
Partner with adults
Experience belonging
Interaction
Encouragement Reframing conflict
Supportive
Skill building
Session flow
Active engagement Welcoming atmosphere
Environment
Psychological and emotional safety
Program space and furniture Emergency procedures
Safe
Healthy food and drinks
Physically safe environment
Environment
Youth Voice and Governance
Professional Learning Community
11
© David P. Weikart Center for Youth Program Quality
11
Self Assessment Data
 Each site has self-assessment results from data entered
into Scores Reporter
Keep in mind…
 Observation scores represent a snapshot – this has
limitations and value.
 These are aggregate scores from multiple observations.
 The overall story is more important than the individual
numbers.
 What you do with the data matters most.
© David P. Weikart Center for Youth Program Quality
12
Self Assessment: Aggregate Data
Younger Youth (N=69)
Older Youth (N=63)
5.00
4.00
3.00
2.00
1.00
I. Safe Environment
II. Supportive Environment
© David P. Weikart Center for Youth Program Quality
III. Interaction
IV. Engagement
13
2. External Assessment Report
Cycle Seven sites that were part of the YPQA Process in 2010 will have External
Assessment results (which appear along-side self-assessment results).
© David P. Weikart Center for Youth Program Quality
14
External Assessment Data
 External assessments were conducted for 17 sites
during the spring 2011 site visits.
Keep in mind…
 External assessment scores are always lower than self
assessment scores.
 Observation scores represent a snapshot – this has
limitations and value.
 The overall story is more important than the individual
numbers.
 What you do with the data matters most.
© David P. Weikart Center for Youth Program Quality
15
3.Site Visit Reports (CEEP)
37 visits to KY 21st CCLC programs between February 16 and April 13
Site Visit Activities Included:
1.
2.
3.
Site coordinator interview
School day teacher interview
Standardized observation protocol for academic and enrichment activities
Rating System:
•
•
12 Items (rated on a scale of 1 to 4)
 1 = Must Address and Improve
 2 = Some Progress Made
 3 = Satisfactory
 4 = Excellent
48 possible points
© David P. Weikart Center for Youth Program Quality
16
Site Visits (CEEP)
Purpose of 2011 Site Visits
Elementary/Middle
School Programs
High
School Programs
1. Activities geared toward
rigorous academic enrichment
1. Activities promote academic
growth, remediation, and
development
2. Links to the regular school day
2. Links to the regular school day
3. Individual support and
opportunities for positive
interactions for youth
3. Participants contribute ideas,
make choices, and having
positive experiences
4. Relationships with schools,
parents, and other community
constituents
4. Establish partnerships and
employ successful recruitment
strategies
© David P. Weikart Center for Youth Program Quality
17
Elementary and Middle School Site Visit Results
Focus Area 1 & 2:
Activities Geared Toward Rigorous Academic Achievement
Links to the School Day
Homework Help
Focus
Area
1
3.0
Supplemental Academic Enrichment
2.9
Active Learning
3.0
School Day Curriculum
Focus
Area
2
2.7
School Personnel Involved
3.6
Activities Tailored to Student Needs
2.9
1
© David P. Weikart Center for Youth Program Quality
1.5
2
2.5
3
3.5
4
18
Elementary and Middle School Site Visit Results
Focus Area 3 & 4:
Opportunities for Individual Support and Positive Interactions
Relationships with Schools, Parents, and Community Organizations
Individual Support
Focus
Area
3
3.1
Positive Interactions with Adults
3.4
Positive Interactions with Peers
3.0
Well-Integrated with School
Focus
Area
4
3.6
Relationships with Parents
3.4
Partners with CBOs
3.8
1
© David P. Weikart Center for Youth Program Quality
1.5
2
2.5
3
3.5
4
19
High School Site Visit Results
Types of Activities Offered
Activity
Percent of Sites
Offering Activity
Homework Help/Tutoring
86%
Credit Recovery
14%
Goal Setting/Career Development
100%
© David P. Weikart Center for Youth Program Quality
20
High School Site Visit Results
Focus Area 1 & 2:
Activities Promote Academic Growth, Remediation, and Development
Links to the School Day
Homework Help/Tutoring
Focus
Area
1
3.2
Credit Recovery
3.0
Goal Setting/Career Development
3.1
School Day Curriculum
Focus
Area
2
3.0
School Personnel Involved
3.9
Activities Tailored to Student Needs
3.0
1
© David P. Weikart Center for Youth Program Quality
1.5
2
2.5
3
3.5
4
21
High School Site Visit Results
Focus Area 3 & 4:
Participants Contribute Ideas, Make Choices, and Have Positive Experiences
Establishes Partnerships and Maintains Successful Recruitment Strategies
Interest Based Choices
Focus
Area
3
Focus
Area
4
3.4
Youth Engaged in Program Decisions
3.3
Opportunities for Positive Interactions
3.3
Intentional Recruitment and Retention
3.3
Well-Integrated with School
3.7
Partners with Parents and CBOs
3.9
1
© David P. Weikart Center for Youth Program Quality
1.5
2
2.5
3
3.5
4
22
4. Center Profiles
 Each site has a Center Profile from the 2009-2010
School Year (that offered programming that year)
 Data elements included in analyses
 Student participation
 Outcomes for regular attendees
 Grades
 Teacher Surveys
 Outcomes for regular attendees who
struggle academically
 Grades
© David P. Weikart Center for Youth Program Quality
23
2009-2010 Center Profile Data: Student Participation
KY
Elementary
Program Sites
KY
Middle/High
Program Sites
12,986
8,930
% of “Regular Participants” (30+ Days)
41%
18%
% of Students Attending 30-59 Days
21%
12%
% of Students Attending 60 or More
Days
20%
6%
% of Regular Participants Eligible for
Free/Reduced Lunch
63%
54%
% of Regular Participants Eligible for
Special Education Services
12%
13%
Data Indicators
Student Participation
Total # of Student Participants
© David P. Weikart Center for Youth Program Quality
24
2009-2010 Center Profile Data: Academic Outcomes
KY
Elementary
Program Sites
KY
Middle/High
Program Sites
% of students who achieved the highest grade
possible in English
16%
12%
% of students who increased their English grade
28%
36%
% of students who decreased their English grade
16%
22%
% of students who achieved the highest grade
possible in math
15%
10%
% of students who increased their math grade
30%
33%
% of students who decreased their math grade
14%
24%
% of students who improved homework completion
84%
89%
% of students who improved their classroom
participation
81%
77%
% of students who improved academically
84%
81%
Data Indicators
Program Outcomes for Regular Participants
© David P. Weikart Center for Youth Program Quality
25
2009-2010 Center Profile Data:
Academic Outcomes for Struggling Students
Data Indicators
KY
Elementary
Program Sites
KY
Middle/High
Program Sites
Program outcomes for students earning a C or lower in the fall:
% of students who increased their
reading grade
57%
69%
% of students who decreased their
reading grade
10%
13%
% of students who increased their math
grade
60%
65%
% of students who decreased their
math grade
10%
13%
© David P. Weikart Center for Youth Program Quality
26
We have plenty of data…what do we do with them?
•
Step 1:
Identify Program
Successes
Step 2:
Identify Areas for
Improvement
•
•
•
•
•
•
•
•
•
© David P. Weikart Center for Youth Program Quality
Program
activities/operations?
Safe environment?
Links to the school day?
Attendance patterns?
Classroom behavior?
Program behavior?
Grade Changes? Reading or
math?
Relationships among youth?
Relationships among
youth/adults?
Engagement with program?
27
Identifying Successes
and Challenges
Using the worksheet provided in your folder, determine
your program’s strengths and weaknesses based on the
data provided.
A joint venture between the Forum and High/Scope.
Reviewing Program
Successes and Challenges
 Were you able to identify program successes?
 Were you able to identify program challenges?
 Were data consistent across data elements?
A joint venture between the Forum and High/Scope.
Putting it all together:
Create the story of your data…
 What is the message or story of your data? What do the
numbers tell you?
 What’s missing from the data? What important things about
program quality do not come through?
 Where are the gaps between what you want to provide and
what the data says you’re providing?
© David P. Weikart Center for Youth Program Quality
30
Anybody hungry??
Break for Lunch:
11:30 am – 1:00 pm
A joint venture between the Forum and High/Scope.
Completing the Program
Improvement Plan
Two copies of the Program Improvement Plan template
have been included in your folder. Additional copies are
available if you need them.
A joint venture between the Forum and High/Scope.
Completing the Improvement Plan
 Step 1: Enter District Name and Program Site
 Step 2: Using the data from your folder and worksheet
activity, develop one goal
 Justify this goal by listing the related data element(s) that
identified this area as needing to be strengthened.
District:
Program Site:
GOAL 1:
Related YPQA
Item(s) and/or
Data Element(s)
© David P. Weikart Center for Youth Program Quality
33
Completing the Improvement Plan: GOALS
 When developing goals, remember:
 Goals should be broad statements…but not too broad!
 Examples of good goals:
 Purposefully connect the afterschool program to the school day.
 Provide opportunities for youth to reflect on their experiences in the
after school program.
 Provide activities geared towards improving reading skills
 Examples of not so good goals:
 Improve academic performance
 Increase student engagement
© David P. Weikart Center for Youth Program Quality
34
Completing the Improvement Plan: OBJECTIVES
 Step 3: Develop objectives for the first goal
Objective 1:
Objective 2:
© David P. Weikart Center for Youth Program Quality
S M A R T
S M A R T
WHEN will
progress be
measured?
WHAT will be
done to
measure
progress?
WHEN will
progress be
measured?
WHAT will be
done to
measure
progress?
35
Program Improvement Plan objectives
should each be SMART:
 Specific
 Measurable
 Attainable
 Relevant
 Timelined
© David P. Weikart Center for Youth Program Quality
36
SMART Objectives…are SPECIFIC
Not very specific…
 Youth will have more opportunities to enhance skills in reading
and math.
Getting better…
 Youth will engage in literacy-based activities during each week.
Even better…
 Literacy-based activities will be provided to students on a
minimum of three days per week.
Ideally, you have this level of specificity…
 By November 2011, literacy-based activities will be provided to
students on a minimum of three days per week.
© David P. Weikart Center for Youth Program Quality
37
SMART Objectives…are Measurable
Which of the following objectives is most measureable?
1. Staff will use student achievement data to plan topics for
tutoring sessions.
2. By May 2012, staff will hold at least three quarterly review
sessions with teachers to discuss student achievement data and
plan tutoring topics.
3. At least half of tutoring sessions provided will be based on
individual student needs
4. Staff members will increase their awareness of students’
individual academic needs.
© David P. Weikart Center for Youth Program Quality
38
SMART Objectives…are Attainable
Which of the following objectives is most likely to be attained?
1. All program activities will be planned and implemented with
student input.
2. By May 2012, all program activities will involve opportunities for
students to share their work with other participants.
3. All students will have opportunities for reflection during program
activities on a daily basis.
4. By May 2012, students will have opportunities for reflection
during at least two program activities per week.
© David P. Weikart Center for Youth Program Quality
39
SMART Objectives…are Relevant
Which of the following objectives are most relevant to the following goal?
GOAL:
Increase the proportion of program activities provided that are
directly aligned with academic standards.
OBJECTIVES:
1. By May 2012, all program activities will involve opportunities for
students to share their work with other participants.
2. By November 2011, literacy-based activities will be provided to
students on a minimum of three days per week.
3. By May 2012, students will participate in at least four activities per
week (outside of homework help) that are intentionally linked to
Kentucky state standards.
© David P. Weikart Center for Youth Program Quality
40
SMART Objectives…are Timelined
Which timeline seems most useful?
1. By spring 2012, all program activities will involve opportunities
for students to share their work with other participants.
2. Next year, staff will hold quarterly review sessions with teachers
to discuss student achievement data and plan tutoring topics.
3. Beginning in October 2011, staff will hold quarterly review
sessions with teachers to discuss student achievement data and
plan tutoring topics.
4. From December 2011 through April 2012, students will have
weekly opportunities to reflect on program activities.
© David P. Weikart Center for Youth Program Quality
41
Completing the Improvement Plan: OBJECTIVES
 Step 4: Double-check each objective to ensure it meets
the criteria for SMART objectives.
Objective 1:
Objective 2:
© David P. Weikart Center for Youth Program Quality
S M A R T
S M A R T
WHEN will
progress be
measured?
WHAT will be
done to
measure
progress?
WHEN will
progress be
measured?
WHAT will be
done to
measure
progress?
42
Completing the Improvement Plan: MEASURING PROGRESS
 Step 5: Indicate when progress will be measured and
what will be done to measure progress
Objective 1:
Objective 2:
© David P. Weikart Center for Youth Program Quality
S M A R T
S M A R T
WHEN will
progress be
measured?
WHAT will be
done to
measure
progress?
WHEN will
progress be
measured?
WHAT will be
done to
measure
progress?
43
Completing the Improvement Plan: ACTIVITIES
 Step 5: Copy each objective to the chart on page 2
 Step 6: List 3 activities that will be conducted in order to
meet the objectives
 Activities should be specific and include timelines
Objective 1:
Objective 2:
1.
1.
2.
2.
3.
3.
© David P. Weikart Center for Youth Program Quality
44
Completing the Improvement Plan
 Step 7: Rinse and Repeat!
 Follow the same steps for Goal 2 on pages 3 and 4
 Step 8: Type Program Improvement Plan using the
electronic copy we’ll email you. Email to Megan Weikel
at [email protected] by June 10, 2012.
 Step 9: Implement the plan!!
© David P. Weikart Center for Youth Program Quality
45
Complete Program
Improvement Plan:
Using the program improvement plan templates
provided, brainstorm, write two goals with related
objectives, and determine needed activities.
A joint venture between the Forum and High/Scope.
Taking it back
 What’s your plan for taking back your plan?
 How will you get to the POS (point of service)?
 How does your plan address your team needs?
© David P. Weikart Center for Youth Program Quality
47
Accountability Behaviors
 Resistance - You don’t understand me or the kids I work with.
You can’t make me do it!
 Survival - This is just the latest fad and it too will pass.
If I keep to myself, no one will call me out.
 Compliance - We will do the bare minimum to get through
licensing, then it’s business as usual.
 Engagement - I see the value of this change, and I’m
committed to taking advantage of this opportunity.
© David P. Weikart Center for Youth Program Quality
48
Next Steps…
 6/10: All Self Assessment Teams complete a Program
Improvement Plan and send it to CEEP
 6/15: CEEP will send out 2011 YPQA Self Assessment
Evaluation Survey and Program Improvement Plan form
to all Self Assessment Teams
 6/30: 2010 YPQA Self Assessment Evaluation Survey
must be completed by all Self Assessment Teams
 December 2011: CEEP will send out Progress Reports
for each program site to complete – based on
implementation of the Program Improvement Plan.
© David P. Weikart Center for Youth Program Quality
49
Evaluations
A joint venture between the Forum and High/Scope.
Planning with Data
May 25, 2011- Louisville, KY
Jenell Holstead
Megan Weikel
Danielle Neukam
A joint venture between the Forum and High/Scope.