Transcript Slide 1

planning
with data
May 8, 2013:
Emily Rouge,
LeeAnn Sell, & Stephanie Schmalensee
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Workshop Objectives
 Consider aspects of change and ways to stimulate a
successful change initiative.
 Read and interpret program quality data
 Outline the strengths and weaknesses of your program
 Create an effective improvement plan for their
organization, based on data.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Agenda
9:15-10:00
The change process and leading
change
10:00-10:15
BREAK
10:15-11:30
Review aggregate and site-level data
11:30-1:00
LUNCH
1:00-2:00
Developing Program Improvement
Plans
2:00-2:30
What’s Next in the QIS / Reporting
2:30
Closing/Evaluations
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
No one Said Change was easy!
“Stepping onto a brand-new path is difficult, but not
more difficult than remaining in a situation, which is
not nurturing to the whole woman.”
― Maya Angelou
“I alone cannot change the world,
but I can cast a stone across the
waters to create many ripples.”
― Mother Teresa
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Change Paired Activity
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Resistance to Change – Table group
activity
 Discuss your concerns about making changes to
your program with others at your table
 How do YOU feel about making changes in your
program?
 What kind of resistance do you anticipate?
 What can you do to counter it?
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Youth Program Quality Intervention (YPQI)
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Where does change happen?
What does this have to do with Program Quality?
Policy
Context
Kentucky
Department of
Education
Program Directors
Organizational
Setting
Instructional
Setting
Site Managers/Staff
Youth
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Break
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Data Folders:
1.
2.
3.
4.
Center Profiles
PQA Self assessments
Site Visits (CEEP) – Cycle 9 Grantees
External Assessments (Some Cycle 9)
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
During this section of the session,
you will:
 Review your program data – in comparison
to statewide data
 Discuss your site’s strengths and
challenges
 Observe the picture your data paints
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
1. Center Profiles
 Each site has a Center Profile from the 2011-2012
School Year (that offered programming that year)
 Data elements included in analyses
 Student participation
 Outcomes for regular attendees
 Grades
 Federal Teacher Surveys
 Outcomes for regular attendees who
struggle academically
 Grades
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
2011-2012 Center Profile Data:
Student Participation
KY
Elementary
Program Sites
KY
Middle/High
Program Sites
16,708
15,579
% of “Regular Participants” (30+ Days)
42%
20%
% of Students Attending 30-59 Days
20%
13%
% of Students Attending 60 or More
Days
22%
7%
% of Regular Participants Eligible for
Free/Reduced Lunch
73%
60%
% of Regular Participants Eligible for
Special Education Services
15%
12%
Data Indicators
Student Participation
Total # of Student Participants
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
2011-2012 Center Profile Data:
Academic Outcomes
KY
Elementary
Program Sites
KY
Middle/High
Program Sites
% of students who achieved the highest grade
possible in reading
19%
12%
% of students who increased their reading grade
36%
41%
% of students who decreased their reading grade
19%
31%
% of students who had no change in their
reading grade
26%
16%
% of students who achieved the highest grade
possible in math
18%
10%
% of students who increased their math grade
36%
43%
% of students who decreased their math grade
19%
29%
% of students who had no change in their math
grade
28%
18%
Data Indicators
Program Outcomes for Regular Participants
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
2011-2012 Center Profile Data:
Federal Teacher Survey Outcomes
KY
Elementary
Program Sites
KY
Middle/High
Program Sites
% of students who improved
homework completion
81%
80%
% of students who improved their
classroom participation
78%
77%
% of students who improved
academically
81%
78%
Data Indicators
Program Outcomes for Regular Participants
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
2011-2012 Center Profile Data:
Academic Outcomes for Struggling Students
Data Indicators
KY
Elementary
Program Sites
KY
Middle/High
Program Sites
Program Outcomes for Students Earning a C or Lower in the Fall
% of students who increased their reading
grade
52%
60%
% of students who decreased their reading
grade
10%
18%
% of students who had no change in their
reading grade
37%
22%
% of students who increased their math
grade
53%
60%
% of students who decreased their math
grade
11%
17%
% of students who had no change in their
math grade
36%
23%
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
2. Self Assessment Report
All sites that entered YPQA results into Scores Reporter
will have self assessment reports. If you did both a Youth PQA
and a School-Age PQA, you will have both results.
Samplple Self Assessment
Samplple Self Assessment
Detailed Report
All Sites Self N=29
Item/Scale/Domain
5.00
I. Safe Environment
4.00
I-E2
A. Psychological and emotional safety is promoted.
1. Emotional Climate
2. Mutual respect (religion, ethnicity,etc.)
B. The physical environment is safe and free of health hazards.
1. Health and safety
2. Sanitation
3. Ventilation and lighting
4. Temperature
C. Appropriate emergency procedures and supplies are present.
1. Emergency procedures
2. Fire extinguisher
3. First aid kit
4. Other safety equipment
5. Supervised entrances
6. Supervised access to outdoor space
D. Program space and furniture accommodate the activities offered.
1. Sufficient space
2. Suitable space
3. Furniture
4. Appropriate furniture
E. Healthy food and drinks are provided.
1. Drinking water
2. Available food and drinks
I-E3
3. Healthy food and drinks
I-A1
I-A2
3.00
I-B1
I-B2
I-B3
2.00
I-B4
I-C1
I-C2
1.00
I-C3
I-C4
I-C5
0.00
I. Safe Environment
II. Supportive
Environment
III. Interaction
I-C6
IV. Engagement
I-D1
Item/Scale/Domain
I. Safe Environment
A. Psychological and emotional safety is promoted.
B. The physical environment is safe and free of health hazards.
C. Appropriate emergency procedures and supplies are present.
D. Program space and furniture accommodate the activities offered.
E. Healthy food and drinks are provided.
II. Supportive Environment
F. Staff provide a welcoming atmosphere.
G. Session flow is planned, presented, and paced for Children.
H. Activities support active engagement.
I. Staff support Children in building new skills.
J. Staff support Children with encouragement.
K. The physical environment is flexible and child-centered
L. Materials are plentiful, appropriate and accessible
III. Interaction
M. Children have opportunities to develop a sense of belonging.
N Adults engage with children in a positive way
O. Children have opportunities to practice leadership skills
P. Adults encourage children to manage feelings and resolve conflicts appropriately
IV. Engagement
Q. Children have opportunities to set goals and make plans
R. Children have opportunities to make choices based on their interests
S. Children are encouraged to take responsibility
T. Children have opportunities to reflect.
Total Scores
All Sites Self
N=29
I-D2
4.69
I-D4
4.80
4.80
4.30
4.76
4.74
I-D3
I-E1
II. Supportive Environment
4.15
4.65
4.53
4.02
4.43
3.61
3.55
2.60
II-F1
4.16
II-G4
4.05
4.70
3.57
4.29
II-F2
II-F3
II-G1
II-G2
II-G3
II-G5
II-H1
II-H2
II-H3
3.68
2.68
3.88
4.61
3.43
4.17
II-I1
II-I2
II-J1
II-J2
F. Staff provide a welcoming atmosphere.
1. Staff greet Children
2. Staff tone of voice and language
3. Staff smile, use friendly gestures, make eye contact
G. Session flow is planned, presented, and paced for Children.
1. Start and end on time
2. Materials and supplies ready
3. Enough materials and supplies for all children
4. Staff explain activities clearly
5. Appropriate time for activities
H. Activities support active engagement.
1. Children engage with materials or ideas
2. Children talk about what they are doing
3. Connections with prior experiences
I. Staff support Children in building new skills.
1. Children encouraged to try new skills
2. Mistakes allowed
J. Staff support children with encouragement.
1. Staff use specific, nonevaluative language
2. Open-ended questions
All Sites Self
N=29
4.69
4.80
4.60
5.00
4.80
4.77
4.73
4.89
4.81
4.30
4.43
4.07
3.53
4.81
4.70
4.96
4.76
4.83
4.62
4.79
4.79
4.74
4.79
5.00
4.57
4.15
4.65
4.40
4.75
4.80
4.53
4.62
4.52
4.69
4.65
4.26
4.02
4.53
3.84
3.65
4.43
4.30
4.60
3.61
3.87
3.39
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Quality Construct:
The Pyramid of Program Quality
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Self Assessment Data
 Each site has self-assessment results from data
entered into Scores Reporter
Keep in mind…
 Observation scores represent a snapshot – this has
limitations and value.
 These are aggregate scores from multiple
observations.
 The overall story is more important than the
individual numbers.
 What you do with the data matters most.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Self Assessment: Kentucky Aggregate Data
School Age N=82
4.7
4.8
5.0
4.5
4.5
Youth N=76
4.1
4.4
4.0
3.7
4.0
3.0
2.0
1.0
0.0
I. Safe Environment
II. Supportive
Environment
III. Interaction
IV. Engagement
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Scores of all Fives
Mr. Rogers School
5
4
3
2
1
 Not useful for
identifying areas of
improvement.
 If you did receive all
fives think about if this
is true in every activity
and with every staff
member?
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Extreme Variation
Sesame Street Academy
5
4
3
2
1
 Might signify a
misunderstanding of
what the tool is
measuring.
 The descriptors are
very specific as far as
what behaviors or
evidence to look for.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Self-Assessments:
Becoming More Objective
ABC Elementary
5
4
3
2
1
 There is typically a trend
towards this distribution
once raters are more
comfortable with the
process and the tool.
 Program self assessment
scores may be lower in
subsequent years because
raters are being more
honest or more critical as
they develop their
reliability.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
3.Site Visit Reports (CEEP)
28 visits to KY 21st CCLC programs between February 18th and
April 18th
Site Visit Activities Included:
1.
2.
3.
Site coordinator interview
School day teacher interview
Standardized observation protocol for academic
and enrichment activities
Rating System:
•
•
12 Items (rated on a scale of 1 to 4)
 1 = Must Address and Improve
 2 = Some Progress Made
 3 = Satisfactory
 4 = Excellent
48 possible points
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Site Visits (CEEP)
Purpose of 2013 Site Visits
Elementary/Middle
School Programs
1. Activities geared toward
rigorous academic enrichment
2. Links to the regular school day
3. Individual support and
opportunities for positive
interactions for youth
4. Relationships with schools,
parents, and other community
constituents
High
School Programs
1. Activities promote academic
growth, remediation, and
development
2. Links to the regular school day
3. Participants contribute ideas,
make choices, and having
positive experiences
4. Establish partnerships and
employ successful recruitment
strategies
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Elementary and Middle School Site Visit
Results
Areas of Strength
• Shares School Resources (3.65)
• Community Based Partners (3.52)
• School Personnel Involved (3.48)
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Elementary and Middle School Site Visit
Results
Potential areas for Growth
• Supplemental Academic Enrichment (3.05)
• Positive Interactions with Peers (3.10)
• Active Learning (3.10)
• Homework Help (3.10)
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
High School Site Visit Results
Types of Activities Offered (N=7)
Activity
Percent of Sites
Offering Activity
Homework Help/Tutoring
100%
Credit Recovery
71%
Goal Setting/Career Development
71%
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
High School Site Visit Results
Areas of Strength
• School Personnel Involved (3.71)
• Program is Well Integrated with the School and
Shares resources (3.71)
• Engagement with Community Based
Organizations and Parents (3.71)
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
High School Site Visit Results
Areas of Potential Growth
• Goal Setting Career Development (3.0)
• Links to the School Day (3.0)
• Intentional Recruitment and Retention
Strategies (3.0)
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
4. External Assessment Report
Some Cycle Nine sites that were part of the YPQA Process in 2012 will
have External Assessment results (which appear along-side selfassessment results).
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
External Assessment Data
 External assessments were conducted for 14 sites
during the spring 2013 site visits.
Keep in mind…
 External assessment scores are always lower than self
assessment scores.
 Observation scores represent a snapshot – this has
limitations and value.
 The overall story is more important than the individual
numbers.
 What you do with the data matters most.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Identifying Successes
and Challenges
Using the worksheet provided in your folder,
determine your program’s strengths and
weaknesses based on the data provided.
Questions to ask yourself:
What was one of your site’s biggest
successes/strengths?
What were some surprises? (Positive or Negative)
Were there obstacles that you encountered?
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Reviewing Program
Successes and
Challenges
 Were you able to identify program successes?
 Were you able to identify program challenges?
 Were data consistent across data elements?
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Putting it all together:
Create the story of your data…
 What is the message or story of your data? What do the
numbers tell you?
 In what ways is this story accurate?
 What’s missing from the data? What important things
about program quality do not come through?
 Where are the gaps between what you WANT to provide
and what the data says you ARE providing?
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Anybody
hungry??
Break for Lunch:
11:30 am – 1:00 pm
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Completing the
Program
Improvement Plan
Two copies of the Program Improvement Plan
template have been included in your folder.
Additional copies are available if you need them.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Completing the Improvement Plan
 Step 1: Enter District Name and Program Site
 Step 2: Using the data from your folder and
worksheet activity, develop one goal
 Justify this goal by listing the related data element(s) that
identified this area as needing to be strengthened.
District:
Program Site:
GOAL 1:
Related YPQA
Item(s) and/or
Data Element(s)
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Completing the Improvement Plan: GOALS
 When developing goals, remember:
 Goals should be broad statements…but not too broad!
 Examples of good goals:
 Purposefully connect the afterschool program to the school day.
 Provide opportunities for youth to reflect on their experiences in
the after school program.
 Provide activities geared towards improving reading skills
 Examples of not so good goals:
 Improve academic performance
 Increase student engagement
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Completing the Improvement Plan:
OBJECTIVES
 Step 3: Develop objectives for the first goal
Objective 1:
Objective 2:
S M A R T
S M A R T
WHEN will
progress be
measured?
WHAT will be
done to
measure
progress?
WHEN will
progress be
measured?
WHAT will be
done to
measure
progress?
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Program Improvement Plan
objectives should each be SMART:
 Specific
 Measurable
 Attainable/Action oriented
 Relevant
 Timelined
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
SMART Objectives:
 S – Specific
Who, What, When, How Much?
 M – Measurable
Can you prove it happened?
 A – Attainable/Action-Oriented
Does it use action verbs and explain what people will actually do?
 R – Realistic
Is it possible, given the program activities and circumstances?
 T – Timelined
What is the timeframe, and does it fit within the expected parameters?
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
SMART Objectives…are SPECIFIC
Not very specific…
 Struggling students will demonstrate improved math
achievement
Getting better…
 Struggling students will demonstrate improved math
achievement by increasing their math grade
Even better…
 50% of struggling students will demonstrate improved math
achievement by increasing their math grade
Ideally, you have this level of specificity…
 By May 2014, 50% of struggling students participating in 30
days or more will increase their math grade between the
first and final grade period.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
SMART Objectives…are Measurable
Which of the following objectives is most measureable?
1. Staff will use student achievement data to plan topics for
tutoring sessions.
2. By May 2014, staff will hold at least three quarterly review
sessions with teachers to discuss student achievement
data and plan tutoring topics.
3. At least half of tutoring sessions provided will be based on
individual student needs
4. Staff members will increase their awareness of students’
individual academic needs.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
SMART Objectives…are Attainable
Which of the following objectives is most likely to be
attained?
1. All program activities will be planned and implemented with
student input.
2. By May 2014, all program activities will involve
opportunities for students to share their work with other
participants.
3. All students will have opportunities for reflection during
program activities on a daily basis.
4. By May 2014, students will have opportunities for reflection
during at least two program activities per week.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
SMART Objectives…are Relevant
Which of the following objectives are most relevant to the following
goal?
GOAL:
Increase the proportion of program activities provided that are
directly aligned with academic standards.
OBJECTIVES:
1. By May 2014, all program activities will involve opportunities for
students to share their work with other participants.
2. By November 2013, literacy-based activities will be provided to
students on a minimum of three days per week.
3. By May 2014, students will participate in at least four activities
per week (outside of homework help) that are intentionally linked
to Kentucky state standards.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
SMART Objectives…are Timelined
Which timeline seems most useful?
1. By spring 2012, all program activities will involve
opportunities for students to share their work with other
participants.
2. Next year, staff will hold quarterly review sessions with
teachers to discuss student achievement data and plan
tutoring topics.
3. Beginning in October 2011, staff will hold quarterly review
sessions with teachers to discuss student achievement data
and plan tutoring topics.
4. From December 2011 through April 2012, students will have
weekly opportunities to reflect on program activities.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Completing the Improvement Plan:
OBJECTIVES
 Step 4: Double-check each objective to ensure it
meets the criteria for SMART objectives.
Objective 1:
Objective 2:
S M A R T
S M A R T
WHEN will
progress be
measured?
WHAT will be
done to
measure
progress?
WHEN will
progress be
measured?
WHAT will be
done to
measure
progress?
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Completing the Improvement Plan:
MEASURING PROGRESS
 Step 5: Indicate when progress will be measured
and what will be done to measure progress
Objective 1:
Objective 2:
S M A R T
S M A R T
WHEN will
progress be
measured?
WHAT will be
done to
measure
progress?
WHEN will
progress be
measured?
WHAT will be
done to
measure
progress?
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Completing the Improvement Plan:
Activities
 Step 5: Copy each objective to the chart on page 2
 Step 6: List 3 activities that will be conducted in
order to meet the objectives
 Activities should be specific and include timelines
Objective 1: By May 2013, lesson plans
will be created for 90% of activities and
will include links to Kentucky state
standards.
1. By September, offer staff meeting on
creating lesson plans linked to standards.
Objective 2: By May 2013, at least one
activity will be offered every week that is
connected to STEM content being taught
during the school day.
1. By September, email teachers weekly
about STEM classroom activities
2. By October, ask staff to turn in lesson
plans every week.
2. By October, meet with staff to discuss
offering STEM activities
3. By November, review all lesson plans
to make sure they address KY standards.
3. By November, collect STEM lesson
plans from staff
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Completing the Improvement Plan
 Step 7: Rinse and Repeat!
 Follow the same steps for Goal 2 on pages 3 and 4
 Step 8: Type Program Improvement Plan using the
electronic copy we’ll email you. Email to CEEP
[email protected] by May 31, 2013.
 Step 9: Implement the plan!!
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
What happens to your program
Improvement plan:
 Send it to CEEP by May 31st
[email protected]
 CEEP staff will review it and offer suggestions for
improvement
 You will receive your program improvement plan
back by June 7th for any needed revisions
 Final program improvement plan due to CEEP by
June 21st
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Complete Program
Improvement Plan:
Using the program improvement plan templates
provided, brainstorm, write two goals with related
objectives, and determine needed activities.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Taking it back
 What’s your plan for taking back your plan?
 How will you get to the POS (point of
service)?
 How does your plan address your team
needs?
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Reasons Why Change Efforts Fail






Announcing change ≠ Implementing change
Concerns are not surfaced or addressed
Lack of involvement in planning the change
No compelling reason to change
No vision and no communication
Lack of skill-building
From Leading at a Higher Level by Ken Blanchard
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Accountability Behaviors--How will you
coach staff who exhibit these behaviors?
 Resistance - You don’t understand me or the kids I work
with. You can’t make me do it!
 Survival - This is just the latest fad and it too will pass. If I
keep to myself, no one will call me out.
 Compliance - We will do the bare minimum to get
through, then it’s business as usual.
 Engagement - I see the value of this change, and I’m
committed to taking advantage of this opportunity.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Actionable Truths
 Radical, sweeping changes work better than small,
gradual changes.
 Change is personal.
 Behavior is key.
 The point of service (where youth and adults
interact) is the bottom line.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
OPPORTUNITYISNOWHERE
OPPORTUNITY IS NOW HERE
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Next Steps…
 5/31: All Self Assessment Teams complete a Program
Improvement Plan and send it to CEEP
 6/3: CEEP will send out 2013 PQA Self Assessment
Evaluation Survey
 6/18: 2013 YPQA Self Assessment Evaluation Survey must
be completed by all Self Assessment Teams
 November 2013: CEEP will send out a Progress Reports
for each program site to complete – based on
implementation of the Program Improvement Plan.
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
Reflection: Weather Forecast
 What is your outlook?
 Do you feel you have
enough information to
be successful in this
process?
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
6
Evaluations
Please take a few minutes to fill out the evaluation
form in your packet before you leave.
Thank you!
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA
planning
with data
May 8, 2013:
Emily Rouge,
LeeAnn Sell, and Stephanie Schmalensee
David P. Weikart Center for Youth Program Quality · PLANNING WITH DATA