Transcript Reflect and Revise: Evaluative Thinking for Program Success

Reflect and Revise: Evaluative Thinking for
Program Success
Tom DeCaigny and
Leah Goldstein Moses
Topics to guide our discussion
• Benefits of evaluation and assessment to staff, program
participants and the organization as a whole
• Building on existing knowledge and resources for
evaluation
• Launching evaluation and assessment efforts
• Choosing the right methods of assessment and
evaluation
• Learning from other organizations to integrate evaluative
thinking and use evaluation to support programs
Let’s get to know each other
• Your presenters: Leah Goldstein Moses and Tom
DeCaigny
• How comfortable are you with evaluation?
• What comes to mind when someone asks you to
evaluate and/or assess your work?
• What are your challenges with evaluation?
First, some definitions
• Assessment: The act of determining the standing of an
object on some variable of interest, for example, testing
students and reporting scores.
• Evaluation: Systematic investigation of the worth or
merit of an object; e.g., a program, project, or
instructional material.
– Source Joint Committee on Standards for Educational
Evaluation. (1994). The Program Evaluation Standards, 2nd ed.
Thousand Oaks, CA: Sage. (used with permission of publisher)
Benefits of evaluation and assessment
• Be accountable to important stakeholders
• Professional and organizational development –
learn how you are doing
• Program management – see where your programs
need continued support or improvement
• Investigation and learning
• Feeds curiosity and fosters innovation
Stories of evaluation benefits
• Utilizing evaluation results to identify a need and
develop a new project (special needs - ARISE case
study)
• Utilizing evaluation results to improve program
quality and design (teaching artist training case
study)
• Evaluation and assessment as part of reflective
artistic practice (A Cycle of Artistic Inquiry case
study)
A Cycle of Artistic Inquiry
Performing Arts Workshop and Dr. Richard Siegesmund (2000)
Ensuring evaluation benefits are shared
• Evaluation, at its best, is “engaged in”, not “done to”.
• When developing or improving evaluation systems, think
about who will be doing the work for the evaluation
(distributing surveys, gathering information, analyzing
the data):
– Is there a way to decrease the burden?
– Is there a way to provide benefits?
• Examples of evaluation efficiency and incentives
Build on existing knowledge and resources
• Internal insights can really support a new
evaluation effort. Determine:
– What do we collect already?
– What does the information we already have tell us?
– What can we report on just from our own internal
record keeping or observations?
– What can we gather in the course of our work – during
existing programs, contact, etc.
– Example in ARISE: student achievement – test scores..
Build on existing knowledge and resources
• Use external information, such as reports done by
organizations you admire:
–
–
–
–
What did they study?
How did they gather information?
Can you apply any of their tools or methods?
Can you infer/generalize anything from their findings
so you don’t have to replicate their effort?
Getting started in evaluation and
assessment
• Logic models are incredibly useful. They help you:
– Determine how your efforts are related to your
expected impact
– Map out what you want to measure, and why
• You can determine what data you already have
and what you are lacking during the logic model
process
Getting started in evaluation and
assessment
• After you have created a logic model and/or
identified data gaps, you can determine what you
are going to collect, when, and in what format
• Surveys are great; but in the arts, you might want
to use an artistic process or other valid alternative
assessments
– Illustrative rubrics
– Observation
– Portfolios
Learning from others’ experiences
•
Notes from our discussion:
–
What evaluation approaches have worked well for you?
Electronic portfolios in classroom. Reflects project-based learning. Can see progression over
course of year. Parents, administrators can also see.
Time to reflect can be challenging but is important.
“Level Best” is a good resource. Tried to find things that existed and could be used in the
organization. Festin
“Theater Communications Group” is a good resource.
Anecdotal information, journals can have a bigger impact on Boards and other audiences that
don’t care much for quantitative. Site visits for Board members are required as part of
their responsibilities.
Having tools at your fingertips – did anything good happen today? Did any challenges
happen? Right at time they are needed.
Using incentives – crayola pencils were good for parents.
– Where have you struggled?
Finding time for reflection. Having right tool for evaluation. Logic models- can be cumbersome or difficult
to use. Having a way to capture, understand and communicate unexpected outcomes. Avoiding bias
through body language, tone – need to make sure to encourage honesty.
Board can ignore quantitative.
Finding sophistication/depth in questions has been hard when they are in a survey.
Our contact information
Tom DeCaigny, Executive Director, Performing Arts Workshop
T: (415) 673-2634 x207 / F: (415) 776-3644
E: [email protected]
www.PerformingArtsWorkshop.org
Leah Goldstein Moses, President, The Improve Group
T: (877) 467-7847 x11 / F: (612) 656-1731
E: [email protected]
www.theimprovegroup.com