Transcript Slide 1

Creative or alternative
methods for collecting data
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
1
• Surveys, end-of-session questionnaires,
interviews and focus groups have become
popular ways to collect data.
• What other ways can you collect data that
may be more novel, more appropriate and
yield better information?
• Remember, always check whether the
proposed method will be culturally
appropriate.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
2
Using existing records and forms
Record books:
– Entries in record books can provide insights
about the environment in a club or project area;
participants’ perceptions of activities;
statements of learning and actions. For
example, members may comment on growth in
leadership skills, “I have learned how to be
more organized” or “I feel more confident in my
leadership role”; or “I led the group to…”
– You might take note of sections that are filled
out and those that aren’t
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
3
Using existing records and forms
Exhibits:
– can be counted and
compared to previous
years’ entries; a team of people can view
exhibits to assess outcomes or list new
ideas that may be used in program
development; judge’s comments can be
gathered and summarized; winning entries
may be noted
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
4
Using existing records and forms
Application, registration and scholarship forms
Forms that are completed by members for
awards and trips and other things such as
camp counselor applications may contain
useful data; for example, answers to
questions such as “How has 4-H enhanced
your problem solving skills?” or, “How would
you describe 4-H to a group of peers?” Or,
include questions to solicit data you need
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
5
Using existing records and forms
Committee and participation lists
You keep a variety of lists that can be used
to track numbers and key demographics of
who is (and who is not) signing up for what.
You can monitor trends over time and get an
idea of how youth are perceiving the
program.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
6
What other existing records
and forms might you use?
Take a few minutes and think about the
various records and forms you have that
either:
(1) already have data that you can use for
your evaluation; or
(2) you can modify to collect the data you
need (perhaps, by adding a question or
two).
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
7
Using games to collect data
Value line or ‘line rating’
– In a value line, people position themselves along a
line to denote their position about a topic.
– Signs are often posted on the wall (or along a line
on the floor) that provide the rating scale (for
example, strongly agree to strongly disagree; very
helpful to not helpful)
– Develop clear strong statements that relate clearly
to the topic and what you want to measure
– You can use the value line at the end of an event;
or use it at the beginning and again at the end to
collect pre-post data or with a sample of clubs
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
8
Example value line
Program: Developing club leadership.
(Example provided by Mary Klemme)
Part of your educational plan is to conduct some
workshops about creating a positive environment
for youth leadership development. You want to
know how clubs are functioning in this area
currently and how effective your workshops are.
You decide to use a value line exercise with a
sampling of clubs both before and after your
workshops.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
9
Statement
Strongly
agree
Agree
Neither agree
nor disagree
Disagree
Strongly
disagree
Youth treasurers should
sign all checks written by
the club.
The club’s general
leader should write the
monthly agenda.
Parents should be able
to vote on club issues at
monthly meetings.
Older teens in the club
do not need to attend
club meetings
Count the number of people standing in the different areas and
record the numbers. You can also use this process to facilitate
discussion about why people took various positions.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
10
Games – Card Sort
• Print brief explanations of program outcomes (or whatever you
are seeking information about and wish people to rate or rank)
on 3 x 5 cards. Make one set of cards for each participant. Or,
let participants generate the items (e.g., benefits they gained
from the program) and create one set of cards for each person.
• Write the rating scale (excellent-poor) or ranks (1st, 2nd, 3rd…)
on baskets placed on a table.
• Ask participants to sort and put their cards into the appropriate
basket.
• You may wish to prioritize the cards in each basket.
• This can be done individually or as small groups working
together.
• Simple key words or graphic images can be used to ease
literacy requirements.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
11
Games…
How might you use such games as:
– Matching
– Jeopardy
– Scattagories
– Etc.
to collect evaluation information?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
12
Using photography to collect data
• Before and after photos
of community service
activities; project activity;
skill development
program, camp, etc.
• Photos taken over the
course of a project to show development and
changes
• Engage youth in taking and interpreting their
own photos
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
13
Photography
Example.
You want to know what the camp experience
means to youth. You randomly select a group
of campers who are asked to take photos
during camp. The photographers then gather in
small groups to discuss their photos, why they
took the photo, what it means to them, etc.
Comments and remarks are recorded.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
14
Photolanguage
A qualitative data
collection technique that uses photographs
to help individuals express themselves more fully
when asked questions. The photographs are
typically black-and-white images in order to allow
viewers to focus on the topic or essence of the
photos.
Select photos related to the evaluation topic that
will stimulate conversation. For example, if you
want to know what youth learned during the
Winter Leadership Camp, you might say, “Select a
photo that helps you describe what you learned
during the leadership camp.”
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
15
Photolanguage continued…
Record what participants say; analyze the data
You can use photolanguage to assess needs,
document outcomes and experiences.
See an article in the Journal of Extension,
“Photos can inspire a thousand words” by
White, Sasser, Bogren and Morgan, June 2009
http://www.joe.org/joe/2009june/iw1.php
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
16
Using creative expression to collect
evaluation data
Various forms of creative expression can be
used to collect evaluative data to document
changes in knowledge, attitudes, skills, feelings
and behaviors.
– Drawing
– Drama
– Role-playing
– Story telling; storyboards
– Music
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
17
Drawing is a natural mode
of expression for boys and
girls. It is a nonverbal
Language and form of
communication that can be
analyzed for structure,
quality and content.
– You might ask participants to draw a
picture of themselves doing their favorite
club activity, after-school activity, camp
activity.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
18
Storyboards as creative expression
• A series of pictures, illustrations or graphics that tell a
story about a program, its impact and what youth
experienced.
• Youth might create their storyboards individually or as a
group.
• They might draw their storyboards on paper or using
computer technologies.
• You might give each person/group a specific aspect to
draw (relative to your evaluation questions) or let them
identify topics they feel would provide useful evaluation
information.
• Once the storyboards are complete, share and discuss.
Record comments as additional evaluative data.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
19
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
20
Journaling
Journals are personal and allow youth to think about their
experiences, what they are feeling and thinking and provides a
useful ongoing account of an activity or experience.
Helping young people journal effectively means more than
giving them a notebook. When using journaling to collect
evaluation data, be clear about what you want to know and
give them a list of questions to address.
Suggest a structure for the journal (or provide pre-printed
pages). Space might be designated for days events; feelings
about the events; reflection on what was learned; what actions
they will take.
You might consider a group journal where all members write
and share their experiences and thoughts in a combined effort.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
21
Journaling
Do a Google search.
There are many resources on journaling on the
web. One is from the University of Tennessee 4HYD, titled Reflection journal at
http://www.utextension.utk.edu/4h/sos/whatisit/reflection/reflection_journal.htm
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
22
Action cards
Action cards are index cards
on which participants record steps, actions, and
accomplishments during the course of a project
or program. Individuals or groups write a short
entry on each card - a few words or sentences.
The cards help to monitor progress, critical
incidents and accomplishments.
Youth can fill in the cards at regularly
scheduled meetings or specified times.
Information on the cards become data for later
analysis.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
23
Group techniques
Any of the common group techniques that we
use in program planning are also useful for
collecting evaluation information:
Brainstorming
Nominal group technique
Delphi technique
Affinity diagramming
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
24
Group technique examples…
• A talking circle: youth form a talking circle and
provide their comments about their experiences
and benefits gained from the program;
comments are recorded
• A modified world café: participants move from
table to table building upon a series of
evaluation questions; the table ‘host’ records
key points
• Brainstorming groups: small groups brainstorm
answers to the evaluation questions with a
summary list recorded on paper
 See also information in the presentation on Interviews – Group interviews
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
25
Goal setting as data collection
Participants set their
own goal(s) at the beginning of the program (e.g.,
desired weight change; change in exercise
pattern; amount of money to save; increase in
public speaking skills, etc). Depending upon the
length of the program, you ask them to revisit
their goal during the program (record progress)
and/or at the end. Progress and achievement are
recorded as evaluation data.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
26
Fun with stickers!
This idea may work well with young children. Count
out a fixed number of stickers and place the same
number in each of three cups (use any number of
cups). Label each cup with “learned a lot”, “learned a
little”, didn’t learn anything”, or whatever response
options fit your question. Ask each youth to take a
sticker from the cup that best answers the question
you ask (your evaluation questions). Tally after each
question. The youth get a fun sticker and you get
evaluation data.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
27
To evaluate a program at its conclusion
Draw a large head, heart and feet on flipchart
paper and post it. Distribute small slips of
paper and ask participants to write down the
major things they learned or got out of the
program. Have them post these in the
appropriate position on the diagram and
discuss them. Record comments.
Or, invite participants to create their own
head, heart and feet and fill in the form, either
individually or with someone else.
You might want to write: ‘What you think’;
‘What you feel’; ‘What you will do’ next to the
head, heart and feet.
Source: Educating for a Change. Adapted from Marsha Sfeir, a Toronto educator.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
28
To evaluate a program at its conclusion
Another variation for quickly collecting data at the
end of a session or program is a card like below:
Three key points:
1.
2.
3.
What is still circling in your
mind!
What is squared away with you that
you can apply and use?
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
29
To evaluate a program at its conclusion
Here is another version of a card for quickly
collecting data at the end of a session or program:
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
30
See the PDF on the web site for
additional ideas using
•
•
•
•
•
•
•
Creative expression
Personal stories or testimonials
Video taping or photography
Expert review
Diaries and journals
Logs
Case study
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
31
Using technology for collecting data
• Many of the ideas in this slide set might be
adapted to the computer.
• Also, see another section on this web site
for ‘using technology in data collection’.
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
32
Analyzing data
when using creative methods
• While many of the methods covered in these slides
seem fun and easy, think about how you will record
and analyze the data you collect.
• Avoid collecting data unless you have a systematic
process for analysis and plan for using the data.
• Many of these methods require the use of
qualitative data analysis that involves categorizing
the data, identifying themes and summarizing the
key points. (see the section on qualitative data
analysis)
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
33
Integrating evaluation into your
programming
• As you can see from this presentation, there
are many ways to integrate data collection
into your programming so that it doesn’t
become an ‘add on’, boring or another
questionnaire to complete.
• Use your own creativity to develop useful
and culturally appropriate data collection
methods. Make sure the method aligns with
your evaluation purpose and what you want
to know!
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
34
Reflection time
What is one thing you
learned (or had reinforced)
from going through this
presentation that you might
try in your own evaluation?
Good luck with your data collection efforts!
© 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation
35