Transcript Evaluation and Measurement - University of Wisconsin–Stout

Evaluation and Measurement
Spring 2010
Quality Training Certificate Program
Quality Training Certificate Program
Certificate Details:
Participants have the option to attend individual classes, or to participate in 6 classes
and obtain a certificate in quality training. Classes will be held twice yearly for a
total of three years to complete the certificate. Classes will continue to cycle, so
new participants will be able to complete the certificate no matter when they
begin. Classes can also be taken at any time as stand-alone experiences. There will
be no charge for attending the classes, but supervisor permission must be given for
attendance during regular work hours.
Attendees from previously held classes will be given credit toward their certificate for
attending.
Classes will last 3 hours each, for a total of 18 total contact hours.
Attendance credit will continue to be given for individual classes taken. When
attendees complete six classes, they will fill out an application for certification, and
send it to BPA, who will verify the information and award the certificate.
Today’s Power Point and Handouts
Will be available online:
http://www3.uwstout.edu/bpa/training.cfm
Overview




Introductions
Goals for Today
Opportunity to suggest additions to today’s agenda
Course Content, including:




Practice Designing an Evaluation
Discussion Questions
Q&A
Breaks as needed
Introductions

Tell us:


Who you are
What worries you most about evaluation and measurement?
Goals for Today





To understand when evaluation can be used
To understand the basic questions that guide the
evaluation process
To identify appropriate evaluation methodologies
To implement basic evaluation methodologies
To utilize evaluation results
Usage of Today’s Curriculum


To apply the skills that you learn today in your everyday
practice
Within the next month, to practice at least one thing that
you learned today
Questions/Topics you Want Covered


Write down anything you want to be sure we cover on
the notecards in front of you
I will collect these when we take our first break
To understand when eval. can be used

What can be evaluated:

3 P’s:





Programs
Processes
Products
Evaluations can be “formal” or “informal” and time
consuming or simple.
All faculty and staff typically engage in some sort of
evaluation in their everyday work – although not all apply
it in a formal way.
To understand when eval. can be used
Two major purposes:
 To answer questions about changes that need to be made
to programs/processes/products (formative),
 To answer questions about continuation/discontinuation
of programs/processes/products (summative)
Most of the evaluations we do at UW-Stout are formative.
To understand when eval. can be used
Examples:
 Most grants require an evaluation section
 The surveys that you fill out at the end of a training
session are a form of evaluation
 UW-Stout has a formal evaluation plan for our e-Scholar
program, first year experience program, customized
instruction program, etc..
 Evaluation can be used to determine if you are meeting
customer needs
To understand when eval. can be used
Discussion question:
 What other examples do you have of when evaluation
can or has been used?
 What was your experience with these evaluations?
To understand the basic questions that
guide the evaluation process
Two basic questions:
•
•
What is the purpose of the process/product/ program?
How will you use the results of the evaluation?
To understand the basic questions that
guide the evaluation process
Considerations:
It is sometimes hard to identify the goals and usage of the
results, but you cannot develop a good evaluation without this
information. Other ways to ask the question:
•
•
•
•
What does success look like?
If this program/process/product did was wildly successful, describe what
that would look like.
Do not proceed with the evaluation if you cannot articulate how
you will use the results.
To understand the basic questions that
guide the evaluation process
Discussion Question:
 How often have you been asked to participate in a survey,
focus group, or other data collection method when it was
not clear how the results would be used?
To understand the basic questions that
guide the evaluation process
• All evaluation methods, data, and analysis
should tie back to the goals and how the data
will be used
• A matrix can facilitate this:
Goal:
Goal 1
Goal 2
Goal 3
Goal 4
How to
know if
goal is
achieved:
Evaluation
methods:
Target:
Target
date:
Example evaluation plan
•
To understand the basic questions that
guide the evaluation process
Practice:
 Think of an evaluation that you have done, or that you
would like to do, or that you anticipate you will be
involved with.
 If you don’t have one, use one of the examples on the flip
chart.
 Fill in:



Evaluation question
Goals of the process/product or program
How the results of the evaluation will be used
To understand the basic questions that
guide the evaluation process


Once you have identified your goals and how the results
can be used, you need to think about how you will know
if the goal is achieved.
At this point, do not think about methods.
Example:
 Goal: To identify appropriate evaluation methodologies
 How achieved: Level of knowledge participants have of
evaluation methodologies
To understand the basic questions that
guide the evaluation process
Discussion question:
 Who has a goal from their sheet that they are
comfortable sharing?
 How will we know if this goal is achieved?
To understand the basic questions that
guide the evaluation process
Practice:
 Fill in the “how to know if goal is achieved” column on
your sheet
To identify appropriate evaluation
methodologies
Considerations:
 Don’t let methods constrain initial discussion – start with
ideas.
 People like to challenge methodologies if they don’t like
the results.
 Too much data is a common problem. All data must tie
back to goals.
 It often takes more time to collect, manage and analyze
data than you think it will.
 Utilize multiple methods.
 Available time and resources will often be the driving
factors for selecting methods. Also your audience.
To identify appropriate evaluation
methodologies
Major evaluation methods:
 Existing data
 Surveys
 Focus Groups/Interviews
 Inventories/Usage data/Database creation
To identify appropriate evaluation
methodologies
Discussion questions:
 What do you think we see most often at Stout?
 What do you think is the most underutilized method at
Stout?
To identify appropriate evaluation
methodologies
Existing data
You want to use existing data wherever possible.
 Pros:




Minimal time commitment
Readily available
Ensures data collected is used for multiple purposes
Cons:

May be limited on what is available
To identify appropriate evaluation
methodologies
Surveys
 Use surveys when:



The information you need to obtain is not available from
existing data, and cannot easily be obtained through inventories
or databases
You don’t need detailed qualitative data
You have enough time to do it right
To identify appropriate evaluation
methodologies
Surveys
 Pros:




Can get at factors not easily obtainable from other sources –
for example, motivation, attitudes
Typically does not require a large time commitment from
participants
Best for likert-scale, checklist short answer-type questions
Cons:



Relies on self-report
Not good if you want in-depth, qualitative comments
Language, response rates, etc… often challenged
Purpose of a focus group:
“The purpose of a focus group is to listen and gather
information. It is a way to better understand how
people feel or think about an issue, product, or service.
Participants are selected because they have certain
characteristics in common that relate to the topic of
the focus group”
Krueger, Richard, and Casey. Focus Groups. 3rd ed. Thousand Oaks, USA: Sage Publications,
2000.
What a focus group is




A carefully planned series of discussions designed to obtain
perceptions on a defined area of interest
Discussions in a permissive, nonthreatening, environment
Groups of 5-12 facilitated by a skilled facilitator
Repeated groups – need to hold at least 3-4 sessions
When to use focus groups:


Looking for a range of ideas or perspectives on an
issue
Trying to determine differences in perspectives
between groups of people


“A group possesses the capacity to become more than the
sum of its parts” (Krueger and Casey, 2000)
Researcher needs information to aid existing
quantitative data
When NOT to use focus groups:






Want people to come to a consensus
Want to educate people
Don’t intend to use results
Other methodologies can produce better results
Other methodologies can produce same results more
efficiently
Don’t have enough time to follow standard focus
group procedures
To identify appropriate evaluation
methodologies
Focus Group/Interviews
 Pros



Allows you to go in-depth in an issue
Allows for a range of ideas – and you hear from people in their
own voice
Cons

Most time-consuming approach
To identify appropriate evaluation
methodologies
Inventories/Usage data/Observations
 Pros:



Typically more accurate than self-report data
Good for tracking participation in events, trainings, programs,
etc..
Cons:



Only works for things you can count or observe
Often have to rely on other people to obtain the data
When multiple people are involved, need to make sure that
everything is counted in the same way
To identify appropriate evaluation
methodologies
Practice:
 Circle the evaluation methods that you feel can be
utilized to obtain the information in the second column of
your sheet. Circle multiple methods if you can use more
than one method.
Tying it all together



You have two more columns on your sheet – a target and
target date.
It is important that you identify specific targets and dates
before you collect any data – as after you collect the data,
people will use it to prove their point.
Often you cannot identify a target until you have your
evaluation methods more fully developed – so we will not
be filling in these columns today.
Tying it all together


What you have on your sheets now is typically your ideal
scenario for how you can go about evaluating your
process/product/program.
Usually, you don’t have time to do it all, so next you need
to determine what is feasible given your time, resources,
and who will be reviewing the results.
Tying it all together
Start by sharing your plan with your key stakeholders to get
their input.




Explain why you have chosen the methods that you have.
Convince them that you have good reason for choosing the
methods that you did.
What concerns do they have about the plan?
Given your available time and resources, you need their help to
prioritize what is most important to them.
It is often helpful to obtain approval of the plan in writing.
Tying it all together
Practice:
 Share your plan with a neighbor. Pretend they are one of
your key stakeholders. Get their input.
Tying it all together
Discussion question:
 Is anyone willing to share their plan with the group?
To implement basic evaluation
methodologies
Now that you have approval on your plan, you need to
implement it.
Next, you need to develop your instruments and implement
them.
To implement basic evaluation
methodologies
Considerations:
 Before collecting your data, develop a data analysis plan.
 Just as it is important to get approval on your evaluation
plan before starting, it is also important to obtain
approval on the instrument and data analysis plan.
 Pilot testing is important – administer the survey/focus
group/form to people similar to those who will receive it.
To implement basic evaluation
methodologies
Existing Data
 Can sometimes find a survey, focus group or other study
that has recently been administered that covers the same
topics that you are assessing
 Several ways to access existing data at Stout:




Contact the BPA office
Survey Clearinghouse (in development)
Information Portal (in development)
Qualitative results:
http://www.uwstout.edu/static/bpa/ir/surveylistqual.html
To implement basic evaluation
methodologies
Surveys
 Qualtrics is the online survey instrument that Stout uses:
http://www3.uwstout.edu/bpa/survey/
 Qualtrics question library in development. Will be
available from the above link.



Will provide suggested demographic questions
Will also provide suggestions for other standard questions
Typically don’t want it to take more than 10 minutes to
complete – time to complete is more important than the
number of questions.
To implement basic evaluation
methodologies
Surveys, cont.
 Look for other surveys you can model after – internally
developed and externally developed. Request permission
from the author to use some of the same questions.



Survey Clearinghouse (in development)
Surveys webpage:
http://www.uwstout.edu/static/bpa/ir/surveylist.html
Sample survey guide:
http://www.uwstout.edu/static/bpa/ir/afu/information/instr
uction.pdf
To implement basic evaluation
methodologies
Focus Groups/Interviews
 Different than a “listening session”
Discussion question:
How are focus groups different than listening sessions?
To implement basic evaluation
methodologies
Core Ingredients
 Continue groups until you hear same themes repeated –
the ARC typically holds 8 sessions. Sometimes it is
possible to do less.
 No more than 12 people in a group
 Circle seating
 Typically requires formal training for facilitators/
moderators
 Pre-determined questions
 Systematic analysis
To implement basic evaluation
methodologies
Inventories/Usage data/Observations
 Must provide training to the people who will be entering
data into databases or inventories. Include specific
instructions.
 Examples:



Sign-in sheets outside of a tutor center (sample in separate
document)
Appointment logs
Formal observation forms, with checklists, open-ended
questions, tally marks, etc… (sample in separate document)
To utilize evaluation results
Evaluation results are primarily used for two purposes:
 Program/process/product improvement (formative)
 Decisions about program/process/product
continuation/discontinuation (summative)
Both are important
To utilize evaluation results
Formative evaluation:
 Typically done early in on program. Usually identifies
improvements that can be made related to program
implementation.
To utilize evaluation results
Formative evaluation, examples:
 Decisions about improvements that can be made to get
more people to attend a pre-college program.
 Decisions about improvements that can be made to get
more people to apply what they’ve learned from training
programs.
 Decisions about new ways to present or deliver
departmental newsletters.
To utilize evaluation results
Summative evaluation:
 Typically done later on in the program – after all of your
target dates have passed. Usually used to make decisions
about continuation/discontinuation.
 Typically summative decisions are the most difficult to
make. If you establish specific targets and target due
dates, and have done formative evaluation prior to the
summative evaluation, these decisions are easier.
To utilize evaluation results
Summative Evaluation, examples:
 Decisions about if funding for a grant will continue
 Decisions about if various initiatives to improve retention
will continue
 Decisions about if on campus funding will become
permanent
 Decisions to keep or eliminate items from the campuswide training and development plan
To utilize evaluation results


We often hear from people that nothing is done with the
feedback they provide on surveys, focus groups, etc…
It is important to invest time into sharing how you have
used the results from the evaluation. People will not
often make the connection between feedback and their
actions on their own.
Questions/Comments?