Transcript A Process From Start to Finish

Program Assessment:
A Process From Start to Finish
RJ Ohgren – Office of Judicial Affairs
Mandalyn Swanson, M.S. – Center for Assessment and Research Studies
James Madison University
Session Outcomes
By the end of this session, attendees will be able to:
• Explain how assessment design informs program design
• Describe the “Learning Assessment Cycle”
• Express the difference between a goal, learning objective
and program objective
• Identify effective frameworks to design learning
outcomes
• Define fidelity assessment and recognize its role in the
Learning Assessment Cycle
By The Numbers
Where We Were v. Where We Wanted to Be
Why Assess?
It’s simple:
• The assessment cycle keeps us accountable and intentional
• We want to determine if the benefits we anticipated occur
• Are changes in student performance due to our program?
If we don’t assess:
• Programming could be ineffective – we won’t know
• Our effective program could be terminated – we have no
proof it’s working
Typical Assessment
• We’ll, we’ve got to do a program. Let’s put some activities
together.
• Let’s ask them questions about what we hope they get out of
it afterwards.
• Um…let’s ask if they liked the program too. And let’s track
attendance.
• Survey says….well, they didn’t really learn what we’d hoped.
But they liked it? And a good bit of people came? Success!
Proper Assessment
What do we want students to know, think or do as a result of
this program?
• Let’s define goals and objectives that get at what we want
students to know, think or do.
• What specific, measurable things could show that we’re
making progress towards these goals and objectives?
• What activities can we incorporate to get at those goals and
objectives?
We have a program!
How are these
approaches
different?
Learning Assessment Cycle
Establish Program
Objectives
Use Information
Analyze & Maintain
Information
Collect Objective
Information
Create & Map
Programming to
Objectives
Select and Design
Instrument
Implementation
Fidelity
Program Goals vs.
Learning Goals
Item
Item
Item
Item
Item
Objective
Item
Objective
Objective
Objective
Goal
Goals, Objectives, & Items
Item
Item
Item
Item
Item
Item
Goals v. Objectives
• Goals can be seen as the broad, general
expectations for the program
• Objectives can be seen as the means by
which those goals are met
• Items measure our progress towards those
objectives and goals
Goals vs. Objectives
Goal
Objective
• General expectation of
student (or program)
outcome
• Can be broad and vague
• Statement of what
students should be able to
do or how they should
change developmentally as
a result of the program
• More specific; measurable
• Example: Students will
understand and/or
recognize JMU alcohol and
drug policies.
• Example: Upon completion
of the BTN program, 80% of
students will be able to
identify 2 JMU Policies
relating to alcohol.
Putting it All Together
GOAL
Objective
Assessment
Objective
Assessment
GOAL
GOAL
Objective
Objective
Assessment
Objective
Assessment
Objective
Assessment
By The Numbers Program Goal
1 of 3
• Goal: To provide a positive classroom experience for students
sanctioned to By the Numbers
• Objective: 80% of students will report that the class met or
exceeded their expectations of the class.
• Item: Class Evaluation #15 – Overall, I feel like this class…
• Objective: 80% of students will agree (or better) with the
statement “the facilitators presented the material in an nonjudgemental way.”
• Item: Class Evaluation #5.5 – The facilitators presented the material
in a non-judgemental way.
• Objective: 60% of students will report an engaging classroom
experience.
• Item: Class Evaluation #5.1 – The facilitators encouraged
participation.
• Item: Class Evaluation #5.4 – The facilitators encouraged discussion
between participants.
By The Numbers Learning Goal
1 of 5
• Goal: To ensure student understanding and/or recognition of
JMU alcohol and drug policies.
• Objective: After completing BTN, 80% of students will be able to
identify 2 JMU Policies relating to alcohol.
• Objective: …identify the circumstances for parental notification.
• Objective: …identify the parties able to apply for amnesty in a
given situation.
• Objective: …identify the geographic locations in which JMU will
address an alcohol/drug violation.
• Objective: …articulate the three strike policy.
By The Numbers Learning Goal
2 of 5
• Goal: To ensure student understanding and/or recognition of
concepts surrounding alcohol.
• Objective: After completing BTN, 60% of students will be able to
provide the definition of a standard drink for beer, wine, and
liquor.
• Objective: …identify the definition for BAC.
• Objective: …describe the relationship between tolerance and
BAC.
• Objective: …identify at least 2 factors that influence BAC.
• Objective… identify the definition of the point of diminishing
returns.
• Objective: …identify how the body processes alcohol and its
effects on the body.
By The Numbers Learning Goal
• Goal: To ensure student understanding and/or recognition of
concepts surrounding alcohol consumption.
• Objective: After completing BTN, 80% of students will be able to
correctly identify the definition of the point of diminishing
returns.
• Item: Assessment Question #12, #29
• Activity: Tolerance Activity, Point of Diminishing Returns discussion
• Objective: After completing BTN, 80% of students will be able to
identify how the body processes alcohol and its effects on the
body.
• Item: Assessment Question #8, #9, #10
• Activity: Alcohol in the Body Activity
Developing Learning Outcomes
• Should be Student Focused – Worded to express what the
student will learn, know, or do (Knowledge, Attitude, or
Behavior)
• Should be Reasonable – should reflect what is possible to
accomplish with the program
• Should be Measurable – “Know” and “understand” are not
measurable. The action one can take from knowing or
understanding is.
• Should have Success Defined – What is going to be considered
passing?
Bloom’s Taxonomy
Less
complex
More
complex
Level
Description
1. Knowledge
Recognize facts, terms, and principles
2. Comprehension
Explain or summarize in one’s own words
3. Application
Relate previously learned material to new
situations
4. Analysis
Understand organizational structure of
material; draw comparisons and
relationships between elements
5. Synthesis
Combine elements to form a new original
entity
6. Evaluation
Make judgments about the extent to
which material satisfies criteria
Bloom’s Taxonomy
Bloom’s Level
Verbs
1. Knowledge
match, recognize, select, compute, define,
label, name, describe
2. Comprehension
restate, elaborate, identify, explain, paraphrase,
summarize
3. Application
give examples, apply, solve problems using,
predict, demonstrate
4. Analysis
outline, draw a diagram, illustrate, discriminate,
subdivide
5. Synthesis
compare, contrast, organize, generate, design,
formulate
6. Evaluation
support, interpret, criticize, judge, critique,
appraise
The ABCD Method
• A = Audience
• What population are you assessing?
• B = Behavior
• What is expected of the participant?
• C = Conditions
• Under what circumstances is the behavior to be
performed?
• D = Degree
• How well must the behavior be performed? To what
level?
From “How to Write Clear Objectives”
The ABCD Method: Example
Audience
By the Numbers Participants
Behavior
Condition
Describe relationship between
tolerance and BAC
After taking the class
Degree
80%
• Objective: After completing BTN, 80% of students
will be able to describe the relationship between
tolerance and BAC.
Common Mistakes
Vague behavior
• Example: Have a thorough understanding of the
university honor code.
Gibberish
• Example: Have a deep awareness and thorough
humanizing grasp on…
Not Student-Focused
• Example: Train students on how and where to find
information.
Program
Implementation
Give the program you say you will. How?
Pre Test
(Low Item Score)
Program
Post Test
(High Item Score)
Pre Test
(Low Item Score)
Program
Post Test
(Low Item Score)
Fidelity Assessment
• Are you doing what you say you’re doing?
• Helps to ensure your program is implemented as
you intended
• Links learning outcomes to programming
• Helps to answer “why” we aren’t observing the
outcomes we think we should be observing
Fidelity Components
• Program Differentiation
• How are the many components of your program
different from one another?
• Adherence
• Was your program delivered as intended?
• Quality
• How well were the components administered?
• Exposure
• How long did each component last? How many
students attended?
• Responsiveness
• Were participants engaged during the program?
Fidelity Checklist - Generic
Student
Learning
Outcomes
Program
Component
Objective X Component(s)
aligned with
Objective X
Duration
Features
Length of
List of
component specific
features
Adherence
to Features
Quality
(Y/N)
Quality
recorded for rating for
each feature each
feature
What is rated?
• The live/videotaped program
Who does the rating?
• Independent auditors
• Facilitators
• Participants
Please walk away with this:
You Must Assess.