Transcript TAMU – Galveston Workshops November 20

Academic Program Assessment
November 18-19, 2014
Ryan J. McLawhon, Ed.D.
Director
Institutional Assessment
[email protected]
Elizabeth C. Bledsoe, M.A.
Program Coordinator
Institutional Assessment
[email protected]
[email protected]
979.862.2918
assessment.tamu.edu
Kimberlee Pottberg
Sr. Admin Coordinator
Institutional Assessment
[email protected]
Agenda
• Components of the WEAVEonline
Assessment Plan & expectations of each
• Question and Answer Session
SACS Expectations
SACS Comprehensive Standard 3.3.1
3.3 Institutional Effectiveness
3.3.1
The institution identifies expected outcomes, assesses the extent
to which it achieves these outcomes, and provides evidence of
improvement based on analysis of the results in
each of the
following areas: (Institutional Effectiveness)
3.3.1.1 educational programs, to include student learning
outcomes
3.3.1.2 administrative support services
3.3.1.3 educational support services
3.3.1.4 research within its educational mission, if appropriate
3.3.1.5 community/public service within its educational
mission, if appropriate
SACS Expectations
SACS Comprehensive Standard 3.3.1
3.3 Institutional Effectiveness
3.3.1 The institution identifies expected outcomes, assesses the
extent to which it achieves these outcomes, and provides
and provides
evidence of improvement
basedofonthe
analysis
evidence
of improvement
based on analysis
resultsofintheeach
results…
of the
following areas: (Institutional Effectiveness)
3.3.1.1 educational programs, to include student learning
outcomes
3.3.1.2 administrative support services
3.3.1.3 educational support services
3.3.1.4 research within its educational mission, if appropriate
3.3.1.5 community/public service within its educational
mission, if appropriate
The Assessment Circle
Modify &
Improve
Develop
Program
Mission &
Outcomes
Interpret/
Evaluate
Information
Design an
Assessment
Plan
Implement
the Plan &
Gather
Information
Adapted from: Trudy Banta, IUPUI
Develop Mission and Outcomes
Develop
Program
Mission &
Outcomes
Mission Statement
• The mission statement links the functions
of your unit to the overall mission of the
institution.
• A few questions to consider in formulating
the mission of your unit:
– What is the primary function of your degree?
– What should students graduating from your
program know or be able to do?
Characteristics of a
Well-Defined Mission Statement
• Brief, concise, distinctive
• Clearly identifies the program’s purpose and larger
impact
• Clearly aligns with the mission of the division and
the University
• Clearly identifies the primary stakeholders of the
program: i.e., students, faculty, parents, etc.
Outcomes/Objectives
There are two categories of outcomes:
Learning Outcomes
Program Objectives
Outcomes/Objectives should…
• Encompass a discipline-specific body of
knowledge
• Limited in number (manageable)
• Specific, measurable and/or observable
• Meaningful
• Describe a level of mastery appropriate to the
degree
• Use Action Verbs
Cognitive Learning
Knowledge - to recall or remember facts
without necessarily understanding them
articulate, define, indicate, name, order, recognize, relate, recall,
reproduce, list, tell, describe, identify, show, label, tabulate,
quote
Comprehensive - to understand and interpret
learned information
classify, describe, discuss, explain, express, interpret, contrast,
associate, differentiate, extend, translate, review, suggest,
restate
Application - to put ideas and concepts to
work in solving problems
apply, compute, give examples, investigate, experiment, solve,
choose, predict, translate, employ, operate, practice, schedule
Analysis - to break information into its
components to see interrelationships
analyze, appraise, calculate, categorize, compare, contrast,
criticize, differentiate, distinguish, examine, investigate, interpret
Synthesis - to use creativity to compose and
design something original
arrange, assemble, collect, compose, construct, create, design,
formulate, manage, organize, plan, prepare, propose, set up
Evaluation - to judge the value of information
based on established criteria
appraise, assess, defend, judge, predict, rate, support, evaluate,
recommend, convince, conclude, compare, summarize
Affective Learning
appreciate, accept, attempt, challenge, defend, dispute, join, judge, praise, question, share, support
Examples of
Learning Outcomes
• Graduates will be able to design a comprehensive
research project from research question to
interpretation of results.
• Students will identify and discuss various aspects of
architectural diversity in their design projects.
Program Objectives
• Process statements
– Relate to what the unit intends to accomplish
• Level or volume of activity (publishing, etc.)
• Compliance with external standards of “good practice in the field” or
regulations (accreditation requirements, etc.)
• Satisfaction statements
– Describe how those you serve rate their satisfaction with your
degree program or activities
Examples of Program Objectives
• Process statements
– Students will contribute to the field by publishing
in peer-reviewed journals and national
conferences.
• Satisfaction statements
– Students will report they are satisfied with the
quality of their education.
Student Learning Outcome
Example:
Student Learning Outcome A
• Students receiving a
degree from this program
will be effective
communicators.
Student Learning Outcome B
• Students receiving a
degree from this program
will be able to effectively
communicate their
research findings both
verbally and in writing.
15
Design an Assessment Plan
Design an
Assessment
Plan
Assessment Measures
After establishing your outcomes…
• Define and identify the sources of evidence you
will use to determine whether you are achieving
your outcomes.
• Detail what will be measured and how
• Identify or create measures which can inform
decisions about your program’s processes and
services.
Measures should be…
• Measurable and/or observable
– You can observe it, count it, quantify, etc.
– Specifically defined with enough context to understand
how it is observable
• Meaningful
– It captures enough of the essential components of the
objective to represent it adequately
– It will yield vital information about your program
• Triangulates data
– Multiple measures for each outcome
– Direct and Indirect Measures
Types of Assessment Measures
(Palomba and Banta, 1999)
There are two basic types of assessment measures:
Direct Measures
Indirect Measures
Direct Measures
Direct measures are those designed to directly
measure:
 what a stakeholder knows or is able to do (i.e.,
requires a stakeholder to actually demonstrate the skill
or knowledge)
Common Direct Measures
• A rubric used to assess a collection of work
samples (student work)
• Observation of behavior
Indirect Measures
Indirect measures focus on:
 stakeholders’ perception of their level of learning
 stakeholders’ perception of the benefit of
programming or intervention
 completion of requirements or activities
 stakeholders’ satisfaction with some aspect of the
program or service
Common Indirect Measures
• Surveys
• Exit interviews
• Count of publications/presentations
• Retention/graduation data
• Demographics
• Focus groups
Choosing Assessment Measures
Some things to think about:
– How would you describe the end result of the
outcome? OR How will you know if this outcome is
being accomplished?
– Where are you currently delivering the outcome?
• Are there any naturally occurring assessment
opportunities?
– Will the resulting data provide information that could
lead to an improvement of your curriculum and/or
pedagogy?
Achievement Targets
• An achievement target is the result, target,
benchmark, or value that will represent success at
achieving a given outcome.
• Achievement targets should be specific numbers or
trends representing a reasonable level of success for
the given measure/outcome relationship.
Examples of Achievement Targets
• Students will score a 2.5 out of 4 on the writing rubric.
• 75% of graduating students will achieve an average
of 3.5 on the dissertation rubric.
Implement & Gather Information
Implement
the Plan &
Gather
Information
Findings
• The results of the application of the measure to the
collected data
• The language of this statement should parallel the
corresponding achievement target.
• Results should be described in enough detail to
prove you have met, partially met, or not met the
achievement target.
Examples of Findings Statements
• Example 1:
– Achievement Target
• The mean score on the state mandated certification exam of
students graduating from the program will meet or exceed the
state average score of 79.
– Findings
• The achievement target was met. The overall mean score of
students from the Teaching, Learning, and Culture program
exceeded that of the state average score of the state
certification exam.
Results: Program overall mean scaled score—91.50, State overall
mean scaled score—79.13.
Interpret/Evaluate Information
Interpret/
Evaluate
Information
Analyzing Findings
• Three key questions at the heart of the analysis:
– What did you find and learn?
– So What does that mean for your academic program or
support unit?
– Now What will you do as a result of the first two answers?
Analysis Question Responses
should…
• Demonstrate thorough analysis of the
given findings
• Provide additional context to the action
plan (why this approach was selected, why
it is expected to make a difference, etc.)
• Update previous action plans – results of
implementation
Modify/Improve
Modify &
Improve
Action Plans
• After reflecting on the findings, you and your
colleagues should determine appropriate action to
improve the program.
• Actions outlined in the Action Plan should be specific
and relate to the outcome and the results of
assessment.
– Action Plans should not be related to the assessment
process itself
An Action Plan will…
• Clearly communicate how the evidence of
student learning achievement or other
Findings inform a change or improvement to
curriculum/pedagogy.
• This DOES NOT include:
– Changes to assessment processes
– Continued monitoring of information
– Changes to the program not informed by the data
collected through the assessment process
Assessment Review
Mission Statement
Outcomes/Objectives
Measures
Targets
Findings
Action Plans
Analysis Questions
Take-Home Messages
• Assess what is important
• Use your findings to inform actions
• You do not have to assess everything
every year
OIA Consultations
• WEAVEonline support and training
• Assessment plan design, clean-up, and
re-design
– And we can come to you!
• New Website: assessment.tamu.edu
Questions?
http://assessment.tamu.edu/conference
References
The Principles of Accreditation: Foundations for Quality Enhancement. SACS COC.
2008 Edition.
Banta, Trudy W., & Palomba, C. (1999). Assessment Essentials. San Francisco:
Jossey-Bass.
Banta, Trudy W. (2004). Hallmarks of Effective Outcomes Assessment. San
Francisco: John Wiley and Sons.
Walvoord, Barbara E. (2004). Assessment Clear and Simple: A Practical Guide for
Institutions, Departments, and General Education. San Francisco: Jossey-Bass.
Assessment manuals from Western Carolina University, Texas Christian
University, the University of Central Florida were very helpful in developing
this presentation.
Putting It All Together examples adapted from Georgia State University, the
University of North Texas, and the University of Central Florida’s Assessment
Plans