Transcript Writing Effective Assessment Plans

Writing Effective Course
Assessment Plans
Kim Anderson
ASLO Subcommittee Chair
Fall 2009
Why Assessment Plans?
• "the systematic collection, analysis,
interpretation, and use of information to
understand and improve teaching and learning"
(Tom Angelo).
• Facilitates periodic, not episodic assessment of student
learning and program outcomes
• Serves as a planning management guideline for program
review process
• Leads to improvement in student learning, retention and
program completion
• Enables faculty to play central role in managing student
learning
• Facilitates collaboration within and among departments
• Required by accreditation
Assessment Plan Development
Things to Consider
•
•
•
•
Time
How consistent is the evaluation?
How robust is the final assessment information?
How will this process meet the standards set by the
college’s planning process?
• How effectively does this process provide guidance to
our stakeholders?
• How well can this information be used to improve
student learning?
• How will our process meet the standards set by
ACCJC?
Key Elements
• Common Methods and Criteria
– In total
– In part
– Common core of evaluation standards
• Discussions
– Regularly scheduled time (department meetings, work group
meetings, planning days, school meetings)
– Participation (full-time and part-time)
• Embedded Assessments
– Decide on course work
– Established assignment(s) and incorporate
Outcomes Assessment
Management
• Internal Management Structure
– Who (identified)
– When (note in assessment plan)
– How (implement by course, not SLO)
• Examples
– Gantt Chart/Excel
– Web based tool
– TracDat
Course Outcomes Assessment
Cycle
• Routine Course Review Cycle - at least every
six (6) years
• Starting up will require shortened cycle for
some
• The Three “Ms”:
– Meaningful
– Measureable
– Manageable
What Are The Components of
Assessment Plans?
• First, clearly state outcomes
• Second, identify an assessment tool
• Third, establish the criteria and level
of expectation for success
• Fourth, gather and analyze the
results
• Fifth, create a plan for improvement
Assessment Plan Format
Intended Outcome
Assessment Task
Criteria/Expected
Level of
Achievement
Results of
Assessment
Use of Data/Plan
1. Identify intended outcomes
SLOs = Core, overarching, broad statements of knowledge and skills
learned in a course or in an outside of class learning opportunity
– Avoid complex statements—if outcome contains more than one
measurable item, then separate into different outcomes
– Use an active verb indicative of expected learning or experience
– State what want at end of experience—the outcomes reflect what
you want to achieve
– The outcomes are achievable, reasonable, and measurable
– If accredited by a national organization, the outcomes address the
expected student learning or service standard of such organizations
– If it is part of a sequence so is well-reasoned in relationship
Outcome Statements Format
Student Learning Outcome (SLO)
Statements about what a student will know, do, or feel
upon completion of the course/learning opportunity
Incorporate the course or learning opportunity’s core
knowledge, skills, and/or attitude; must be specific to
course/learning opportunity
Use an active verb and is primarily critical thinking
(Bloom’s Taxonomy)
2. Choose Assessment Task
• Assessment Tools
✓Direct reasonable replications of real world tasks; capstone exam/project,
pre- and post-test, rubric, licensing exams, case study
{Indirect performance proxies; surveys, student interviews, employee
questionnaire (cannot be used exclusively for course assessment)}
✓Quantitative numerical or statistical values; use actual numbers (scores,
rates, etc.) to express quantities of a variable
{Qualitative descriptive information from questions or statements that
provides feedback and possible suggestions; surveys, or summary reports}
Formative ongoing; to improve learning or process steps
Summative final; to gauge quality end product or process
• Common agree so can aggregate results; time efficient & ease of use
• Who will conduct the course assessment
• When time frame & cycle; rotate so not every year but all every cycle
• How embed in course assignment or service experience
Suggested Assessments
Examples for SLOs
Capstone project or Final exam = direct assessment methods
integrate knowledge, concepts, and skills associated with an entire
sequence of study in a course
Rubric = flexible direct assessment to score any product or
performance (e.g. essays, portfolios, skill performance, oral speech,
project/product creation, etc.)
Pre-Test/Post-Test = direct assessment tool administered at the
beginning and at the end of a course to evaluate the progress
students make
Other Options may be used depending on SLO but tool development
will need to be in conjunction with the Office of Institutional
Effectiveness
Checklist for Assessment Tools
•
•
•
•
•
•
•
•
•
•
•
Embedded in everyday activities.
Authentically measures student skills, knowledge, or ability.
Based upon clearly defined material.
Evaluated by explicit criteria.
Collects valid evidence concerning a specific outcome.
Provides reliable results or reproducible evaluative results.
Assessment criteria and expectations have been validated through discussion with
colleagues.
Systematically administered to students.
Conforms to institutional mission.
Consistent with department’s program outcomes and the college’s Instructional
Program Outcomes.
Cost effective concerning resources, supplies and administration.
3. Establish criteria/expected
level of achievement
1. Minimum expectation for achievement (%,
fraction, actual number); 1st time determine a
majority then base expectations on previous
assessment information when available
2. What is the success level in regards to the
assessment tool/task (“satisfactory” on the rubric, a
passing grade, numeric score, narrative indicator)
3. Which students count in data (successful
completion of course only, all queried,
representative sample, random sample)
Sample Document Layout
Intended Outcome
Assessment Task
Criteria /Expected
Level of
Achievement
Results of
Assessment
Use of Data/Plan
Write the intended
outcome in this
column.
• Student Learning
Outcomes (SLOs) are
identified in the course
outline of record.
What:
Describe the assessment
tool or task
How:
Explain the
implementation of the task
Who:
Identify persons
responsible for conducting
the assessment
When:
Identify the timeframe of
assessment task;
commencement and for
how long
(semesters/years)
Expected Achievement:
Establish a minimum
expected benchmark for
achieving the outcome
Success Level:
Determine the acceptable
success standard based on
the identified assessment
task
Students Included:
Describe the students to
be included (e.g. successful
completers, all queried,
random sample)
Results:
Key Findings:
Conclusions:
Plan:
Re-evaluation Date:
Begin Outcomes
Assessment Process for
this course at this point.
4. Gather and analyze
results of assessment
• Gather, organize, aggregate, analyze, and report
• Process protocol and who’s involved; periodicity; deadlines; regular cycle
in conjunction with routine course review cycle dates
• Discussions at various meetings – need time
• Aggregate
– Within the course/department/program
– Office of Institutional Effectiveness
• Analysis and evaluation
– Through discussion with vested individuals, which can mean within
and outside of the department/program
– Was the data valid and helpful; what does it say that is working well;
what could be improved
– Discuss the discipline issues
5. Create a Plan
• Based on results ascertain a step or steps to improve student
learning
– Curriculum (content emphasis, sequencing, assignments)
– Pedagogy (individual/group work, technology, learner
styles)
– Assessment (SLO, assessment tool)
• Establish definite actions and who is responsible
• Determine time frame for implementation
• Identify follow-up process
• On-going assessment cycle
Outcomes Assessment Loop
Intended Outcome
(SLO)
Use of Data/Plan
(Modifications/Time Frame/Who
Impliments)
Results of Assessment
(Results/Highlights/Conclusion)
Assessment Task
(What/How/When/Who)
Criteria/Expected Level of
Achievement
(Criteria/Standards/Who is
Counted)
Capstone
• Exam or Essay
– Same in all sections
– Shared questions that assess SLOs
– Common essay topic but with agreed upon scoring system
(rubric)
• Project
– Individual = culminating/summative
– Collaborative = formal/informal, typically lab settings
– Agreed upon scoring system (rubric)
What is a Scoring Rubric?
A rubric is a continuum of scoring categories
and guidelines for teachers to evaluate and to
give feedback to individual students
AND
to evaluate groups of student work to assess
for course effectiveness.
Essentials of a Good Scoring Rubric
• Criteria must align with the SLO by identify the
various characteristics of the outcome.
• Scoring categories specify various levels of
achievement in each characteristic.
• Assessment task should mirror real-life challenges
as appropriate.
• Two independent raters will arrive at a similar
score for a given response based on the scoring
rubric.
Components of a Scoring Rubric
• Assignment = task components of a product or
performance/clear definitions of each
characteristic to be assessed
• Levels of Achievement = scale of success by
letter grades, numbers, or narrative
• Evaluation = criteria standards or
competencies
• Documentation
Assignment = task
• Product
– Essay
– Portfolio
– Final project
– Shop product
– Document/Map
• Performance
– Speech
– Dance/Act
– Play an
instrument/Sing
– Skill activity
– Lab procedure
Elements or characteristics of the assignment
Achievement = scale
• Letter Grade
A, B, C, D, F
• Numbers
1-3 (simple) ; 1-4; 1-5 (detailed)
• Narrative
“beginning, developing, competent,
exemplary”; many versions
Evaluation = criteria
• Evaluation criteria & standards
– Demonstration of requirements
– Evidence of specific progress
– Feedback on competencies
– Miss, meet or exceed standards
Documentation = samples
•
•
•
•
•
Product
Performance
Letter grade
Number scale
Narrative explanations
“Product” Excerpt
Excel Spreadsheet Final Project
Below Expectation
1
Satisfactory
2
Exemplary
3
Requirements
Document Sheet
Requirements
document is
incomplete.
Many sections are
missing. Descriptions
are missing or not
written thoroughly.
Requirements
document is not
complete.
Includes most of the
assigned sections.
Missing 2 or more
sections or lacks
accurate descriptions.
Requirements
document is complete.
Includes all assigned
sections in addition to
thorough and accurate
descriptions.
Worksheet Design &
Formatting
Incomplete-missing
more than two:
Missing labels &
values
Title & subtitle
Column headings
Missing title merged
and centered
3 or more spelling
errors
Somewhat completemissing two:
Pertinent labels &
values
Title & subtitle
Column headings
Title merged and
centered
Fewer than 2 spelling
errors
High level of
professionalism
All labels & values
included
Appropriate titles &
subtitles added
Title merged and
centered
No spelling errors
“Performance” Sample
Golf Bunker Shot
Beginning
Developing
Competent
Stance
Closed stance
Posture – bent over,
stiff knees, weight
even
Open stance
Posture – straight
back, slightly bent
knees, weight slightly
forward
Open stance
Posture - straight
back, slightly bent
knees, weight
forward
Ball Position
Ball positioned too far
forward in the stance
Ball positioned too
close or too far to the
golfer
Ball positioned in the
middle of the stance
Ball positioned at a
distance from the
golfer that impacts
the gold posture
Ball positioned
middle to back of the
stance
Ball positioned the
correct distance from
the golfer that allows
good golf posture
Swing Technique
Closed club face
Halting,
shortened/too large
of a swing
Eyes move
Erratic arc of the
swing
Parallel club face
Jerky, but complete
swing
Eyes down
Complete swing, but
off line
Open club face
Smooth, rhythmic
swing
Eyes behind the ball
“Down the target line”
swing
“Grade” Sample
Political Science Essay
•
•
•
•
•
A = The A-range essay implies a strong argument and provides convincing specific support from the various readings. The
writer demonstrates mature command of language through a variety of sentence structures, word choices, quotes or
paraphrases from the readings (consistently cited correctly). Control of usage and mechanics, despite occasional flaws,
contributes to the writer’s ability to communicate the purpose of the paper. The writer thoroughly understands the
concepts/theories involved and through the essay can convince others of his/her viewpoints or helps make the reader aware of
something completely new or original. These are powerful due to organization and creativity.
B = The B-range essay shows effort and promise for the writer. It presents a thesis (argument) and often suggests a plan of
development that is carried out effectively. Mastery of the readings/theories may not be fully indicated with the use of quotes
or paraphrases, but the writer provides enough supporting details, makes competent use of language, and sometimes varies
sentence structure. Occasional errors in usage and mechanics do not interfere with the writer’s ability to communicate the
purpose of the paper.
C= The C-range essay presents a thesis (argument) and often suggests a plan of development which is generally c arrived out.
The writer may or may not have completed all of the required readings and utilizes generalizations or list for support.
Command of the theories under consideration is weak or shaky. Sentence structure tends to be repetitious, and errors in
usage and mechanics sometimes interfere with the writer’s ability to communicate the purpose of the paper.
D = The D-range may present a thesis (argument); however, the plan of development is usually not carried out. This
indicates the writer may have spent little time with the readings or thinking about the concepts involved. The writer provides
support that tends to be sketchy and/or illogical. Sentence structure is simplistic, repetitious and occasionally awkward.
Language is often inappropriate in tone or style. Errors in usage and mechanics are frequent.
F = This paper presents a thesis that is vaguely worded, weakly asserted or there is no central argument present. Support, if
any, tends to be rambling and superficial. Sentence structure is difficult to follow and errors in usage and mechanics interfere
with the writer’s ability to communicate the purpose of the paper.
“Numeric” Sample
Philosophy - Critical Thinking
4
Consistently does all or almost all of the following:
Accurately interprets evidence, statements, graphics, questions, etc.; Identifies the salient arguments
(reasons and claims) pro and con.; Thoughtfully analyzes and evaluates major alternative points of
view.; Draws warranted, judicious, non-fallacious conclusions.; Justifies key results and procedures,
explains assumptions and reasons.; Fair-mindedly follows where evidence and reasons lead.
3
Does most or many of the following:
Accurately interprets evidence, statements, graphics, questions, etc.; Identifies relevant arguments
(reasons and claims) pro and con.; Offers analyses and evaluations of obvious alternative points of
view.; Draws warranted, non-fallacious conclusions.; Justifies some results or procedures, explains
reasons.; Fair-mindedly follows where evidence and reasons lead.
2
Does most or many of the following:
Misinterprets evidence, statements, graphics, questions, etc.; Fails to identify strong, relevant
counter-arguments.; Ignores or superficially evaluates obvious alternative points of view.; Draws
unwarranted or fallacious conclusions.; Justifies few results or procedures, seldom explains
reasons.; Regardless of the evidence or reasons, maintains or defends views based on self-interest
or preconceptions.
1
Consistently does all or almost all of the following:
Offers biased interpretations of evidence, statements, graphics, questions, information, or the points
of view of others.; Fails to identify or hastily dismisses strong, relevant counter-arguments.; Ignores
or superficially evaluates obvious alternative points of view.; Argues using fallacious or irrelevant
reasons, and unwarranted claims.; Does not justify results or procedures, nor explain reasons.;
Regardless of the evidence or reasons, maintains or defends views based on self-interest or
preconceptions.; Exhibits close-mindedness or hostility to reason.
“Narrative” Excerpt
English 1/ESL 1 Final Essay Examination
Sampling of Narrative Criteria
A. Respond to the writing task.
The paper must respond to the specific topic and the reading and must show understanding of the reading.
A paper simply restating the content of the reading is not an adequate response. The paper must be at
least 400 words long.
B. Treat the reading fairly.
The paper must avoid plagiarism by attributing the words and ideas of others to their source. Borrowed
words and combinations of words must be set off by quotation marks. The paper must not misinterpret or
misquote others’ words.
C. Maintain a consistent style and sense of audience.
The paper must not have abrupt or confusing shifts in person, voice or tense.
E. Be organized logically and coherently.
The paper must be organized according to a discernible plan, with the main points
supporting the thesis. The reader must be able to follow the plan with ease.
I. Adhere to the conventions of Standard English.
The paper should have no more than minimal errors in spelling, punctuation, capitalization, apostrophe,
and case, and should use standard forms of verbs, adjectives, and adverbs.
Course Assessment Plan
Recap
• Establish periodic assessment of student
learning in courses to improve student success
• Column 1: SLOs
• Column 2: Assessment Tool-What, How, Who,
When
• Column 3: Achievement/Success-Expected
Achievement, Success Level, Students
Included