Transcript Writing Effective Assessment Plans

Writing Effective Assessment
Plans
Eva Bagg
Associate Dean of Institutional Effectiveness
Kim Anderson
Course Evaluation Subcommittee Chair
Summer 2009
Why Assessment Plans?
• "the systematic collection, analysis,
interpretation, and use of information to
understand and improve teaching and learning"
(Tom Angelo).
• Facilitates periodic, not episodic assessment of student
learning and program outcomes
• Serves as a planning management guideline for program
review process
• Leads to improvement in student learning, retention and
program completion
• Enables faculty to play central role in managing student
learning
• Facilitates collaboration within and among departments
• Required by accreditation
Who Is Involved in Organizing
Assessment Plans?
•
•
•
•
Faculty and co-curricular professionals
Department Heads
Deans and Directors
Associate Dean of Institutional Effectiveness
Who is Involved with Assessment Plans?
All Colleagues contribute and should participate
Assessment Plan Development
Things to Consider
•
•
•
•
Time
How consistent is the evaluation?
How robust is the final data set?
How will our process meet the standards set by the
college’s planning process?
• How will our process meet the standards set by ACCJC?
• How effectively does our process provide guidance to
our associates?
• How well can our data be used to improve student
learning?
What Are The Components of
Assessment Plans?
• First, clearly state outcomes
• Second, identify an assessment tool
• Third, establish the criteria and level
of expectation for success
• Fourth, gather and analyze the
results
• Fifth, identify the actions taken for
improvement
Assessment Plan Format
Intended Outcome
Assessment Task
Criteria/Expected
Level of
Achievement
Results of
Assessment
Actions Taken
1. Identify intended outcomes
SLOs = Core, overarching, broad statements of knowledge and skills
learned in a course or in an outside of class learning opportunity
SUOs = statements of key service effectiveness, efficiency, accuracy,
volume, compliance, and experience that is distinct to the unit;
process and satisfaction
– Avoid complex statements—if outcome contains more than one
measurable item, then separate into different outcomes
– Use an active verb indicative of expected learning or experience
– State what want at end of experience—the outcomes reflect what
you want to achieve
– The outcomes achievable, reasonable, and measurable
– If accredited by a national organization, the outcomes address the
expected student learning or service standard of such organizations
– If it is part of a sequence so is well-reasoned in relationship
Outcome Statements Format
SLO
SUO
Statements about what a student will
know, do, or feel upon completion of the
course/learning opportunity
Statements about what a client will
experience, receive, or understand as a
result of a given service
Client = anyone receiving a service (e.g. student, faculty
member, staff member, community member)
Service = any established function central to the unit (e.g. front
desk interactions;, processing of requests; management, repair,
or maintenance of systems)
Incorporate the course, learning
Process = what unit intends to accomplish
Volume of unit activity
opportunity, or program’s core
Efficiency of processes as conducted
knowledge, skills, and/or attitude; must
Accuracy of work produced
be specific to course/learning opportunity Compliance with regulations & “good practices”
Effectiveness of service in accomplishing outcome
Use an active verb and is primarily critical
thinking (Bloom’s Taxonomy)
Satisfaction = what client experienced;
with entire process received or with
individual services
2. Choose Assessment Task
• Assessment Tools
Direct reasonable replications of real world tasks; capstone exam/project, preand post-test, rubric, computerized tracking counts
Indirect performance proxies; surveys, student interviews, employee
questionnaire
Quantitative numerical or statistical values; use actual numbers (scores, rates,
etc.) to express quantities of a variable
Qualitative descriptive information from questions or statements that provides
feedback and possible suggestions; surveys, or summary reports
Formative ongoing; to improve learning or process steps
Summative final; to gauge quality end product or process
Process service received; effective, efficient, correct, accurate, compliance
Satisfaction quality of the service interaction; overall or specific service
• Common agree so can aggregate results; time efficient & ease of use
• Who students, alumni, employers, clients
• When time frame & cycle; rotate so not every year but all every cycle
• How embed in course assignment or service experience
Suggested Assessments
Examples for SLOs
Examples for SUOs
Capstone project or Final exam = direct assessment
methods integrate knowledge, concepts, and skills
associated with an entire sequence of study in a
course
Surveys (exit, satisfaction, perception) = ask
questions providing respondents an opportunity to
answer (open-ended questions) or choose a response
from finite set of answers (closed-ended questions);
completed orally or in writing; administered by
phone, mail, e-mail, via the web
Rubric = flexible direct assessment to score any
product or performance (e.g. essays, portfolios, skill
performance, oral speech, project/product creation,
etc.)
Internal & External Reporting Systems = established
or required software or database opportunities
External Audit Reports
PeopleSoft custom queries
Chancellor’s Office Reports (Data Mart, MIS, ARCC)
Reports from business intelligence tools
Pre-Test/Post-Test = direct assessment tool
administered at the beginning and at the end of a
course to evaluate the progress students make
Computerized Tracking System (log books) =
compilation of numbers of clients who use a service
and when can generate data for process functioning
Focus Groups = guided discussion whose intent is to
gather open-ended comments about a specific issue;
provides feedback from the clients who use the
services as to the performance and process; requires
a facilitator to ask questions and collect comments
Checklist for Assessment Tools
•
•
•
•
•
•
•
•
•
•
•
Embedded in everyday activities.
Authentically measures student skills, knowledge, or ability/service process.
Based upon clearly defined material.
Evaluated by explicit criteria.
Collects valid data concerning a specific outcome.
Provides reliable results or reproducible evaluative results.
Assessment criteria and expectations have been validated through discussion with
colleagues.
Systematically administered to students or clients.
Conforms to institutional mission and outcomes.
Reflects outcomes consistent with department or program outcomes.
Cost effective concerning resources, supplies and administration.
3. Establish criteria/expected
level of achievement
1. Minimum expectation for success (%, fraction,
actual number); 1st time determine a majority then
base expectations on previous assessment
information when available
2. What is the achievement standard (satisfactory, a
passing grade, numeric score, narrative indicator)
3. Which students/clients count in data (successful
completion of course only, all queried, those
identified from this to that date, random sample)
4. Gather and analyze
results of assessment
•
•
•
•
Gather, organize, aggregate, analyze, and report
Process protocol and who’s involved; periodicity; deadlines; regular cycle
Discussions at various meetings – need time
Aggregate
– Within the course/department/program
– Office of Institutional Effectiveness
• Analysis and evaluation
– Through discussion with vested individuals, which can mean within
and outside of the department/program
– Was the data valid and helpful; what does it say that is working well;
what could be improved
– Discuss the discipline issues
– Talk with other departments and community members that are part of
the holistic pathway through the program/unit
5. Actions Taken
• Based on results ascertain a step or steps to
improve student learning, service, or process
• Establish definite actions and who is
responsible
• Implement the actions and document them
on the assessment plan
• Identify follow-up process and timeframe
• On-going assessment cycle
Sample Document Layout
Intended Outcome
Assessment Task
Criteria /Expected
Level of
Achievement
Results of
Assessment
Actions Taken
Write the intended
outcome in this
column.
• Student Learning
Outcomes (SLOs) are
identified in the course
outline of record.
• Program Level
Outcomes
(instructional
SLOs/SUOs) are
identified statements
for a program/unit that
will be assessed to
measure the
effectiveness of the
program/unit.
Explain the assessment
task for each outcome
in this column. Make
sure that a balance of
direct and indirect
assessment methods
are used programwide. Describe what
the assessment is, who
will be responsible for
the task, how the
assessment task will
be implemented,
when the assessment
will be conducted, and
where activities will be
carried out.
The expectations for
success should be
explained in this
column. Establish a
minimum expected
score for achieving the
outcome (e.g. %,
fraction, actual
number), determine
the acceptable success
standard (e.g.
satisfactory, a C grade,
narrative indicator), as
well as the
students/clients to be
included (e.g.
successful completers,
all queried, random
sample).
Internal and external
conditions should be
considered.
Once the assessment
data is collected
analyze the
information through
small group discussion
and in relationship to
previous assessment
data (if available).
Report the actual
results and compare
with the original
expectations in this
column. Highlight key
findings from the data.
Develop supportable
conclusions from this
information.
Determinations will
then be made on what
action should be taken.
Report here what the
members have done as
a result of the findings.
Identify any changes
made, who has made
the modifications and
indicate the time
frame for for reassessment of the
outcome. The actions
should generally be
precise and detailed
and written in the past
tense; although there
are times when no
action is necessary and
this can be stated as
well.
Outcomes Assessment Loop
Intended Outcome
(SLO or SUO)
Actions Taken
(Modifications/Time Frame/Who
Implements)
Results of Assessment
(Results/Highlights/Conclusion)
Assessment Task
(What/How/When/Who/Where)
Criteria/Expected Level of
Achievement
(Criteria/Standards/Who is
Counted)