Transcript Assessment - Suffolk County Community College

SCCC Comprehensive Assessment Plan for Institutional
Effectiveness (CAPIE)
Workshop
Spring, 2007
Dr. Tina Good
Dr. Allen Jacobs
Dr. Louis Attinasi
1
CAPIE Workshop Agenda

PowerPoint Presentation
–
–
–


2
Assessment as a Process, Tina Good
Goals and Objectives, Allen Jacobs
Assessment Tools, Lou Attinasi
Hands-on practice developing goals and objectives
(with worksheets)
Discussion about developing unit assessment plans
and timelines (with forms and templates)
Follow-up CAPIE Meetings and
Workshops




3
Meetings with units to assist with development of
assessment plans and timelines prior to
implementation
Regular contact with units during implementation of
plans
Workshops on interpreting and reporting assessment
results
Workshops focusing on connecting unit goals and
objectives to new strategic plan
College Assessment Resources





4
Office of Institutional Effectiveness
Office of Curriculum and Assessment
Assessment Advisory Council
Strategic Planning Council
CAPIE Website
Assessment
a process
5
What does Assessment as a Process
mean?

Process
–
The fact of going on or being carried on, as an
action, or a series of actions or events; progress,
course.
–

Process requires
–
–
–
6
Oxford English Dictionary
Action
Reflection
Revision
What does Assessment as a Process
mean?

Assessment
–
the ongoing process of establishing clear,
measurable expected goals and objectives;
ensuring that there are sufficient opportunities for
affected areas to achieve those outcomes; and
systematically gathering, analyzing, and
interpreting evidence to determine the degree to
which outcomes match expectations
–
7
CAPIE
What does Assessment as a Process
mean?


8
Assessment ≠ Test
Process ≠ Product
Assessment

9
The Anxiety
Why does the word “assess” cause
anxiety?

to assess:
–
–
–
–
To estimate the value of (property) for taxation.
To charge (a person or property) with a special
payment, such as a tax or fine.
To charge (a foul or penalty) against a player,
coach, or team. 5.
To determine the value, significance, or extent of;
appraise.
–
10
The American Heritage Dictionary
Why does the word “assess” cause
anxiety?

Accountability and Improvement
–
justifying the investment of resources by
providing substantive evidence and analysis
which demonstrates that the investment yields
significant results
–
–
11
CAPIE
To do something for the purpose of improving
suggests automatically a fundamental bias in the
process—the assumption that improvement is
needed.
Assessment as Opportunity
1. A forum for articulating to ourselves and to
our constituent groups what we value.
2. A forum for collaborating with constituent
groups such as students, faculty, taxpayers,
SUNY, accrediting agencies, etc., on what
should be valued.
12
Assessment as Opportunity
3.
A systematic method for using assessment
results



13
To recognize and celebrate what we are doing well
To inform planning and resource allocation decisions
To improve programs and services
Assessment Plan for Institutional
Effectiveness


How well are we doing what we say we are
doing?
The effectiveness of an institution rests upon
the contribution that each of the institution’s
programs and services makes toward
achieving the goals of the institution as a
whole.
–
14
Middle States Commission on Higher Education
Four-Step Planning-Assessment Cycle
1. Defining clearly articulated institutional and unit-level
goals;
2. Implementing strategies to achieve those goals
3. Assessing achievement of those goals; and
4. Using the results of those assessments to improve
programs and services and inform planning and
resource allocation decisions.
–
15
Middle States Commission on Higher Education
Evaluation
1. The part of the assessment process which
uses professional judgment to form
conclusions about the data.
2. Using assessment information in
combination with professional judgment to
make appropriate decisions about what has
been assessed
–
16
CAPIE
How do you measure the abstract?


17
Develop goals first then figure out how to measure to
what degree you are achieving them.
Do not confuse assessment measures, tools,
indicators (retention, graduation, transfer, and
placement rates, financial ratios, surveys, cost
analyses, etc.) with assessment processes
(gathering quantitative and qualitative evidence to
determine the degree to which you are achieving
your goals).
The Myth and The Metaphor
Love is like a rose.
18
Multiple Measures

Because there is no one perfectly accurate
assessment tool or strategy, institutions should use
multiple measures to assess achievement.
–
–
–

Quantitative/Qualitative
Direct/Indirect
Benchmarks
All assessment tools and strategies should clearly
relate to the goals they are assessing
–
19
Middle States Commission on Higher Education
Characteristics of Assessment
Processes





Useful
Cost-effective
Reasonably accurate and truthful
Planned
Organized, systematized, and sustained
–
20
Middle States Commission on Higher Education
How do we build a Culture of
Assessment








21
Keep it simple
Do not assess everything
Do not reduce assessment process to measuring
only that which is most easily measured
Build on what we already have
Develop reasonable plans and timelines
Understand there are multiple audiences when
developing reports
Link assessment results with planning
Keep process open and transparent
College of Excellence
Assessment + Planning = Institutional Effectiveness
22
Goals and Objectives…
where assessments begin
23
TERMINOLOGY….









24
Outcomes
Mission
Goals
Objectives
Standards
Aims
Behavioral Objectives
Cognitive Objectives
Affective Objectives









Learning Goals
Learning Objectives
Behavioral Outcomes
Desired Outcomes
Intended Outcomes
Achieved Outcomes
Process Standards
Content Standards
Value standards
Terminology in Turmoil
25
Translucent Terminology




26
Lack of precision
Confusion with non-technical uses
Overlap in meaning
Inconsistent application
As a solution…
Keep it simple.
Keep it consistent.
27
The Overall Concept – statements of purpose
Broad
C
O
N
T
I
N
U
U
M
MISSION
Not Measurable
GOALS
OBJECTIVES
Measurable
Narrow
28
Action Plan
Steps taken to achieve an
objective.
29
MISSION




30
The broadest statement of the purposes of an
institution.
The starting point for programmatic development
and assessment.
More than just decoration.
The ability of an institution to achieve its mission
= institutional effectiveness.
Institutional Mission Statement
Suffolk County Community College promotes
intellectual discovery, physical development,
social and ethical awareness, and economic
opportunities for all through an education that
transforms lives, builds communities, and
improves society.
31
Programmatic Mission Statement



32
The broadest statement of the purposes of
an institutional program or unit.
Can be more restricted than the institutional
mission statement.
Must derive from or reflect some aspect or
aspects of the institutional statement.
GOALS



33
Add some specificity to the aims of the
mission.
They retain breadth.
They are not typically couched in language
that lends to assessment, i.e., they are not
directly measurable.
OBJECTIVES




34
Like mission and goals, objectives are statements of
aim, purpose, or outcomes; however, they are
narrow in scope and specific.
Objectives link to goals.
They are written with language that allows for
measurement.
Achieving objectives implies the achievement of
goals.
The Metaphor
35
Writing goals & objectives…
...it’s all about the choice of words.
36
Writing goals…




37
Abstract
Can be lofty, florid, emotional, etc.
Broad, global, general, etc.
Typically, not measurable
Writing objectives…





38
Concrete
Can be mundane, unadorned,
dispassionate, etc.
Narrow, specific, explicit, etc.
Always measurable
Time bound
The Language of Goals








39
Appreciation
Understanding
Knowledge
Comprehension
Critical thinking
Ethical awareness
Improving society
Communication skills








Competency
Happiness
Excellence
Life-long learning
Community service
Professionalism
Good citizenship
Physical development
DEVELOPING GOALS:
Some Things to Consider…




40
What do we value? (Goals are important.)
Is this a cumulative outcome? (Goals
generally require that several objectives be
met.)
Is there a defined endpoint? (Goals are
generally ongoing.)
Is it long-term? (Achieving goals usually
takes time.)
The Language of Learning Objectives








41
Name
Define
Classify
Explain
Use
Illustrate
Write
Demonstrate








Compare
Experiment
Calculate
Create
Argue
Predict
Select
Estimate
More Language
(for non-learning objectives)






42
Increase
Decrease
Collect
Implement
Develop
Promote






Distribute
Gather
Establish
Inform
Create
Communicate
The Components of an Objective
43

Audience:
Who?

Behavior:
What?

Condition:
How?

Degree:
How much?

End point:
How long?
Audience: Who?
State specifically who will be performing…
All Students
ESL Students
All Faculty
Business Faculty
All Graduates
Nursing Graduates
44
Behavior: What?


45
State specifically what it is you expect the
audience to do.
Employ terms to identify behaviors that you
can see, hear, touch, smell, or taste, i.e., use
measurable verbs.
Condition: How?



46
State the circumstances under which the
behavior will occur.
State what can or cannot be used in the
performance.
A statement of the condition is often omitted
if the condition is obvious.
Degree: How much or how well?


47
A statement of acceptable performance, e.g.,
In how much time? or To what degree of
accuracy? or To what level?
This is sometimes omitted if it is obvious.
End Point: How long?


48
Objectives reference a timeframe in which
they are to be met.
Timeframes need to be reasonable and
realistic.
Kinds of Objectives
49

Learning Objectives

Programmatic/Unit Objectives
Types of Learning Objectives



50
Cognitive: intellectual learning and problem
solving, what the student is able to do
(Student will be able to…)
Affective: feelings or emotions, how the
student chooses to act (Student will choose
to…)
Psychomotor: physical skills, what the
student can perform (Student will be able
to…)
Types of Programmatic/Unit Objectives
51

Operational – support institutional goals

Strategic – support strategic goals
Examples
Goal:_______________________________________________________________________
________________________________________________________________________
________________________________________________________________________
Objective 1:
Audience (Who?)
Behavior (What?)
Condition (How?)
Degree (How much or how well?)
End point (How long?)
( _Cognitive _ Affective _ Psychomotor)
( _ Operational _ Strategic)
_____________________________
_____________________________
_____________________________
_____________________________
_____________________________
Written Objective:_____________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
52
Learning Objective
Goal :__Students will have a basic understanding of Spanish grammar.______________
________________________________________________________________________
________________________________________________________________________
Objective 1:
Audience (Who?)
Behavior (What?)
Condition (How?)
Degree (How much or how well?)
End point (How long?)
( X Cognitive _ Affective _ Psychomotor)
( _ Operational _ Strategic)
_Students successfully completing SP11
_Conjugate irregular ar, er, & ir verbs___
_From a list of 50 infinitives___________
_With 80% accuracy_________________
_One semester______________________
Written Objective: __Given a list of 50 irregular ar, er, and ir verbs, students who______
successfully complete SP11 will be able to write the conjugations of these verbs_
with 80%_accuracy._____________________________________________________
______________________________________________________________________
53
Learning Objective
Goal : _Nursing Program graduates will be proficient at performing venipunctures.____
Objective 1:
Audience (Who?)
Behavior (What?)
Condition (How?)
Degree (How much or how well?)
End point (How long?)
( _Cognitive _ Affective X Psychomotor)
( _ Operational _ Strategic)
_Graduates of the Nursing Program_______
_Inject a solution into a vein_____________
_With a 20 gauge, 1 ½” hypodermic needle
_and 3 cc syringe______________________
_On first try, 9 out of 10 times____________
_Upon completion of Nursing Program_____
Written Objective: __Given a 20 gauge, 1 ½” hypodermic needle and a 3 cc syringe,___
Nursing Program graduates will be able to administer an intravenous injection on
their first attempt 9 out of 10_times._______________________________________
54
Actual Example 1
Outcome Behaviors:
Upon completion of theatre
workshop, students will have
learned:
1) techniques of script analysis in
preparation for a production
2) the relationship of your specific
discipline to all other disciplines
in a given production
3) to develop standards and
practices for rehearsals and
performances in all disciplines.
4) to sustain the rehearsed
production through the
performance schedule.
55
Outcome Behaviors:
Upon completion of theatre
workshop, students will be
able to:
1) Analyze scripts in preparation
for a production
2) discuss the relationship of your
specific discipline to all other
disciplines in a given production
3) develop standards and
practices for rehearsals and
performances in all disciplines.
4) sustain the rehearsed
production through the
performance schedule.
Actual Example 2
COURSE OBJECTIVES
By the end of the semester, the
students will be able to:

Know their minimum and maximum
target heart rate (THR)

Know when they are working within
their THR and be able to do so for
20-30 minutes

Participate in aerobic combinations
helping to improve their
cardiovascular system

Have a better understanding of
weight control, nutrition and fat
reduction

Select a safe and appropriate
aerobic program
56
COURSE OBJECTIVES
By the end of the semester, the
students will be able to:

Cite their minimum and maximum
target heart rate (THR)

Identify when they are working
within their THR and be able to do
so for 20-30 minutes

Participate in aerobic combinations
helping to improve their
cardiovascular system

Formulate a nutritionally
balanced diet aimed at weight
control and fat reduction.

Select a safe and appropriate
aerobic program
Actual Example 3
COURSE OBJECTIVES
Each student, upon completing
the course, should have an
understanding of the
following essential skills
pertaining to:
I. The topics of techniques of
playing the guitar
II. The topics of music theory
for guitar
III. Sight-reading music for
guitar
IV. Guitar Harmony
57
COURSE OBJECTIVES
Upon completing the course,
students will be able to:
I.
Play xyz on the guitar (to
what degree of accuracy?).
II. Discuss topics (specify) of
music theory for guitar.
III. Sight-read music for guitar
(level of difficulty?).
IV. Discuss guitar harmony.
Programmatic Objective
Goal:_ Support the food-service industry._____________________________________________________
_________________________________________________________________________________________
Objective 1:
Audience (Who?)
Behavior (What?)
Condition (How?)
Degree (How much or how well?)
End point (How long?)
( _Cognitive _ Affective _ Psychomotor)
( X Operational _ Strategic)
The Culinary Program__________________
Increase the number of program graduates
_____________________________________
By 10 students a year _________________
For the next 5 years____________________
Written Objective: __The Culinary Arts Program will increase the number of its graduates by 10 students
a year for the next five years.____________________________________________________________
Action Plans:
_____________________________________________________________________________________
_____________________________________________________________________________________
58
Facilities Objective (a poor example)
Goal: _Provide an environment conducive to learning.______________________________
________________________________________________________________________
Objective 1:
Audience (Who?)
Behavior (What?)
Condition (How?)
Degree (How much or how well?)
End point (How long?)
( _Cognitive _ Affective _ Psychomotor)
( X Operational _ Strategic)
College HVAC/R personnel___________
_Install air conditioners______________
_While classes are not in session______
_In all non-air conditioned classrooms_
_By mid-August of this year__________
Written Objective: _By August 15 of this year, at times when classes are not in session,_
college HVAC/R personnel will install air conditioners in all classrooms that are not
currently air conditioned.__________________________________________________
59
Facilities Objective (a better example)
Goal: _Provide a comfortable environment for learning and working.______________________________
_____________________________________________________________________________________
Objective 1:
Audience (Who?)
Behavior (What?)
Condition (How?)
Degree (How much or how well?)
End point (How long?)
( _Cognitive _ Affective _ Psychomotor)
( X Operational _Strategic)
_All students and personnel______________________
_Experience temperatures of 68-72 degrees_________
_____________________________________________
_Whenever they occupy indoor spaces at the College
_By August 15, 2007___________________________
Written Objective: _All students and personnel will experience temperatures of 68-72 degrees, whenever they
occupy indoor spaces at the College, by August 15, 2007.______________________________________
Action Plans: a. Install air conditioners in all non-air-conditioned rooms by August 15, 2007.________________
b. Repair broken thermostats by August 15, 2007_______________________________________
60
Human Resources Objective
Goal: Promote employee effectiveness.__________________________________________
________________________________________________________________________
________________________________________________________________________
Objective 1:
Audience (Who?)
Behavior (What?)
Condition (How?)
Degree (How much or how well?)
End point (How long?)
( _Cognitive _ Affective _ Psychomotor)
( X Operational _ Strategic)
_Personnel division of HR________
_Update all job descriptions_______
_In consultation with administrators
_To reflect current job expectations
_By end of next year_____________
Written Objective: __In consultation with appropriate administrators and supervisors,_
the personnel division of Human Resources will update the job descriptions of all
employees to reflect their current job expectations by the end of next year._______
61
Financial Affairs Objective
Goal: __Promote data-driven decision making by administrators._________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
Objective 1:
Audience (Who?)
Behavior (What?)
Condition (How?)
Degree (How much or how well?)
End point (How long?)
( __Cognitive __ Affective ___Psychomotor)
( _ Operational _X_ Strategic)
The Office of Financial Affairs______________
Develop formulas for non-tangible costs_____
By analyzing expenditures over last 5 yrs.___
Including costs for energy, security, time, etc.
By August 31 of this year__________________
Written Objective: _By August 31 of this year, the Office of Financial Affairs will develop formulas for
computing non-tangible costs, including costs for energy, security, custodial care, and_______
administrative time, by examining expenditures made in these areas over the last 5 years._____
62
Assessment Tools
an overview
63
Assessment Plan
What is it?
•
•
64
Agreement about the measures and processes to be
used
Indicates a readiness to act on the basis of a shared
understanding
Assessment Plan
Elements
•
•
•
•
•
•
65
Subject Matter for Assessment
Methodology
Time Line
Use of Assessment Information
Provisions for Administering the Plan
Provisions for Assessing the Assessment
Assessment Plan
66
•
The plan will specify an assessment schedule
•
Note: No institution/unit can be expected to
assess everything at the same time
Assessment Plan
Decisions must be made about:
•
•
•
•
67
How plan will be formatted
How plan will be evaluated
How much it will cost to implement it
What should be the starting point
Assessment Plan
Key Point:
•
68
There is a clear order to the assessment process:
formulation of the intended educational outcomes or
administrative objectives must precede the
identification of the means of assessment.
Current Assessment Efforts



69
General Education Assessment
Academic Program Reviews
Student Services Program Reviews
Academic Assessment
General Education – Achievement of Learning
Outcomes
Direct Measures – provide direct evidence of outcome
achievement
•
•
•
•
•
•
70
Standardized Tests
Local (Criterion-referenced) Tests
Qualitative scoring rubrics applied to written work, oral presentations,
etc.
Imbedded measures
Portfolios
National Licensure and Certification Exams
Academic Assessment
General Education – Achievement of Learning
Outcomes
Indirect Measures – provide indirect evidence of
outcome achievement
•
•
•
71
Questionnaires
Interviews
Focus groups
Academic Assessment
Academic Program Review
Involves assessment of:
•
•
•
•
•
72
Program goals and objectives (currency, mission-relatedness)
Curriculum & Instruction (currency, effectiveness)
Students (learning; enrollment, graduation & transfer rates; satisfaction)
Resources (library, technology, instructional & office space)
Staffing (level of full-time faculty, professional development of faculty)
Academic Assessment
Academic Program Review
Involves the following assessment tools:
•
•
•
•
73
Benchmarking with peer institutions (curriculum, staffing)
Standardized tests, local rubrics (learning)
Surveys (curriculum, students)
Institutional data (students, resources, staffing)
Assessment of Student Affairs
Units
Student Affairs Program Reviews
Have involved the following assessment tools (indirect
measures):
•
•
74
Commercial Self-study Questionnaires
Surveys of Students & Faculty
Assessment of Student Affairs
Units
Student Affairs Program Reviews
Could involve direct measures, e.g., direct evidence of
student development as a result of student affairs
programming
•
•
•
75
Standardized tests
Local tests
Other demonstrations of skill or knowledge attainment (e.g.,
portfolios)
Need to Assess Non-Academic
Areas
76

We cannot know how well we are doing—or
how we can do better—in non-academic
areas of the College unless we regularly and
systematically assess our activities in these
areas

Our institutional accrediting body—Middle
States Commission on Higher Education—
requires assessment of all areas of SCCC
Assessment Tools for NonAcademic Units

Direct Measure of Effectiveness

Objective: By Spring 2008, the Business Office will have
reduced by an average of five working days the time it
takes to process a travel reimbursement request form.
Method: Use of institutional transactional database to
determine the average time it took to process a travel
reimbursement request form in Spring 2007 with the
length of time it took in Spring 2008.
Measure: Average length of time it takes to process a
travel voucher


77
Assessment Tools for NonAcademic Units

Attitudinal Measure of Effectiveness

Objective: Ninety percent of the Business Office’s
customers during AY 2007-2008 will indicate satisfaction
with the overall service they were provided by the office.
Method: Survey of customers
Measure: Percentage of survey respondents who
indicated they were “satisfied” or “very satisfied” when
asked on the survey how satisfied they were with the
overall service they received from the Business Office


78
Assessment Tools for NonAcademic Units

Effectiveness based on Meeting Professional
Standards

Objective: By Spring 2008, the Business Office will be in
100% compliance with all standards set by the National
Association of College and University Business Officers
for business office processes
Method: Gap analysis involving a comparison of
Business Offices processes with NACUBO standards.
Measure: Extent of discrepancy between NACUBO
standards for processes and status of Business Office
processes


79
Assessment Tools for NonAcademic Units

Effectiveness based on Best Practices
(Benchmarking)

Objective: By Spring 2008, the effectiveness of the
Business Office’s accounting practices will place it among
the top 25% of community colleges in NACUBO
Method: Benchmarking against institutions that
participate in the annual NACUBO survey of member
institutions
Measure: Business Office’s placement among the
community colleges responding to NACUBO survey


80
Assessment Tools for NonAcademic Units
Other Units (e.g., Business Affairs, Facilities, Human Resources, Institutional
Advancement, Institutional Effectiveness, Workforce Development)
Typical measures
•
•
•
•
•
81
Institutional or Unit Data that demonstrate improved efficiency or effectiveness
User-satisfaction and other attitudinal surveys
Benchmarking against standards set by professional groups (e.g., NACUBO)
Benchmarking against units in comparable institutions (e.g., NCCBP, NACUBO, Educause, ,AACC Digital
CC Survey)
Performance Indicators
Validity and Reliability
Two Important Characteristics of Good
Assessment Measures
82
•
Validity: the extent to which assessment procedures and
instruments accurately measure what they are designed to
measure
•
Reliability: the extent to which assessment procedures and
instruments consistently measure what they are designed to
measure
Validity and Reliability
Example:
Unit X wants to determine its customers’ satisfaction over the last
6 months?
Which survey item is a valid measure?
Item 1: “Overall, how satisfied are you with the service Unit X has
provided you over the last 6 months? Very dissatisfied → Very
satisfied
Item 2: “How often have you used services provided by Unit X
over the last 6 months? Very infrequently → Very frequently
83
Middle States Assessment Model
Middle States Model for Comprehensive Assessment
84
Questions?
85