Transcript What’s Going On? Five Year Reviews and Assessment: Sharing Experiences so Far Ruth Guzley, Mike Ward and Bill Loker, AURA.

What’s Going On? Five Year
Reviews and Assessment:
Sharing Experiences so Far
Ruth Guzley, Mike Ward and Bill
Loker, AURA
Five Year Program Reviews:
The Broader Context …


Were routine, episodic efforts of little
utility, much futility
What do we want to know about our
programs?



What are we trying to accomplish?
How well are we meeting our goals?
Academic success, good use of resources
Assessment:
Doing the Right Thing for the Right Reasons
The Intentional Curriculum: Do individual
courses add up to a program?
 The Reflective Department: Are we working
together to provide a quality educational
experience for our students?
 The Culture of Evidence: How do we know
the answers to the above questions?
Assessment can provide these answers …

The Assessment Juggernaut

WASC reaccredidation … educational
effectiveness reviews …
http://www.wascsenior.org/wasc/Doc_Lib/2001%20Handbook.pdf

Cornerstones … CSU system-wide
accountability effort focused on quality of
Baccalaureate programs …
http://www.csuchico.edu/prs/AccountabilityReport_2002.doc

Educational effectiveness and other
stakeholders … students, parents, legislators,
etc
Aligning Five Year Review with
Assessment Demands …


New Five Year Review reflects
assessment concerns
Transforms FYR from an episodic time
sink into a cumulative exercise in
evidence gathering to



improve educational outcomes and
satisfy stakeholder accountability demands
Assessment only one component of FYR
From the FYR template …
Articulating a Collective Vision: Mission and
Program goals
Criteria for Review
1.1 The program has a clear and published mission statement that is
appropriate for higher education and consonant with the mission and
strategic priorities of the university.
1.3 The program periodically reviews and revises the mission
statement as appropriate. The review process involves appropriate
stakeholders.
Program Performance (Hint: Insert mission statement here.)
From the FYR template … (cont’d)
Organizing For Learning: Curriculum
Criteria for Review
2.2 The program has specified its expected learning outcomes and
they have been widely shared among its members, including faculty,
students, staff, and … external stakeholders.
2.5 The program has established processes for assessing student
learning outcomes and for assuring that students are achieving core
competencies for completion of the program.
Program Performance … (Hint: you don’t want a blank here.)
From the FYR template … (cont’d)
Organizing For Learning: Faculty Resources
Criteria for Review
3.1 Faculty recruitment, incentives and evaluation practices are
aligned with … program missions and educational objectives.
3.3 The deployment of faculty reflects the mission and program
goals …
Program Performance … (Hint: Need mission, goals and
outcomes statements to answer these questions.)
The FYR is consequential
3. Final Review and Approval of Continuous Approval
Plan
“The Review Team – consisting of an Office of the Provost
representative, College Dean, Department Chair, Program
Director, and an undergraduate student – will make one of four
recommendations to the Provost ….
1. Recommend continuation …
2. Recommend continuation , but cite specific concerns …
3. Recommend the program remain under continuing
review ….
4. Recommend suspension ….
(Hint: Outcome #4 not so good …)
The Right Thing, Right Reasons,
& Get a Jump on FYR





Aligning Mission, Goals, Student Learning
Outcomes and Curriculum is Good Practice
Best Practice engages in a process of
continuous assessment & improvement
SLO assessment is a necessary component of
FYR … you don’t want blanks!
Departments do things other than teach, but
excellence in undergraduate education is our
#1 strategic priority …
Assessment of SLOs: evidence of success
Toward A Comprehensive
Assessment Plan:




Program Mission
Goals
Student Learning Outcomes (SLOs)
Course Alignment Matrices
An Outcomes / Course Alignment Matrix
Course SLO 1
Stage 1 – Ensuring a
Cohesive Curriculum
In which courses are
particular outcomes
introduced (I),
practiced (P),
mastered (M)?
151
I
177
P
201
SLO 2 SLO 3 SLO 4
I
I
P
I
277
P
P
340
M - oral
M
presentation
477
P - term
paper
Assessment of SLOs

Direct Assessment Methods:





Published Tests
Local Exams
Embedded Assignments & Activities
Portfolios
Indirect Assessment Methods:



Surveys
Interviews
Focus Groups
Assessment of SLOs

Direct Assessment Methods:


Indirect Assessment Methods:


Examine Student Work …
Examine Student / Alumni Perceptions …
Effective Assessment Programs Use
Multiple Measures … Triangulate
Using Rubrics to “Objectively” Evaluate
Student Work …


Explicit Scoring for Student Work:

Holistic Scoring

Analytical Rubrics
Program Faculty Should Discuss, Debate,
Develop Some Consensus on What Methods
and Measures (Rubrics) to Use …
Evaluation and Use of Data for
Continuous Improvement

Assessment Data Must be Evaluated
Systematically, Periodically


Recommend Annual Collection / Assessment /
Action Plan
Modify Curriculum and Pedagogy to Improve Student
Learning …
 Typical Actions:



Modify Content, Sequencing of Course(s)
Modify Pedagogical Approaches
Follow-up Focus Group or Survey to
Triangulate …
Moving Toward Continuous
Improvement:

The Entire Process (Goals, Outcomes,
Assessment Measures) Must be Evaluated
Periodically

Recommend Five Year Re-evaluation to
synchronize with FYR process



Are we measuring reasonable work samples?
Is our process meaningful? Are we better informed
about what our students are learning or not?
Is our process sustainable? Can we be more efficient
and still have valid measures?
The Assessment Plan Design
Process …

Wherever you begin developing your
Assessment Program will NOT be the ideal
place to start …

You Must Begin Somewhere …

Iteration will save you ...



Continuous Refinement of Methods
Consensus and Team Building
Alignment of Program Resources with Program
Goals and Faculty Values …
Case Study: Mechanical Engineering
Program Mission
Mechanical Engineering Program Mission Statement
The Mechanical Engineering Program has the primary mission of providing a
high-quality undergraduate engineering education with
1. A curriculum that is firmly grounded in engineering fundamentals
2. A faculty that provides superior teaching and mentoring both in and out
of the classroom
3. A faculty whose focus is undergraduate education
4. Class sizes that encourage student participation
5. Project experiences that build on fundamentals and develop team skills
6. Facilities and equipment that are readily accessible
7. An environment that is conducive to learning and encourages students
from different genders and backgrounds
Note: Not Goals or Outcomes for Student Learning !
Mechanical Engineering Student Learning Outcomes
Mechanical Engineering Program graduates demonstrate the ability to:
A. Apply knowledge of math, science, and mechanical engineering to formulate
and solve problems.
B. Conduct experiments, and analyze and interpret the resultant data.
C. Design mechanical components and systems to meet desired needs, and
design experiments to evaluate system performance with respect to
specifications.
D. Use current mechanical engineering tools.
E. Communicate technical matters effectively in oral, written, and graphical
form.
F. Function effectively as members of multi-disciplinary teams.
G. Understand their professional and ethical responsibilities, the impact of their
activities on society and the environment, and appreciate contemporary
issues facing society.
H. Use information resources and recognize the importance of continued
learning.
Mechanical Engineering Embedded Assessment Matrix:
SLOs:
Courses
where
student
work is
measured
A B C D E F G H
Embedded Assessment of Outcome C in ME 138:
Design mechanical components and systems to meet
desired needs …
•Student grades on design projects are based on
assessment of 6 design projects: 3 individual, 3 group
•A standardized rubric is used to score each design
•The performance metric:
•A cumulative grade of C or better on design projects must be
attained to demonstrate the ability to design mechanical
components and systems (competence).
Other Measures of Mechanical
Engineering Outcome C (Triangulation!)
Outcome C: Design mechanical components and systems to meet
desired needs …
•Senior Exit Survey Question: How well prepared are
you to design a component or system to meet desired
needs?
•Mean Responses (4.29 Spring 2002, 4.0 Spring 2003)
•Scale: 5=Very Well Prepared – 1=Very Unprepared
•A mean response of 4.0 is considered acceptable to faculty