Transcript Fred Trapp Bob Pacheco Mary Allen      Ensure continuity in assessment when passing the baton Orient new faculty to program assessment Provide a historical record of.

Fred Trapp
Bob Pacheco
Mary Allen





Ensure continuity in assessment when
passing the baton
Orient new faculty to program assessment
Provide a historical record of changes and
their rationale
Inform subsequent assessments of the same
outcomes, such as using the same rubric or
calibration exemplars
Accessible warehouse of information to use
for program review or accreditation




Review of the currency of the outcomes
Review of the curriculum map
Review of annual assessment findings and
associated program changes
Discussion of the evidence of impact of
previous program changes





Students
Students
Students
Students
Students
understand economic theories.
can write and speak effectively.
complete a dissertation.
pass the licensure exam.
can conduct literature reviews.
Course
PLO1
PLO 2
PLO3
PLO4
100
X
X
X
X
200
X
X
X
X
300
X
X
X
X
310
X
X
X
X
312
X
X
X
X
340
X
X
X
X
400
X
X
X
X
410
X
X
X
X
425
X
X
X
X
430
X
X
X
X
490
X
X
X
X

The department chair assessed three
outcomes this year using exit interviews with
graduating students. She found that students
satisfactorily mastered all three outcomes.

We assessed students’ mastery of research
methods by calculating the average grade in
the Research Methods class. The average
grade was 2.96. This is good.

We selected 50 capstone projects at random,
calibrated 10 faculty on the use of an analytic
rubric with four dimensions, and assessed the
quality of students’ writing skills. Inter-rater
reliability estimates ranged from .87 to .93.

We decided that we’d be satisfied if at least
80% of the students scored at level 3 or
higher on each dimension of the rubric. We
were satisfied with students’ control of
syntax & mechanics and their use of sources,
but we were disappointed with their synthesis
of ideas and the overall organization of their
writing.

We learned that students are not explaining
theories at the level we expected. We decided
to close the loop by expanding the emphasis
on theory in each survey course (310, 312,
315, 316, and 321) and by placing more
emphasis on theory in exams in these
courses. Faculty will devote more time to
theory during class meetings, and at least
20% of students’ grades will now reflect their
ability to explain theories.


Does closing the loop require additional
budget for the program?
Have a procedure for allocating funds for
well-based budget requests.


Identify common outcomes with weak results
to identify the need for a campus solution,
such as the need to establish a Writing
Center, to expand ESL assistance, or to offer
faculty a WAC workshop.
Warehoused annual reports and program
reviews are easily accessed for integration
into WASC self studies and for visiting teams
to review prior to their visit. Annual reports
demonstrate sustainable, effective
assessment is in place.

How can the report writing experience:
◦ Help faculty explore the student learning process?
◦ Determine the extent to which the curriculum is
working?
◦ Where can time, energy and/or money be allocated
for continuous improvement in learning?
◦ Exploit the writing process and dialogue about
results to gain broader institutional learning
experiences? **
◦ Help meet our quality assurance pledge to the
community?
**Adriana Kezar ed. Organizational Learning in Higher Education New
Directions for Higher Education. No. 131, Fall 2005. Jossey-Bass.
15
1.
2.
3.
4.
5.
6.
7.
Assessment focus- course, program, general
ed, etc.
What outcomes were assessed?
How and when were they assessed?
Who was assessed?
What were the results?
Who reviewed the results, made sense of the
them and what conclusions were reached?
What are the implications for practice and/or
policy or future assessment work?

Promising Vehicles for Expanding
Information to the Public
◦ Brief narrative report from annual
assessment reports
◦ Simple statistical reports on learning
outcomes or surveys
◦ Best practices stories supported by
assessment
Peter Ewell. Accreditation & the Provision of Additional Information to the Public about
Institutional and Program Performance, CHEA, May 2004

National Institute for Learning Outcomes Assessment
(NILOA)
◦ 2010 Webscan report Exploring the Landscape: What
Institutional Websites Reveal About Student learning
Outcomes Assessment Activities
◦ 2010 Connecting State Policies on Assessment with
Institutional Assessment Activity
◦ 2011 Providing Evidence of Student Learning: A
Transparency Framework

Ease dropping
◦ www.learningoutcomeassessment.org
◦ Projects are at least three semesters long
◦ Individual and high-impact courses (all sections)
included
◦ Project proposal by a faculty group
◦ Course improvements based on data analysis
◦ Reassessment expected
◦ Results/report shared across the college and web posted

Ease dropping
◦ http://www.ccbcmd.edu/loa/CrseAssess.html
◦ Two-page executive summaries available

CHEM 108
◦ An initial “failure” turned to success and
collaboration with a four-year school

HLTH 101
◦ Addressing an achievement gap with professional
development and increased communication with
students

CRJU 101 and 202
◦ Statewide group assessment development effort and
creativity in the interventions used

Program learning outcomes data collected in a
student E-portfolio
◦ Directing internal and external evidence (1 to 10
measures)
◦ Indirect evidence (1 to 4 measures)

Evidence drawn from samples of student work
for faculty to apply an agreed upon holistic
rubric
◦ Eight general education outcomes (student success
skills)
◦ Discipline-specific exit competencies or outcomes

Hocking College example reports and
analysis
◦ Culinary Arts Technology (on cloud)
◦ Forestry Management Technology (on cloud &
college web)
◦ Nursing Technology (on cloud)

JFK University example report
◦ Counseling Psychology (on cloud)


General education studies completed 2007-08; 2005-06
◦ Numeracy
◦ Scientific inquiry
◦ Problem solving/critical thinking (2008-09)
◦ Information literacy
◦ Workplace skills (CTE) (2009-10)
General education studies completed 2006-07; 2004-05
◦ Arts & humanities
◦ Cultural diversity
◦ Oral communication
◦ Written communication

Ease dropping
◦ http://www.mesacc.edu/about/orp/assessment/ind
ex.html
◦ Annual reports and summaries available
 Nine years of history and experience
 14 years of assessment work

Ease Dropping
◦ Assessment work began in 1970
◦ http://assessment.truman.edu/
 Assessment Almanac- A compilation of results from each
year’s assessment work (versions from 1997 to 2009 are
posted)
 General Education outcomes are assessed in the context of
the major field of study
 Portfolio Project- required of all seniors to show best work assessed by
faculty for the nature & quality of the liberal arts and sciences learning
outcomes (versions from 1997 to 2008 are posted)

Why archive?
◦ Compliance vs. institutional learning

Where to keep the completed report?
◦ Decentralized vs. central office

Options for how to keep the reports?
◦ Paper
◦ Electronic templates
◦ Interactive database


To all affected participants
Campus committees
◦ Curriculum, assessment, resource allocation group, unit
(department) leadership, general academic and college
leadership





Campus fairs, brown-bag lunches, poster sessions for
information sharing
Faculty professional development programs
Accreditation self-study committee work groups
Local governing board presentation
College web site for the public

Feedback & recognition
◦ Feedback rubric for annual assessment
reports
 Conversations and action
 Collection and analysis of evidence
 Implementation of findings
◦ Recognition (achievement & excellence)
Ease dropping
https://www4.nau.edu/assessment


Seal of Assessment Achievement
Academic programs earning this recognition
have demonstrated in their annual report that
•
learning outcomes have been assessed through
two or more methods, and
•
findings have been discussed
among the faculty.


Seal of Assessment Excellence
Academic programs earning this recognition have
demonstrated
◦
◦
◦
◦
a thorough implementation of assessment plan(s)
the reporting of meaningful assessment data
the discussion of findings among faculty
and perhaps students
the use of findings to showcase
student achievements and
to make curricular adjustments.

Voluntary System of Accountability
◦ APLU & AASCU (520 public institutions, award 70% of
bachelor’s degrees in the US each year)
◦ College Profile (includes learning outcomes & links to campus)
 Proactive initiative to document learning gains and average
institutional scores (choice of 3 national instruments)
 Proactive initiative to illustrate unique campus learning outcomes
assessment work
 Promoting a learning institution
◦ Ease Dropping
 http://www.collegeportraits.org/
 CSUPomona
http://www.collegeportraits.org/CA/CPP/learning_outcomes

National Association of Independent Colleges
and Universities
◦ Assessment programs on campus tied to
institution’s mission
◦ Ease Dropping
 http://www.naicu.edu/special_initiatives/accountability
/Student_Assessment/id.514/default.asp
 Pepperdine University
 http://services.pepperdine.edu/oie/learningoutcomes/learning-outcomes-overview.aspx

Filesanywhere.com
http://www.filesanywhere.com/fs/v.aspx?v=8a69668b5c6773a96f6d

Mary Allen (independent consultant)
◦ [email protected]

Robert Pacheco
◦ [email protected]

Fred Trapp (Cambridge West
Partnership)
◦ [email protected]
Questions and Comments