Transcript Presentation to Adjunct Faculty Group

From Here, Go Anywhere
Overview of the Reporting Process
Program/Discipline Review
and Outcomes Assessment Report
“Effective assessment must begin
with real concerns of the
stakeholders and result in useful
information and recommendations
related to the purpose of
assessment.”
– D.W. Farmer & E.A. Napieralski
Dr. Michael E. Bryan, Coordinator
Office of Student Outcomes Assessment
May 12, 2004
From Here, Go Anywhere
Intended
Outcomes
Audience will understand the “Many faculty have been ‘doing
assessment’ for their own sake,
Program/Discipline Review
and have not been overwhelmed
Initiative including:
History and driving forces
Supporting resources
Reporting process (e.g.,
deadline, schedule, tasks)
Report format with various
components (e.g., intended
outcomes, assessment
measures, analysis)
Various uses (e.g., program-,
discipline-, and course-level)
with the task, without particular
experience in evaluation
methodology.” – O.J. Nichols (1995)
From Here, Go Anywhere
What is Assessment?
The systematic collection and analysis of
information to improve student learning.
“…a learner-centered, teacher-directed approach designed to
improve student learning in the individual classroom.”
– B.D. Wright (1991)
From Here, Go Anywhere
Assessment and Grades
Grades are global evaluations that represent
the overall proficiency of students. They
don’t tell you about student performance on
individual learning goals.
“…assessment involves taking a
second look at materials generated
in the classroom so that in addition
to providing a basis for grading
students, these materials allow
faculty to evaluate their teaching.”
– B.D. Wright (1991)
Why Assess?
From Here, Go Anywhere
“…assessment involves taking a second look at materials generated in the
classroom so that in addition to providing a basis for grading students, these
materials allow faculty to evaluate their teaching.” – B.D. Wright (1991)
Rationale and Driving Forces
“The institution demonstrates that each educational
program for which academic credit is awarded is (a)
approved by the faculty and the administration, and
(b) establishes and evaluates program and learning
outcomes.”
“The institution places primary responsibility for
the content, quality, and effectiveness of its
curriculum with its faculty.”
Principles of Accreditation, Standards for All Educational Programs,
Section III, 1 and 12, August 2003
From Here, Go Anywhere
Reporting
Mandates
“Reports of Institutional Effectiveness (ROIE) is an
annual report required by the State Council of Higher
Education for Virginia (SCHEV) and is intended to
provide meaningful information on the academic
quality and operational efficiency of Virginia’s public
institutions of higher education.”
“Each institutional report is organized into five
sections: institutional mission, college profile
measures, system-wide measures, and institutionspecific measures, and the core competencies.”
ROIE Homepage, http://research.schev.edu/roie/
“One of the most untruthful things possible, you know, is a
collection of facts, because they can be made to appear so many
different ways.” - Karl A. Menninger
From Here, Go Anywhere
Reporting
Mandates
Assessment processes at VCCS colleges have been
mandated for the past eighteen years. Senate Document No.
14, 1986 Virginia General Assembly, provided the legislative
mandate for assessment reporting and SCHEV established
the initial guidelines for colleges and universities to respond.
The Final Report of the Governor’s Blue Ribbon
Commission on Higher Education (February 2000) called for
the creation of a Quality Assurance Plan to:
(1) define core competencies for written communications, oral
communications, quantitative reasoning, scientific reasoning,
critical thinking, and information literacy;
(2) identify measures to assess students’ knowledge and skills;
(3) provide a vehicle to present results publicly.
Report of the VCCS Task Force on Assessing Core Competencies,
July 18, 2002
From Here, Go Anywhere
Local Reporting
Mandates
“Annual reports will be produced to account for what
has been accomplished in enhancing current
programs and implementing new ones, and to identify
emerging developments that call for updates to the
curriculum planning process.”
TCC Comprehensive Five-Year Curriculum Plan (2003-2008), Page 1,
August 2003
“While you are
experimenting, do not remain
content with the surface of
things. Don't become a mere
recorder of facts, but try to
penetrate the mystery of their
origin.” - Ivan Pavlov
From Here, Go Anywhere
Local Reporting
Mandates
Curriculum planning at TCC will occur in accord with the following principles:
The curriculum will be structured and articulated in ways that provide students with
multiple opportunities for connection and advancement as they enter, progress
through, exit, and potentially return to the college.
The curriculum will be responsive to the distinctive needs of business, industry,
government, agencies, and the larger community, continually reinventing itself in
relation to the immediate requests and long-term developments in various
occupational and technical fields.
The curriculum will be developed in consultation with employers and transfer
institutions that are the direct recipients of graduates.
All curricula will be built and maintained on a foundation of excellence. “In the
college transfer area, the college will benchmark itself to the most rigorous
standards of achievement in 4-year baccalaureate degree-granting institutions. In
the fields of occupational-technical study, the college will look to business and
benchmark its performance goals to the standards of the relevant industries
themselves” (Bearings on the Future: The TCC Strategic Plan).
TCC Comprehensive Five-Year Curriculum Plan (2003-2008), Pages 1 & 2,
August 2003
From Here, Go Anywhere
Local Reporting
Mandates
TCC will pursue the following corresponding strategies:
The college will regularly review programming at peer community colleges across
the nation with the goal of identifying current and emerging curricular areas for
possible development and implementation. As resources permit, faculty and
administrative teams will… bring back model programs and practices for adaptation
and implementation at TCC.
Through focus groups, advisory committees, and other means of formal and
informal relationships with employers and the higher education community, the
college will actively solicit input for needed program enhancements and development
as well as communicate the range and availability of existing programs.
The college will actively partner with those who are the primary stakeholders in the
successful outcome of students, looking for ways to engage their support and
sponsorship of students throughout the learning process.
The college will put into place a systematic process for assessing programmatic
and student learning outcomes, the results from which will be used for continuous
improvement of disciplines and programs. Other college activities, processes, and
initiatives (e.g., faculty hiring, adjunct faculty initiative, accreditation, professional
development and renewal, budget development, grant proposals, major gifts
campaign) will be aligned with the pursuit of curricular excellence.
From Here, Go Anywhere
Recap Why We
Assess
External Needs
Requirements for SACS Accreditation
Report of Institutional Effectiveness for
SCHEV
Quality Assurance Plan for VCCS
Internal Needs
Curriculum Planning Process &
Annual Reports for TCC
Intellectual Curiosity
Professionalism
From Here, Go Anywhere
Consultation Services provided by
the Coordinator for the Office of
Student Outcomes Assessment
(OSOA)
Assessment-centered assistance
such as word-smithing intended
outcomes or developing assessment
measures
Related pedagogical assistance
such as instructional strategies,
syllabus design, recordkeeping or
tracking data
OSOA Website
http://www.tcc.edu/welcome/
collegeadmin/OIE/SOA/
index.htm
Supporting
Resources
Reporting
Process
From Here, Go Anywhere
Tasks
“Lead Deans, in consultation with program/discipline faculty,
complete in its entirety the Program/Discipline Review and
Outcomes Assessment Report for each of their respective
programs and/or disciplines scheduled for review.”
Approved by the Vice President for Academic and Student Affairs on April 9, 2004
http://www.tcc.edu/welcome/collegeadmin/OIE/SOA/review/Process.htm
Provosts screen the reports and offer feedback prior final
submission to the Vice President for Academic and Student
Affairs.
“One important distinction in assessment methods is between
techniques that directly determine whether students have
mastered the content of their academic programs and those that
ask students to reflect on their learning.” – C.A. Palomba & W.
Banta (1999)
From Here, Go Anywhere
Reporting
Process
Annual Deadlines
April 1st – Lead Deans submit draft report to Provost(s) for
review.
May 1st – Lead Deans submit final report to Vice President
for Academic and Student Affairs.
Scheduled Program/Disciplines
4 Programs and 2 Disciplines were in the planning phase
last year, but are in the implementation phase this year.
An Additional 15 Programs and 24 Disciplines are in the
planning phase this year (2003-04), and will implement their
review and assessment plan next year (2004-05).
Report Format
From Here, Go Anywhere
Sections to the
Plan
Background/
Descriptive Section
Programmatic
Outcomes Section
Student Learning
Outcomes Section
Other Noted Data
Section
Report Format
From Here, Go Anywhere
Elements to the Outcomes Sections
Intended Student Outcomes: Knowledge, skills, or values students
should be able to demonstrate after completing your program or
discipline.
Intended Programmatic Outcomes: Non-student aspects that
contribute to your program’s viability.
Assessment Measures: Tool or indicator that yields the most timely
and accurate data used to determine attainment of a stated outcome.
Minimum Competency: A standard that represents a satisfactory,
acceptable-level of performance.
Actual Results: The real results of the data collected for each
assessment measure.
Highlights: Analysis and cited evidence that confirms outcome
attainment.
Enhancements: Analysis and cited evidence that identifies areas for
improvement; strategies planned for the coming year to address
these areas.
Enhancement Update: Describe the impact and level of success of
each enhancement activity (COMING NEXT YEAR).
Report Format
From Here, Go Anywhere
Working with the Form
MS Word Template  Online Form
Simply download the form from the OSOA
website to begin working with it
http://www.tcc.edu/welcome/collegeadmin/
OIE/SOA/review/Form.htm
Demonstrate functions, then we’ll get back
to writing outcomes and selecting
assessment measures
Step #1:
Outcomes
Writing Intended Student Learning
Outcomes
From Here, Go Anywhere
Outcomes describe specific learning behaviors that
students should exhibit in the context of the
program/discipline/course.
Outcomes are the specific skills, values and attitudes
students should exhibit that reflect the broader goals
(e.g., for students in a freshman writing course, this
might be “students are able to develop a cogent
argument to support a position”).
Often in the assessment literature, “objectives”and
“outcomes” are used interchangeably.
“A student drove himself so hard that he missed the learning curve.”
- Pun of the Day
Step #1:
Outcomes
Effectives Outcomes
From Here, Go Anywhere
Use action words that specify definite,
observable behaviors.
Indicate an appropriate level of attainment.
Are assessable through one or more
indicators.
Comprehensively and meaningfully define
a goal.
“Two mathematicians
arguing about even
Are realistic and achievable. numbers were at
odds.”
Use simple language.
– Pun of the Day
From Here, Go Anywhere
Step #1:
Outcomes
From Here, Go Anywhere
Step #1:
Outcomes
Step #1:
Outcomes
Difficulty Writing Outcomes?
From Here, Go Anywhere
For each of your stated goals, what are the
specific student behaviors, skills, or abilities that
would tell you this outcome is being achieved?
Ideally and briefly, what would a skeptic need
(what evidence needs to be present, what
specific behavior needs to be visible) in order to
see that your students are achieving the major
outcomes you have set out for them?
In your experience, what evidence tells you when
students have met these outcomes – how do you
know when they’re “getting” it?
Step #1:
Outcomes
From Here, Go Anywhere
Curriculum Mapping
Program
Goal
Program
Outcome
Required Courses
Depth of Coverage
Basic Introduction or
Outcome Review
1
Advanced Application
Goal 1
Basic Introduction or
Outcome Review
2
Advanced Application
Basic Introduction or
Outcome Review
3
Advanced Application
Goal 2
Basic Introduction or
Outcome Review
4
Advanced Application
ABC
101
ABC
110
ABC
200
X
X
X
ABC
210
ABC
220
X
X
X
X
X
X
X
X
X
X
X
X
ABC
295
X
X
X
X
From Here, Go Anywhere
Step #2:
Measures
Identify & Review Existing
Assessment Measures
What information on student learning/performance do you
currently collect? What assessments are administered?
Direct and Indirect Measures
How do these data sources relate to your newly articulated
outcomes? Does the data provide rich understanding?
Are there gaps between the information you collect and
your outcomes?
What other information do you need to have to understand
whether students are meeting these outcomes?
Step# 2:
Measures
From Here, Go Anywhere
Outcome-Assessment
Comparison Matrix
ACC 201
Outcome 1:
Accounting
Cycle
Outcome 2:
Closing
Entries
Outcome 3:
Financial
Statements
Homework
(HW)1
HW2
Quiz1
X
X
X
X
X
X
X
HW3
X
Midterm
X
Step #2:
Measures
From Here, Go Anywhere
Outcome-Test Item Matrix
ACC 201
Homework 2
Outcome 1
(75% of
Questions)
Outcome 2
(25% of
Questions)
Outcome 3
(Bonus
Question)
Question Question Question Question Question
1
2
3
5
4
X
X
X
X
X
Step #2: Measures
From Here, Go Anywhere
Assess at Specific Points in Time or
Over Time?
Think about the length and scope of the
assessment techniques you are currently
administer or you’d like to implement.
“Education is what
survives when
what has been
learned has been
forgotten.”
- B. F. Skinner
From Here, Go Anywhere
Step #3:
Results
Establish Minimum Competency
What constitutes exemplary, satisfactory, needs
improvement (e.g., rubrics/scoring guides with
examples of A/B/C/F-level work)?
What are your realistic
expectations (e.g., 90%
will graduate/pass)?
“The great difficulty in
education is to get
experience out of ideas.”
- George Santayana
From Here, Go Anywhere
Step #3:
Results
Collect Data &
Compute Actual Results
Determine when and how to collect the data.
For course-based assessment data  faculty of
selected courses administer assessments and
compile their results.
Lead Deans decide how to collect and enter the
course-based assessment data for analysis.
Compare expectations against actual results.
“You can observe a lot by just watching.”
“We made too many wrong mistakes.”
- Yogi Berra
From Here, Go Anywhere
Step #3:
Results
From Here, Go Anywhere
Step #3:
Results
From Here, Go Anywhere
Step #4: Highlights
& Enhancements
Closing the Loop: Interpreting and Using the Results
Highlights
Analyze the assessment measure data and cite
evidence that confirms the program/discipline/
course is accomplishing its intended outcome.
Enhancements
Cite evidence that identifies areas for
improvement and state enhancement
strategies/activities planned for the coming
year that address these areas.
Implement planned strategies the following
year (Fall 04) and report their impact (Spring
05).
From Here, Go Anywhere
Step #5: Other
Noted Data
Other Noted Data
Note all significant trends,
developments, accomplishments,
issues, opportunities, threats, etc..
that were NOT cited elsewhere in the
report.
Make this the “brag book” portion of
the report (e.g., faculty accolades,
regional/state/national awards).
From Here, Go Anywhere
Programmatic
Outcomes
Programmatic Outcomes
Top portion of the report form
(between the Background/Descriptive
Section and the Student Learning
Outcomes Section).
Apply same process to the nonstudent portion of the report.
Programmatic outcomes ensure your
program’s viability (e.g., FTEs,
headcounts, modernized equipment).
Closure Activity
From Here, Go Anywhere
Writing Intended Outcomes and
Selecting Appropriate Assessment
Measures (Time Permitting)
Write an outcome
Then pick a…
course
Measure, rubric
and/or test item
time to collect
way to collect
Additional
Information
Need Assistance? Clarification?
From Here, Go Anywhere
Visit the Student Outcomes Assessment
(SOA) Website
http://www.tcc.edu/welcome/collegeadmin/OIE/SOA/
index.htm
Call or Email
Mike Bryan
(757) 822-1073
[email protected]
Questions?
From Here, Go Anywhere