Transcript Closing the Loop Electronically

From Here, Go Anywhere
Assessment:
Method for Understanding Student Learning
“Effective assessment must begin
with real concerns of the
stakeholders and result in useful
information and recommendations
related to the purpose of
assessment.”
– D.W. Farmer & E.A. Napieralski
Michael E. Bryan
Office of Student Outcomes Assessment
April 13, 2004
Objectives
From Here, Go Anywhere
Know what is assessment “Many faculty have been ‘doing
Examine the driving forces assessment’ for their own sake,
and have not been overwhelmed
behind assessment (Why) with the task, without particular
Understand the benefits &
experience in evaluation
methodology.” – O.J. Nichols (1995)
tools of assessment
Define course goals and
objectives
Determine when & why to use
assessments
Identify strategies & methods
to collect data
Put it all together
From Here, Go Anywhere
What is Assessment?
The systematic collection and analysis of
information to improve student learning.
“…a learner-centered, teacher-directed approach designed to
improve student learning in the individual classroom.”
– B.D. Wright (1991)
From Here, Go Anywhere
Assessment and Grades
Grades are global evaluations that represent
the overall proficiency of students. They
don’t tell you about student performance on
individual learning goals.
“…assessment involves taking a
second look at materials generated
in the classroom so that in addition
to providing a basis for grading
students, these materials allow
faculty to evaluate their teaching.”
– B.D. Wright (1991)
Why Assess?
From Here, Go Anywhere
“…assessment involves taking a second look at materials generated in the
classroom so that in addition to providing a basis for grading students, these
materials allow faculty to evaluate their teaching.” – B.D. Wright (1991)
Rationale and Driving Forces
“The institution demonstrates that each educational
program for which academic credit is awarded is (a)
approved by the faculty and the administration, and
(b) establishes and evaluates program and learning
outcomes.”
“The institution places primary responsibility for
the content, quality, and effectiveness of its
curriculum with its faculty.”
Principles of Accreditation, Standards for All Educational Programs,
Section III, 1 and 12, August 2003
From Here, Go Anywhere
Reporting
Mandates
“Reports of Institutional Effectiveness (ROIE) is an
annual report required by the State Council of Higher
Education for Virginia (SCHEV) and is intended to
provide meaningful information on the academic
quality and operational efficiency of Virginia’s public
institutions of higher education.”
“Each institutional report is organized into five
sections: institutional mission, college profile
measures, system-wide measures, and institutionspecific measures, and the core competencies.”
ROIE Homepage, http://research.schev.edu/roie/
“One of the most untruthful things possible, you know, is a
collection of facts, because they can be made to appear so many
different ways.” - Karl A. Menninger
From Here, Go Anywhere
Reporting
Mandates
Assessment processes at VCCS colleges have been
mandated for the past eighteen years. Senate Document No.
14, 1986 Virginia General Assembly, provided the legislative
mandate for assessment reporting and SCHEV established
the initial guidelines for colleges and universities to respond.
The Final Report of the Governor’s Blue Ribbon
Commission on Higher Education (February 2000) called for
the creation of a Quality Assurance Plan to:
(1) define core competencies for written communications, oral
communications, quantitative reasoning, scientific reasoning,
critical thinking, and information literacy;
(2) identify measures to assess students’ knowledge and skills;
(3) provide a vehicle to present results publicly.
Report of the VCCS Task Force on Assessing Core Competencies,
July 18, 2002
From Here, Go Anywhere
Local Reporting
Mandates
“Annual reports will be produced to account for what
has been accomplished in enhancing current
programs and implementing new ones, and to identify
emerging developments that call for updates to the
curriculum planning process.”
TCC Comprehensive Five-Year Curriculum Plan (2003-2008), Page 1,
August 2003
“While you are
experimenting, do not remain
content with the surface of
things. Don't become a mere
recorder of facts, but try to
penetrate the mystery of their
origin.” - Ivan Pavlov
From Here, Go Anywhere
Local Reporting
Mandates
Curriculum planning at TCC will occur in accord with the following principles:
The curriculum will be structured and articulated in ways that provide students with
multiple opportunities for connection and advancement as they enter, progress
through, exit, and potentially return to the college.
The curriculum will be responsive to the distinctive needs of business, industry,
government, agencies, and the larger community, continually reinventing itself in
relation to the immediate requests and long-term developments in various
occupational and technical fields.
The curriculum will be developed in consultation with employers and transfer
institutions that are the direct recipients of graduates.
All curricula will be built and maintained on a foundation of excellence. “In the
college transfer area, the college will benchmark itself to the most rigorous
standards of achievement in 4-year baccalaureate degree-granting institutions. In
the fields of occupational-technical study, the college will look to business and
benchmark its performance goals to the standards of the relevant industries
themselves” (Bearings on the Future: The TCC Strategic Plan).
TCC Comprehensive Five-Year Curriculum Plan (2003-2008), Pages 1 & 2,
August 2003
From Here, Go Anywhere
Local Reporting
Mandates
TCC will pursue the following corresponding strategies:
The college will regularly review programming at peer community colleges across
the nation with the goal of identifying current and emerging curricular areas for
possible development and implementation. As resources permit, faculty and
administrative teams will… bring back model programs and practices for adaptation
and implementation at TCC.
Through focus groups, advisory committees, and other means of formal and
informal relationships with employers and the higher education community, the
college will actively solicit input for needed program enhancements and development
as well as communicate the range and availability of existing programs.
The college will actively partner with those who are the primary stakeholders in the
successful outcome of students, looking for ways to engage their support and
sponsorship of students throughout the learning process.
The college will put into place a systematic process for assessing programmatic
and student learning outcomes, the results from which will be used for continuous
improvement of disciplines and programs. Other college activities, processes, and
initiatives (e.g., faculty hiring, adjunct faculty initiative, accreditation, professional
development and renewal, budget development, grant proposals, major gifts
campaign) will be aligned with the pursuit of curricular excellence.
From Here, Go Anywhere
Recap Why We
Assess
External Needs
Requirements for SACS Accreditation
Report of Institutional Effectiveness for
SCHEV
Quality Assurance Plan for VCCS
Internal Needs
Curriculum Planning Process &
Annual Reports for TCC
Intellectual Curiosity
Professionalism
From Here, Go Anywhere
How is TCC doing as a learning
institution?
What do you really want students to
know and be able to do by the end
of your course?
What are you trying to measure?
What do you need to measure?
Do you know why your data results
are what they currently are?
What can you do to help students
learn what you believe they need to
know?
Is your plan for improvement based
on actual instructional strategies?
Essential
Questions
Process
From Here, Go Anywhere
Steps to the Process
1.
2.
3.
Identify & articulate what students should learn in
your class (Goals and Objectives)
Review & implement tools to compile, measure and
analyze student learning data
Use the information gathered to improve/adapt
curricula, pedagogy, and goals
“One important distinction in assessment methods is between
techniques that directly determine whether students have mastered
the content of their academic programs and those that ask students
to reflect on their learning.” – C.A. Palomba & W. Banta (1999)
Step 1
From Here, Go Anywhere
Establish Goals and Objectives
Course Goals
Describe broad learning outcomes and
concepts
What do you want students to learn/do/apply/take with
them?
Expressed in general terms
e.g., clear communication, problem-solving skills
“Much education today is monumentally ineffective. All too often we are
giving young people cut flowers when we should be teaching them to
grow their own plants.” - John W. Gardner
Goals
From Here, Go Anywhere
Difficulty Writing Goals?
Why do you use current assignments, course structure, and
activities?
What is it you want to help students learn through these
course elements?
What do you want your students to learn and in what ways do
you want them to grow?
In the past, have your goals for students been realistic?
What do your students usually learn and in what ways do they
usually grow?
Where do students have difficulty; what do they consistently
not get?
If you ran into a student who had taken your class the
previous semester, what would you hope the student would
say about what s/he took away from your course?
Objectives
From Here, Go Anywhere
Course Objectives
Objectives describe specific learning behaviors that
students should exhibit in the context of the course.
Objectives are the specific skills, values and attitudes
students should exhibit that reflect the broader goals
(e.g., for students in a freshman writing course, this
might be “students are able to develop a cogent
argument to support a position”).
Often in the assessment literature, “objectives”and
“outcomes” are used interchangeably.
“A student drove himself so hard that he missed the learning curve.”
- Pun of the Day
Objectives
From Here, Go Anywhere
Effectives Course Objectives
Use action words that specify definite,
observable behaviors
Indicate an appropriate level of attainment
Are assessable through one or more
indicators
Comprehensively and meaningfully define
a goal
“Two mathematicians
arguing about even
Are realistic and achievable
numbers were at
odds.”
Use simple language
– Pun of the Day
From Here, Go Anywhere
Levels of
Objectives
Word Power
From Here, Go Anywhere
Objectives
From Here, Go Anywhere
Difficulty Writing Objectives?
For each of your stated goals, what are the
specific student behaviors, skills, or abilities that
would tell you this objective is being achieved?
Ideally and briefly, what would a skeptic need
(what evidence needs to be present, what
specific behavior needs to be visible) in order to
see that your students are achieving the major
objectives you have set out for them?
In your experience, what evidence tells you when
students have met these objectives – how do
you know when they’re “getting” it?
Syllabus
From Here, Go Anywhere
Review the Course Syllabus
Step 2
From Here, Go Anywhere
Measure Attainment of
Goals and Objectives
What assessments do you currently use?
Are the assessments clearly tied to your goals
and objectives?
Do you need to revise/improve existing
assessments to determine student proficiency or
goal attainment?
Do you need to identify/develop new
assessments?
“The great difficulty in education is to get experience out of ideas.”
- George Santayana
Current Use
From Here, Go Anywhere
Identify & Review Existing
Assessments/Measures
What information on student learning/performance do
you currently collect?
How do these data sources relate to your newly
articulated goals and objectives?
Are there gaps between the information you collect and
your course objectives?
What other information do you need to have to
understand whether students are meeting these
objectives?
From Here, Go Anywhere
Need for
Improvement?
Additions, Deletions, and
Modifications
Do you need to revise/improve existing
assessments to determine student
proficiency or goal attainment?
Do you need to identify/develop new
assessments?
“You can observe a lot by just watching.”
“We made too many wrong mistakes.”
- Yogi Berra
Timing
From Here, Go Anywhere
Assess at Specific Points in Time or
Over Time?
Think about the length and scope of the
assessment techniques you are currently
administer or you’d like to implement.
Key Questions
From Here, Go Anywhere
From Here, Go Anywhere
Specific Points in
Time
Key Points in the Semester:
at the start of the semester
after a particularly in-depth lecture
at times when you think ideas are not
flowing as freely as they might
to clear up potential areas of confusion
to encourage reflective thinking on
particular issues or topics
“Too often we give children answers to remember rather than
problems to solve.” - Roger Lewin
From Here, Go Anywhere
Specific Points in
Time
Types of Assessments:
Primary Trait Analysis
Minute Papers
Misconception/Preconception Check
Muddiest Point
Punctuated Lectures
Chain Notes
Classroom Opinion
Polls
Reading Reaction
Paper Reaction
Over the Course
From Here, Go Anywhere
Assessing Student Learning Over
the Course of the Semester:
Tracking student achievement over time is one
of the best ways that you have to document
that students are really accomplishing what
you intend.
Incorporating classroom assessment into your
teaching and curriculum design facilitates
specific documentation of results that clearly
demonstrate student learning from the
beginning of the semester until the end of the
course. “A math professor in an unheated room is cold and
calculating.” – Pun of the Day
Over the Course
From Here, Go Anywhere
Types of Assessments Over the
Course of the Semester:
Primary Trait Analysis
Systematic progression of assignments
Pre/Post Test
Portfolio Analysis
“Education is what survives
when what has been learned
has been forgotten.”
- B. F. Skinner
Step 3
From Here, Go Anywhere
Closing the Loop: Interpreting and Using the Results
of Classroom Assessment
Where Do We Begin?
Knowing WHY you are assessing is KEY to
deciding WHAT to do with the data after you
collect it.
Gather Data for a Variety of Reasons:
Classroom-based improvement in teaching
and learning
Institution-wide program/discipline review
requirements
External accreditation initiatives
Analysis
From Here, Go Anywhere
Organize the Information
Action plan
From Here, Go Anywhere
Document Your Effectiveness
Reporting
From Here, Go Anywhere
Assessment Matrix
So Much More
From Here, Go Anywhere
Tip of the Assessment Iceberg
Countless types of assessments such as
performance, portfolio, instructionally-oriented,
computer-aided, standardized
Item analysis - modern test theory (IRT)
Briefly touched on selecting when and how
often to assess
Assessing student background and experience
Engaging students throughout the process
Preparing students for taking assessments
Considerations for large lecture classes
Tracking individual/cohort students
Closure Activity
From Here, Go Anywhere
Fun Activity to Illustrate
Importance of Scoring Guides
(Time Permitting)
Additional
Information
Need Assistance? Clarification?
From Here, Go Anywhere
Visit the Student Outcomes Assessment
(SOA) Website
http://www.tcc.edu/welcome/collegeadmin/OIE/SOA/
index.htm
Call or Email
Mike Bryan
(757) 822-1073
[email protected]
Questions?
From Here, Go Anywhere