Tools For Outcomes Assessment
Download
Report
Transcript Tools For Outcomes Assessment
Developing an Outcomes
Assessment Program:
The CCBC Approach
Dr. Irving Pressley McPhail, Chancellor
The Community College of Baltimore County
Delgado Community College – February 18, 2005
LearningFIRST 2.0
The Strategic Plan For
CCBC
FY 2004 to FY 2008
Learning
First 2.0
Learning
Support
Learning
College
Effective
Communication
Infusing
Technology
Student
Learning
Enrollment
Management
Community
& Institutional
Advancement
Organizational
Excellence
Valuing
Diversity
CCBC: A Vanguard Learning
College
One of twelve community colleges from across
the U.S. and Canada chosen to participate in the
League’s Vanguard Learning College Project
Five areas of concentration:
Organizational Culture
Staff Recruitment and Development
Learning Outcomes Assessment
Technology
Underprepared Students
Middle States Standard 14:
Assessment of Student Learning
Assessment of student learning
demonstrates that the institution’s
students have knowledge, skills, and
competencies consistent with
institutional goals and that students at
graduation have achieved appropriate
higher education goals.
Middle States Standard 14:
Assessment of Student Learning
In order to carry out meaningful
assessment activities, institutions must
articulate statements of expected
student learning at the institutional,
program, and individual course levels,
although the level of specificity will be
greater at the course level. Course
syllabi or guidelines should include
expected learning outcomes.
SACS – Principles of Accreditation
3.3 Institutional Effectiveness
3.3.1 The institution identifies expected
outcomes for its educational programs and its
administrative and educational support
services; assesses whether it achieves these
outcomes; and provides evidence of
improvement based on analysis of those
results.
Council on Innovation and
Student Learning
(CISL)
How it all began!!
Chancellor’s Charge to CISL
Serve as a college-wide think tank
Lead the transformation of the CCBC into
a premier, learning centered college
Serve as change agents
Help to frame policies, procedures and
infrastructure needed to become a learning
college
Getting Started - Fall 1998
Council on Innovation and Student
Learning
Membership included trustees, chancellor, faculty,
professional staff and classified staff
Assessment subcommittee created the Learning
Outcomes Assessment Plan for the College
(35-40 members)
(8 members)
Guide to Learning Outcomes Assessment and
Classroom Assessment (updated Spring 2003)
www.ccbcmd.edu/loa/index.html
Measuring Student Learning
Measuring Student Learning
Non-measures
Student satisfaction
surveys
Program evaluation
MHEC reporting
Student grades
Retention rates
Graduation rates
Transfer rates
Indirect measures
Exit interviews of
graduates
Employer surveys
Transfer studies
Measuring Student Learning
Direct measures include:
Standardized tests
Portfolio assessment
Capstone experience
Locally developed tests
Externally reviewed exhibitions and
performances
CCBC’s Learning Outcomes
Assessment Program
CCBC’s Assessment Plan
Principles
Primary reason for assessment is to improve
and expand student learning
Development of an effective program is a longterm, dynamic process
Must involve a multi-method approach
Must include training and support for faculty
and staff
Results are not used punitively for students or
faculty
CCBC’s Assessment Plan
Principles
Seek to use the most reliable, valid methods
and instruments available
Never an end in itself, only a means to an
end – the improvement of student learning
Continuous Quality Improvement
Design
Implement
Assess
CCBC’s Outcomes Assessment
Philosophy
Assessment is:
A natural and on-going part of instruction
Consistent with traditional instructional
practices
Designed to meet specific objectives
Conducted in a risk-free environment
LOA Project Collaboration
Learning Outcomes Associate
Planning, Research and Evaluation Liaison
Learning Outcomes Assessment Advisory Board
Deans’ Council
General Education Review Board
Developmental Education Advisory Committee
LOA Project Collaboration
Learning Outcomes Associate
Development of design
Survey/tool development
Staff development
Logistical plan
Institutional Research
Data entry
Data analysis
Data interpretation
Types of Designs
Portfolio assessment
Standardized tests
External graders/experts
Pre- and post-tests
Cooperation with other schools
Creativity Abounds!
Project Elements
Stage 1:
Design Course Project
Determine Measurable Objectives
Select Assessment Instrument
Include External Validation
Control Important Variables
Develop Full Proposal
Project Elements
Stage 2:
Implement Design
Administer Assessment Instrument(s)
Collect Data
Analyze Data
Project Elements
Stage 3:
Test Objectives
Met? > Write Final Report
Not Met? > Design Course Improvements
Write Interim Report
Project Elements
Stage 4:
Implement Course Improvements
Collect Data
Analyze Data
Stage 5:
Final Report
Discipline/Campus
College
In Summary
Establish Climate of Continuous
Improvement
Empower Faculty
Provide Substantial Administrative and
Fiscal Support
Share Results - Celebrate Successes
Course Level Assessment
High Impact LOAs
FY ‘03
CINS 101
HLTH 101
PEFT 101
SDEV 101
ENGL 101
FY ‘04
FY ‘05
MGMT 101
SOCL 101
PHIL 101
BIOL 110
SPCM 101
ARTS 104
ENVS 101
PSYC 105
MATH 082
HLTH 101:
Health and Wellness
High impact project
Pre-test/post-test design
100 item, four option multiple choice format exam
Topics included subjects commonly covered in
introductory health textbooks
Draft instrument was reviewed by a health/curriculum
expert at a four-year university
Two and four-year colleges and universities have been
invited to share our instrument so that we might
develop external comparisons
Health 101 LOA
Results
Matched pair t-test analysis
Students’ scores improved significantly from preto post-test
At least 75% of students scored a 75% or higher
on the post-test
Scores varied by campus; small difference in
scores based on race
Reassessment will be conducted after
recommendations have been implemented
Health 101 LOA
Recommendations
Three topic areas were identified as needing
improvement: nutrition, heart disease, and
human sexuality
The campus with students who consistently
scored higher on the pre-test will offer more
honors’ sections
A faculty guide, “Strategies for Teaching Health
Education” has been developed and shared with
full-and part-time faculty
RDNG 052: College Reading
Instruments
Used the Nelson-Denny Reading Test (Forms G
and H) pre and post for the 2001 and the 2003
assessments
Used the Learning Attitudes Study Strategies
Inventory (LASSI) for the two semesters of pilot
projects and the 2001 assessment
Design
Pre-post project design
Matched pair data analysis
RDNG 052: College Reading Fall 2001 Results
Mean pre-post difference of 3.5 (raw
points) was statistically significant at the
.001 level
Post-test score was greater than pretest score in 74% of cases
Resulting grade level increase of .7
grade level (in a 15 week semester)
RDNG 052: College Reading Fall 2001 Results
Exiting students read at 9.6 mean grade
level
Differences among campuses—learned
from best practices on each campus
LASSI results showed significant
improvement in 10 of 12 scales from pre
to post
RDNG 052: College Reading –
Fall 2001 Interventions
Interventions included:
Using more Internet-based learning
opportunities
Enhancing instructional activities that focused
on literal and inferential comprehension
Greater focus on working with adjunct faculty
RDNG 052: College Reading –
Fall 2003 Results
Mean pre-post difference of 3.7 was statistically
significant at the .001 level
Post-test score was greater than pre-test score
in 75% of cases (larger sample)
Resulting grade level increase of 2.0! (in a 15
week semester)
Exiting students read at 10.1 mean grade level
Differences among campuses; every campus
improved
RDNG 052 - 2001 to 2003:
Improvements in Student Learning
Significant improvements in
students’ reading levels from 2001
to 2003 indicate increased learning
in RDNG 052.
RDNG 052 College Reading Final Recommendations
The Cycle of Continuous Improvement
Research Accuplacer cut scores between
RDNG 051 and RDNG 052
Continue to close the performance gap
Continue to share best practices
Continue to study post-Reading success rates
(success in subsequent courses)
Making the Transition from Course
to Program Level Assessment
Program Review
General Education
Developmental Education
General Education
The Academic Profile
And
The “GREATs”
The Academic Profile
Norm-referenced, externally developed
test of general knowledge
Widely used
Sample size: 1,017 Students
Purpose - to establish baseline data upon
implementation of new General Education
Program in Fall 2001
Academic Profile - Findings
The Academic Profile provides CCBC with a
baseline measure of how our students are
acquiring academic skills developed through
General Education courses compared to
national norms.
CCBC students scored at or close to the national
norms on the total mean score and all sub-scores the
first time the test was administered.
The Academic Profile will be administered again in
Fall 2004 (sample size 2,000+).
Academic Profile –
Data Implications
Work on Critical Thinking skills
Reinforce skills learned in one class in
other classes
Provide Culturally Mediated Instruction
Possible statewide initiatives
GREAT Projects
General Education Assessment Teams (GREAT) Projects for all
Maryland Higher Education Commission general education
categories
Common Graded Assignments (CGA) and accompanying Scoring
Rubrics
Created by faculty teams
External Consultant
Faculty Training
Three semesters of Pilot Projects
Full implementation began in Fall 2003
Every course assessed once every three years
GREAT Data (Fall, 2003)
Implications
Increased awareness of faculty regarding how
General Education courses are defined by the six
criteria
Midrange to higher scores in Content may
indicate a traditional approach in how “Content” is
conveyed with little expectation for higher level
use of content in application, critical thinking,
analysis, and synthesis activities
Workshops on how to use the GREAT data to
improve student learning
LOA Successes
Creating a “culture of assessment” with
increased faculty participation and buy-in
Using outcomes assessment for Program-level
assessment
Forging new partnerships between faculty
teams, institutional research staff, the Vice
Chancellor for Learning and Student
Development’s office, and the Outcomes
Associate in analyzing data and making
curricular and pedagogical recommendations for
change
LOA Successes
Creating a newly updated Guide for Learning
Outcomes Assessment and Classroom Learning
Assessment, a model guide for assessment
www.ccbcmd.edu/loa/index.html
Establishing the Learning Outcomes
Assessment Advisory Board
Recognition by the League for Innovation in the
Community College and Middle States
Commission on Higher Education as a national
leader in Learning Outcomes Assessment
For Further Information:
Dr. Irving Pressley McPhail
Chancellor
The Community College of Baltimore County
800 South Rolling Road
Baltimore, Maryland 21228-5317
Telephone: 410-869-1220
Fax: 410-869-1224
Email: [email protected]
Website: www.ccbcmd.edu
For Further Information:
Dr. Alvin Starr, Acting Vice Chancellor for
Learning and Student Development
Email: [email protected]
Dr. Rose Mince, Assistant to the VCLSD
Email: [email protected]
Professor Tara Ebersole, Outcomes Associate
Email: [email protected]
Questions /Comments