Transcript Tools For Outcomes Assessment

Developing an Outcomes
Assessment Program:
The CCBC Approach
Dr. Irving Pressley McPhail, Chancellor
The Community College of Baltimore County
Delgado Community College – February 18, 2005
LearningFIRST 2.0
The Strategic Plan For
CCBC
FY 2004 to FY 2008
Learning
First 2.0
Learning
Support
Learning
College
Effective
Communication
Infusing
Technology
Student
Learning
Enrollment
Management
Community
& Institutional
Advancement
Organizational
Excellence
Valuing
Diversity
CCBC: A Vanguard Learning
College


One of twelve community colleges from across
the U.S. and Canada chosen to participate in the
League’s Vanguard Learning College Project
Five areas of concentration:





Organizational Culture
Staff Recruitment and Development
Learning Outcomes Assessment
Technology
Underprepared Students
Middle States Standard 14:
Assessment of Student Learning
Assessment of student learning
demonstrates that the institution’s
students have knowledge, skills, and
competencies consistent with
institutional goals and that students at
graduation have achieved appropriate
higher education goals.
Middle States Standard 14:
Assessment of Student Learning
In order to carry out meaningful
assessment activities, institutions must
articulate statements of expected
student learning at the institutional,
program, and individual course levels,
although the level of specificity will be
greater at the course level. Course
syllabi or guidelines should include
expected learning outcomes.
SACS – Principles of Accreditation

3.3 Institutional Effectiveness

3.3.1 The institution identifies expected
outcomes for its educational programs and its
administrative and educational support
services; assesses whether it achieves these
outcomes; and provides evidence of
improvement based on analysis of those
results.
Council on Innovation and
Student Learning
(CISL)
How it all began!!
Chancellor’s Charge to CISL

Serve as a college-wide think tank

Lead the transformation of the CCBC into
a premier, learning centered college

Serve as change agents

Help to frame policies, procedures and
infrastructure needed to become a learning
college
Getting Started - Fall 1998

Council on Innovation and Student
Learning

Membership included trustees, chancellor, faculty,
professional staff and classified staff


Assessment subcommittee created the Learning
Outcomes Assessment Plan for the College


(35-40 members)
(8 members)
Guide to Learning Outcomes Assessment and
Classroom Assessment (updated Spring 2003)

www.ccbcmd.edu/loa/index.html
Measuring Student Learning
Measuring Student Learning

Non-measures







Student satisfaction
surveys
Program evaluation
MHEC reporting
Student grades
Retention rates
Graduation rates
Transfer rates

Indirect measures



Exit interviews of
graduates
Employer surveys
Transfer studies
Measuring Student Learning

Direct measures include:
Standardized tests
 Portfolio assessment
 Capstone experience
 Locally developed tests
 Externally reviewed exhibitions and
performances

CCBC’s Learning Outcomes
Assessment Program
CCBC’s Assessment Plan
Principles





Primary reason for assessment is to improve
and expand student learning
Development of an effective program is a longterm, dynamic process
Must involve a multi-method approach
Must include training and support for faculty
and staff
Results are not used punitively for students or
faculty
CCBC’s Assessment Plan
Principles



Seek to use the most reliable, valid methods
and instruments available
Never an end in itself, only a means to an
end – the improvement of student learning
Continuous Quality Improvement
Design
Implement
Assess
CCBC’s Outcomes Assessment
Philosophy
Assessment is:
 A natural and on-going part of instruction
 Consistent with traditional instructional
practices
 Designed to meet specific objectives
 Conducted in a risk-free environment
LOA Project Collaboration






Learning Outcomes Associate
Planning, Research and Evaluation Liaison
Learning Outcomes Assessment Advisory Board
Deans’ Council
General Education Review Board
Developmental Education Advisory Committee
LOA Project Collaboration


Learning Outcomes Associate

Development of design

Survey/tool development

Staff development

Logistical plan
Institutional Research



Data entry
Data analysis
Data interpretation
Types of Designs






Portfolio assessment
Standardized tests
External graders/experts
Pre- and post-tests
Cooperation with other schools
Creativity Abounds!
Project Elements

Stage 1:

Design Course Project
Determine Measurable Objectives
 Select Assessment Instrument
 Include External Validation
 Control Important Variables


Develop Full Proposal
Project Elements

Stage 2:




Implement Design
Administer Assessment Instrument(s)
Collect Data
Analyze Data
Project Elements

Stage 3:
 Test Objectives
Met? > Write Final Report
Not Met? > Design Course Improvements
Write Interim Report
Project Elements

Stage 4:




Implement Course Improvements
Collect Data
Analyze Data
Stage 5:

Final Report


Discipline/Campus
College
In Summary




Establish Climate of Continuous
Improvement
Empower Faculty
Provide Substantial Administrative and
Fiscal Support
Share Results - Celebrate Successes
Course Level Assessment
High Impact LOAs
FY ‘03





CINS 101
HLTH 101
PEFT 101
SDEV 101
ENGL 101





FY ‘04
FY ‘05
MGMT 101
SOCL 101
PHIL 101
BIOL 110
SPCM 101
 ARTS 104
 ENVS 101
 PSYC 105
 MATH 082
HLTH 101:
Health and Wellness






High impact project
Pre-test/post-test design
100 item, four option multiple choice format exam
Topics included subjects commonly covered in
introductory health textbooks
Draft instrument was reviewed by a health/curriculum
expert at a four-year university
Two and four-year colleges and universities have been
invited to share our instrument so that we might
develop external comparisons
Health 101 LOA
Results





Matched pair t-test analysis
Students’ scores improved significantly from preto post-test
At least 75% of students scored a 75% or higher
on the post-test
Scores varied by campus; small difference in
scores based on race
Reassessment will be conducted after
recommendations have been implemented
Health 101 LOA
Recommendations



Three topic areas were identified as needing
improvement: nutrition, heart disease, and
human sexuality
The campus with students who consistently
scored higher on the pre-test will offer more
honors’ sections
A faculty guide, “Strategies for Teaching Health
Education” has been developed and shared with
full-and part-time faculty
RDNG 052: College Reading

Instruments



Used the Nelson-Denny Reading Test (Forms G
and H) pre and post for the 2001 and the 2003
assessments
Used the Learning Attitudes Study Strategies
Inventory (LASSI) for the two semesters of pilot
projects and the 2001 assessment
Design


Pre-post project design
Matched pair data analysis
RDNG 052: College Reading Fall 2001 Results



Mean pre-post difference of 3.5 (raw
points) was statistically significant at the
.001 level
Post-test score was greater than pretest score in 74% of cases
Resulting grade level increase of .7
grade level (in a 15 week semester)
RDNG 052: College Reading Fall 2001 Results



Exiting students read at 9.6 mean grade
level
Differences among campuses—learned
from best practices on each campus
LASSI results showed significant
improvement in 10 of 12 scales from pre
to post
RDNG 052: College Reading –
Fall 2001 Interventions

Interventions included:



Using more Internet-based learning
opportunities
Enhancing instructional activities that focused
on literal and inferential comprehension
Greater focus on working with adjunct faculty
RDNG 052: College Reading –
Fall 2003 Results





Mean pre-post difference of 3.7 was statistically
significant at the .001 level
Post-test score was greater than pre-test score
in 75% of cases (larger sample)
Resulting grade level increase of 2.0! (in a 15
week semester)
Exiting students read at 10.1 mean grade level
Differences among campuses; every campus
improved
RDNG 052 - 2001 to 2003:
Improvements in Student Learning

Significant improvements in
students’ reading levels from 2001
to 2003 indicate increased learning
in RDNG 052.
RDNG 052 College Reading Final Recommendations

The Cycle of Continuous Improvement
 Research Accuplacer cut scores between
RDNG 051 and RDNG 052
 Continue to close the performance gap
 Continue to share best practices
 Continue to study post-Reading success rates
(success in subsequent courses)
Making the Transition from Course
to Program Level Assessment



Program Review
General Education
Developmental Education
General Education
The Academic Profile
And
The “GREATs”
The Academic Profile




Norm-referenced, externally developed
test of general knowledge
Widely used
Sample size: 1,017 Students
Purpose - to establish baseline data upon
implementation of new General Education
Program in Fall 2001
Academic Profile - Findings

The Academic Profile provides CCBC with a
baseline measure of how our students are
acquiring academic skills developed through
General Education courses compared to
national norms.


CCBC students scored at or close to the national
norms on the total mean score and all sub-scores the
first time the test was administered.
The Academic Profile will be administered again in
Fall 2004 (sample size 2,000+).
Academic Profile –
Data Implications




Work on Critical Thinking skills
Reinforce skills learned in one class in
other classes
Provide Culturally Mediated Instruction
Possible statewide initiatives
GREAT Projects







General Education Assessment Teams (GREAT) Projects for all
Maryland Higher Education Commission general education
categories
Common Graded Assignments (CGA) and accompanying Scoring
Rubrics

Created by faculty teams
External Consultant
Faculty Training
Three semesters of Pilot Projects
Full implementation began in Fall 2003
Every course assessed once every three years
GREAT Data (Fall, 2003)
Implications



Increased awareness of faculty regarding how
General Education courses are defined by the six
criteria
Midrange to higher scores in Content may
indicate a traditional approach in how “Content” is
conveyed with little expectation for higher level
use of content in application, critical thinking,
analysis, and synthesis activities
Workshops on how to use the GREAT data to
improve student learning
LOA Successes



Creating a “culture of assessment” with
increased faculty participation and buy-in
Using outcomes assessment for Program-level
assessment
Forging new partnerships between faculty
teams, institutional research staff, the Vice
Chancellor for Learning and Student
Development’s office, and the Outcomes
Associate in analyzing data and making
curricular and pedagogical recommendations for
change
LOA Successes

Creating a newly updated Guide for Learning
Outcomes Assessment and Classroom Learning
Assessment, a model guide for assessment



www.ccbcmd.edu/loa/index.html
Establishing the Learning Outcomes
Assessment Advisory Board
Recognition by the League for Innovation in the
Community College and Middle States
Commission on Higher Education as a national
leader in Learning Outcomes Assessment
For Further Information:
Dr. Irving Pressley McPhail
Chancellor
The Community College of Baltimore County
800 South Rolling Road
Baltimore, Maryland 21228-5317
Telephone: 410-869-1220
Fax: 410-869-1224
Email: [email protected]
Website: www.ccbcmd.edu
For Further Information:
Dr. Alvin Starr, Acting Vice Chancellor for
Learning and Student Development
Email: [email protected]
Dr. Rose Mince, Assistant to the VCLSD
Email: [email protected]
Professor Tara Ebersole, Outcomes Associate
Email: [email protected]
Questions /Comments