Transcript Assessment Symposium WELCOME! May 22, 2007 Study Abroad Center helping students gain global perspective Lisa Loberg Director, Study Abroad.

2007
Assessment Symposium
WELCOME!
May 22, 2007
Study Abroad Center
helping students gain global perspective
Lisa Loberg
Director, Study Abroad
Continuous Improvement Project
Process
• Purpose:
– To review coordination between campus offices as student
participation increases
– Examine systems to create new procedures
• Team Representatives:
– Business Office
– Institutional Research
– International Admissions
Registrar
Study Abroad
• Outcomes:
– Created Datatel codes for all programs and screen
– Entirely new system of budgeting and accounting for department
– Building relationships and improving communication
• Review:
– Team to review implemented changes and make new
recommendations as needed
Welcome Back Survey Purpose
Evaluation of:
1. Specific Study Abroad Program
•
•
•
•
•
Academics
Location
Housing
Activities
Overall experience
2.
Study Abroad Center
3.
Impact of student experience
• Effectiveness
• Suggestions for improvement
• Evidence of student ability to
•
•
•
Demonstrate Cultural Competency
Growth in Identity and Values
Appreciate Diversity
2006-07 Survey Design Goals
•
Review previous data and questions
•
Incorporate questions that measure
Student Learning Outcomes
•
Convert paper-based survey to
electronic Flashlight survey
Design Chronology
• Prior to 2004: No Study Abroad Experiences Survey.
• Spring 2005 1st version:
– Paper survey with all open-ended questions.
– Useful but difficult to tabulate results.
• Spring 2006 2nd version:
– Paper survey with both open-ended questions and rank-order scale.
– Rank-order helpful; needs development.
• Fall 2006 3rd version:
– Paper survey with open-ended questions and rank-order scale with scale
levels defined.
– Provided more accurate information; needs collection improvement.
• Spring 2007 Current version:
– Electronic Flashlight survey with open-ended questions and rankorder scale. Incorporated specific questions to measure student
learning outcome goals.
– Results tabulated immediately. Facilitates reporting.
Analysis of Data
Study Abroad Emerging Themes
• Experiences are life-changing.
• Cultivates global perspective and respect for people of diverse
cultures.
• Increases students’ ability to Demonstrate Cultural
Competency, Grow in Identity and Values, and Appreciate
Diversity.
Lessons Learned
• Creating surveys in Flashlight is simple and is effective.
• Flashlight results are tabulated immediately and easily
reported.
• Attention to survey design enriches quality of responses and
ability to use data.
• Survey design can capture the achievement of Student
Learning Outcomes.
2007-08 Assessment Goal
• Every returning student completes a
survey
• Incorporate additional questions
relating to Student Learning Outcomes
• Add a demographics section to make
national reporting of statistics even
easier
Criminal Justice-Program Review
Robert J. Meadows
Helen Ahn Lim
COAS - Department of Criminal Justice
CJ Program Features
• Two full-time faculty
• Approximately 100 majors
 Robert J. Meadows, Ph.D.,
Ed.D. ; Professor and Program • Offers B.S. in Criminal Justice
(42 units)
Chair
• Offers Minor in Legal Studies
 Helen Ahn Lim, Ph.D.;
(20 units)
Assistant Professor
• Comprehensive with
• Three to four adjunct instructors interdisciplinary focus and cross
listing of courses (sociology,
• Approximately 100 majors
psychology, political science)
• About 10-12 classes each
semester with an average class • One of fastest growing UG
majors
size of 20 students
• Internship requirement
• Alpha Phi Sigma honor society
2006-07 Program Review Goal
To assess how the Program prepares students for
employment or post graduate study, to identify
graduate’s perceptions regarding the value of their
degree and courses, and to learn of their
education/employment progress through alumni surveys
Data Collection
Beginning September 2006, examined:
• Curriculum information and
course syllabi
• Program goals
• CJ and CLU Student
Learning Outcomes
• Trends (Enrollment and FacultyStudent Ratios)
• Library resources
• Alumni survey data
• Review by external reviewer
Alumni Survey Data Collection and Analysis
• Collected data from 2000-2006 CJ graduates
• 98 graduates identified (46 usable questionnaires)
• Questionnaire responses
1. Employment of alums (33/46 or 71% in jobs
related to CJ) including law enforcement,
corrections, law, security and Government jobs
related to justice.
2. Alums attending or completing post graduate
studies (14/46 or 38%) including degrees in
Business (MBA), Psychology, Education, and
Criminal Justice.
Alumni Survey Data Collection and Analysis
• Questionnaire responses
3. Degree preparation for career (36/46 or 78%
believed the degree somewhat prepared or
prepared them very much for their present
vocation).
4. Degree preparation for graduate or professional
studies. (All respondents attending post grad
studies believed the degree prepared them for
post graduate study).
5. Overall reflection of learning experience (39/46
or 85% indicated that their educational
experiences and courses taken in criminal
justice were either excellent or good).
Analysis of Data
What did the data say?
• The degree provides a relevant and valuable education
• Overall, curriculum is consistent with Program goals, CJ and CLU
Learning Outcomes
• Higher scores than most other programs for Challenge and
Engagement in Arts and Sciences
• Most diverse student profile in Arts and Sciences
What themes emerged?
• Problem with understaffing given the number of majors
What was learned?
• Alumni Survey: Most graduates are employed in relevant field;
several attend graduate or law school. Demonstrates the value of
senior seminar course and internship requirement.
• Courses: Consistent with major criminal justice programs in other
schools. However, Alumni Survey suggests that more courses
would further enrich the program.
• CLU is the only private residential school offering a four-year
degree in Ventura, Los Angeles and Santa Barbara Co.
Goals and Recommendations
Outcomes of Program Review and Measurable Goals
• Comprehensive and unique program (meets CJ goals and
aligns with CLU Learning Outcomes)
• Experienced tremendous growth and is a highly marketable
degree
• Need to hire more full-time faculty in the future (in order to
expand curriculum, and further enhance the quality of the
program)
• Provide more leadership opportunities for students (group
projects, CJ honor society)
Recommendations for growth based on analysis
• Hire more full-time faculty
• Improve facilities and offices
Connecting the Dots:
School of Business Assessment System
An Overview
Charles Maxey
Bonnie Johnson
Key Problem Points
• Continuous Improvement Model
• Institutional  School or Department
Program  Course  Assignment 
Assessment  Data Collection 
Reporting, Discussion, Analysis 
Educational Effectiveness and
Continuous Improvement

Student Learning Outcomes
California Lutheran University and CLU School of Business
(Alignment and Expansion)
University:
•
•
•
Professional Preparation
Field Specific Knowledge and Experience
Ethical and Professional Judgment
Interpersonal and Teamwork Skills
•
•
Liberal Learning
Critical Thinking
Information Literacy
Written Communication
Oral Communication
Ability to comprehend issues from disciplinary and
interdisciplinary perspectives
Understanding of cultural and global diversity
Integration of knowledge with ethical reflection
•
•
•
•
Character and Leadership Development
Growth in Identity and Values
Appreciation of Diversity
Service to the Community
Principled Leadership
•
•
•
•
•
School of Business:
•
•
•
•
•
•
•
•
•
•
•
•
Professional Preparation
Field Specific Knowledge and Experience
Ethical and Professional Judgment
Interpersonal and Teamwork Skills, Principled Leadership
Planning and Organization
Statistical Data Analysis
Liberal Learning
Creative and Critical Thinking
Information Literacy and Computer Skills
Written Communication
Oral Communication and Listening Skills
Evaluation
Understanding of cultural and global diversity
Creation of Value
Character and Leadership Development
Standard Pages
Following the Process
• Sample Syllabus: Bus367 – Behavior in
Organizations (taught by Dr. Susan
Murphy)
• Sample Assignment Description from Bus367 –
Personal Application Assignment
• Sample Rubric for Assessing Personal
Application Assignment
• Sample Report for Further Departmental
Discussion and Continuous Improvement
CLU’s Assessment System
Library Services Review
Cindy Campbell
Director of Library Operations and Systems
Information Systems & Services
2006-07 Assessment Goal
To evaluate Pearson Library services for
the purposes of improvement:
– Library Services in general
– Library collection
– Library space
– Information literacy instruction
Assessment Design
Measurements
• LibQual Survey
• MyCLU Portal Survey
Participants
• Stratified random sampling
• University population
• Confidence interval of 10%
Data Collection
LibQual
• Survey Levels and Scale
–
–
•
Minimum, Desired, & Perceived
1 to 9 with 9 being the most favorable
22 core items in 3 areas of service provided by
the ISS Library Service group:
 Affect of Service
 Library as Place
 Information Control
MyCLU Portal Survey
Analysis of Data with Resulting Goals
Superiority = Perceived – Desired
• Employees who are consistently courteous
• Giving users individual attention
Adequacy = Perceived – Minimum
Emerging Themes
Community space
Library space
Lighting
Resulting Goals
Evaluate how we are
currently using the library
space
Evaluate lighting issues
Analysis of Data with Resulting Goals
Emerging Themes
Resulting Goals
Library Collections
Increase our on-line
resources
Update our print resources
Information
Literacy
Publicize our information
literacy program at a higher
level
Website navigation Redesign our website
Department of Philosophy
Program Review and Evaluation
William Bersley, Chair and Associate Professor
Nathan Tierney, Professor
Xiang Chen, Professor
The Philosophy Department conceives of philosophy as an enterprise of both
mind and spirit. The faculty are committed to providing the student with the
knowledge and skills of critical and reflective thinking, moral judgment, and
understanding of major philosophical traditions and perennial philosophical
questions. Such knowledge and skills not only enable the student to become a
philosophically literate person, but facilitate the integration of this knowledge
with each student’s process of moral, spiritual and intellectual growth.
2006-07 Program Review Goal
To compare departmental grade scores to
mean grade distributions for Arts and
Sciences and Humanities Division.
Assessment Design
Data collected and examined in Review
• Trends (Enrollment and
Faculty-Student Ratios)
• Teaching Effectiveness
• Scholarly Productivity
• Challenge and
Engagement
• Critique of Curriculum
• Department Student
Learning Outcomes
• Demand data
• Load data
• Capacity data
• Student evaluation data
Grade distribution data
• Comparison with similar
programs
• Student Profiles
• Curriculum Map
Data
Grade Distributions (4 point scale)
Spring
‘05
Philosophy
Fall ‘04
Spring
‘04
Fall’03
Spring
‘03
Fall ‘02
3.21
3.15
2.70
2.98
3.25
3.25
Traditional 3.23
Undergrads
(School)
3.16
3.22
3.15
3.25
3.17
Traditional 3.14
Undergrads
(Division)
3.08
3.17
3.06
3.20
3.21
Analysis of Data
• Departmental grade distributions were comparable to
both Arts and Sciences (School) and the Humanities
Division, with the exception of Spring ‘04 and Fall ‘03
which were somewhat below the averages of the
School and Division.
• External review of this (and other) data suggests
creation of departmental/internal assessment
strategy to improve quality, e.g., longitudinal
evaluation of capstone writings and supplementary
survey/questionnaires for core required courses to
determine value to undergraduate education and
skills transferable to other courses.
Goals and Recommendations
• Provide faculty with current data concerning grade distribution
in order to prevent grade inflation (see previous data)
• Sustain highly evaluated “vital teaching” and articulate the
centrality of Philosophy within the university
• Increase the role of Philosophy in the Honors Program
• Implement the use of Philosophy 400 to initiate a Philosophy
Honors Program/Capstone Seminar (requires staffing and
budgetary support)
• Develop an assessment strategy that authentically improves
Philosophy on its own terms
• Increase mentoring of sustained student projects
• Implement the establishment of a Center of Global Ethics under
the leadership of Dr. Nathan Tierney
The Academic Assistance Program
(The AAP)
An essential piece of our retention strategies
Damien A. Peña, MSW
Senior Director of Academic Programs
What is the AAP?
• Academic Assistance Program (AAP)
• Retention strategy for Probation students and
Conditionally Admitted students
• Modeled after student success plans in
Student Support Services
• Evaluates each student’s academic progress
through administered assessments
• Self-funded (advising fee assessed to
students)
2006 – 2007 Program Goals
To provide Academic Assistance (strategies
and skills) to:
• Students on Probation
– To be removed from probationary status
• Conditionally Admitted Students
– To achieve academic success at CLU
• Self-selected Students
– To stay “ahead”
Design and Data Collection
Triangulated Strategy
• Exit Interview
– Gather student perceived performance as a result of being in
the program
– 1 on 1 with the Academic Counselor
– 5 questions
– 93 responses out of 99
• Exit Flashlight Survey
– Gather student perceived performance using anonymous
survey during the last session
– Walked student to computer
– 9 Open ended questions
– 17 scale questions concerning services and the CAAR Office
– 93 responses out of 99
• Retention Data
– Determine number of students removed from Probation
– Data pending until grades submitted
Analysis of Data
• Students’ perceived outcomes/goals
– Wanted to improve grades
– Wanted to improve study habits
• Positive impact on students
– Accountability
– Connection with someone on campus
• Challenges of program
– Communication with Faculty
– Capacity issues
– Library hours
2007-2008 Program Goals
Addressing Challenges
• Communication with Faculty:
• Capacity: Outside funding to address capacity issues
• Library Hours:
In Addition
• Assess AAP performance using benchmarks
identified as indicators of success
• Conduct longitudinal study using quantitative and
qualitative data (retention data & exit surveys)
• Expand services for self-selected students
• Achieve 100% return rate for student evaluations
Athletic Facility Scheduling
Dan Kuntz, Director of Athletics
Coaches, Assistant Coaches,
Athletics Administrative Assistant
2006-2007 Assessment Goal
Goal/Problem
To develop and implement an Athletic Facility Master Schedule
incorporating internal and external requests for:
Gilbert Arena
Soiland Recreation Center
Game and practice fields
Samuelson Aquatics Center
Tennis courts
To evaluate Effectiveness of Athletics Facility Scheduling Plan
Factors and limitations
•
•
•
•
•
•
•
19 sports programs / nearly 400 student athletes / 6 practice sessions per week per
program / between August 15 and May 15 / overlapping seasons.
18 sports / 4 to more than 25 home contests (No track facility for Track and Cross
Country home contests).
Single venues used by multiple sports teams limits schedule options.
Status of coaches and class schedules.
PT v FT - afternoon hours for practices.
Lack of lighted facilities for outdoor sports.
SCIAC schedules set days and times.
Qualitative and Quantitative Research Design
Methodology
• Data collected annually
• Collaborate with Athletic facility users to discover needs for facility use
• Data Collection Measures:
Electronic facility request forms, e-mails, phone calls, meetings,
observations, conference schedules, practice times, internal and
external special event requests
• Population: Potential users, internal, community
Athletics
Admissions
Conferences and Events
Residence Life
Intramurals
Open facility requests for
Academics
Student Activities
ESSM Classes
Alumni Relations
Renters
Church Relations
Club sports
Marketing and Communications
Recreation
Multi Cultural / International Programs
students, faculty, staff, alumni and CLA community members
Qualitative and Quantitative Research Design
Variables and Constraints
Weather, Academic Schedules of Students, Athletic Facility Maintenance,
Student Employment Staffing, Special Events, Facility quantity /quality /design.
Practices and/or Games by Months:
Baseball: January - May + October
M/W Basketball: October - March
Football: October – Nov + April
M/W Soccer - August - Dec + April
M/W Swim and Dive: Sept - March
M/W Tennis: January - May
Softball: January - May + October
Volleyball: August - November + April
No track athletes off campus:
M/W Cross Country: Sept - November
Golf: February – May
M/W Track and Field: January - May
Data Analysis Prioritization Measures: Requests for scheduling time filtered
1.
2.
3.
Academic
Athletic
Student Activities
4.
5.
General University Requests
Outside University Requests
Analysis of Data
Data Reveals Emerging Themes
• Requests growing from internal and external community users for access to
CLU Athletic facilities.
• New emerging ideas for Athletic facility usage.
Additional factors
• Maintenance costs are climbing.
Lessons Learned
• There is confusion about scheduling needs and processes.
• Diplomacy is necessary.
• Education and communication is needed in the scheduling processtolerance.
• Centralized scheduling is important.
• Open access to Athletics facilities can be limited by rental of facilities.
Future Challenges
• Class scheduling. Lack of lights on fields. Lack of sufficient quality and
quantity of fields. Weather variables. League requirements. Special
access.
• Demand for Athletic facility access and usage will likely grow internally and
externally.
2007-08 Assessment Goal
2006 – 2007 Outcomes
•
•
Internal and external Athletic facility schedules are coordinated.
Users are satisfied with access.
Growth Recommendations
1.
2.
3.
4.
5.
Centralize Athletic facility scheduling for efficiencies to potentially reduce
conflicts, miscommunication, and misunderstanding.
Prioritize scheduling of Athletic facilities for CLU students, faculty, staff and
alumni despite growing pressures for revenue production from facility
rentals.
Increase personnel resources.
Purchase scheduling software.
Implement On-line scheduling.
Measurable goals for 2007 - 2008
•
•
•
•
Access and flexibility for CLU program offerings sustained.
Provide more student, faculty, alumni and staff access to Athletic facilities.
Attract sports events and activities that bring prospective students and
families to CLU.
Develop innovative community outreach programs that attract visitors to see
the value of CLU programs.
Sharing Responsibility
for Multiple Learning Outcomes
New partnerships across Academic Departments and Student Affairs
Dr. Silva Karayan
Director, Center for Academic Service-Learning
Stine Odegard
Coordinator, Community Service Center
Sharing Responsibility
for Multiple Learning Outcomes
Purpose:
To enhance CLU’s educational effectiveness
in a global era, focus on the integration of
multiple learning outcomes.
Service-learning pedagogy facilitates
mission-focused integrative learning through
intentional collaboration:
•
•
•
•
Academic Affairs Office
Academic Departments
Center for Academic Service-Learning (CASL)
Community Service Center (CSC)
Key Partnerships on Campus
• Community Service and Co-curricular Service-learning (CSC)
– One coordinator, 4 student workers (10 hours/week)
– 2 week-long trips, 4 weekend experiences, 35 day experiences
– Co-curricular service-learning: action-reflection model
• Center for Academic Service-learning and SLO’s (CASL)
– One director, many faculty partners from various departments
– Ongoing faculty development opportunities & individualized mentoring
– High quality service-learning project components:
• Integrative learning
• High Service
• Student voice
• Collaboration
• Reflection
• Evaluation
– Paradigm Shift: From “Deficit Model” to “Empowerment Model”
– Orientation Shift: From “Charity and benevolence” to “Learning and making
a difference”
Assessment Goal and Design
Survey Goal
To determine student interest and discover the meaning of their experience in
the March 10, 2007 Community Service Day (extra-curricular activity)
Survey Design and Data Collection
•
•
•
•
Pre administered on-site, paper format (67 respondents)
Post survey (Flashlight) administered via email (42 respondents)
Review of BCSS/NSSE data
Qualitative and quantitative data
Demographics of 117 participants
•
•
M/F ratios: 23 males (19.65%), 94 females (80.35%)
Fields of study: majorities of the survey respondents
– (Bus.-12%, Comm.-12%, ESSM-8%, Lib. Studies-5%)
•
Previous service involvement:
65% participated in service at CLU before
85% participated in service during high school
Analysis of Data
• Q4: “Overall, my involvement in Service Day was
important to me because:”
• Q3 and Q5: Student likes and needs:
• BCSS(2004)/NSSE (2005):
Service to the Community
Engagement
Structured events, social events, meaningful tasks
High
1%
8%
Low
50%
41%
Low
High
Expectation
2007-08 Assessment Goals
• Measurable goals for 2007 – 2008
– Offer more integrated service opportunities (meet student
expectations)
– Increase number of faculty incorporating service-learning
pedagogy
– Provide additional faculty development and mentoring
opportunities
– Expand number and variety of community partners
• Strategy to achieve goals
– Intentional collaboration through shared SLO’s and integrative
learning
• CSC: co-curricular and community service focus
• CASL: academic service-learning focus
– Reach a greater number of faculty and support their efforts to
incorporate service-learning pedagogy to achieve SLO’s
Ambassadors for a Peaceful
Multicultural World
A Peer Facilitated Diversity Workshop Program
Our Team
Amber Scott and Juanita Hall – program coordinators &
trainers (through Anti-Defamation League)
Our Student Ambassadors 11 (05/06); 12 (06/07)
2005/2006
2006/2007
2006-07 Assessment Goals
Our Goals:
• To foster an appreciation for diversity
• To improve the campus climate for diversity
Our Study :
• To determine the effectiveness of the workshops in achieving our goals
– What student participants learned throughout 2 years of workshops
– Did their learning fit our identified Learning Outcomes
• To discover which exercises achieved the identified learning outcomes.
Assessment Design
We collected post-workshop reflections from
participants at the close of each workshop:
• Students responded to the following prompt:
“Please tell us what you learned about yourself or others
as a result of your participation in this diversity workshop.”
• Reflections separated by workshop and
Ambassador teams: for the Ambassador’s own
personal reflections and self-evaluation, and to
match Learning Outcomes with particular
exercises.
• Reflection packet was dated and labeled with
faculty names, class, and facilitators.
Data Collection
Reflections (312) from:
• 9 workshops in 2005/2006 (132)
• 10 workshops in 2006/2007 (180)
Where: majority in freshmen seminar
classes; others included leadership
retreats, a student government
meeting, and individual classes.
Analysis of Data
Process
• Read for themes, coded, quantified
• Inter-rater reliability (2 readers)
Results
• Student reported their learning matched Learning Outcomes
– All 3 “Appreciate Diversity” Learning Outcomes
– 4 of the 5 “Grow in Identity and Values”
What was learned?
• Workshops are useful for the character development of our
students
• Workshops help enhance the campus climate for diversity
• Certain diversity exercises were more effective than others
in promoting student growth and appreciation of diversity
Analysis of Data
2005/2006 Workshops
132 Student participants
Appreciate Diversity
Grow in Identity and Values
Understand
social
importance
of diversity
Awareness
of
similarities
and
differences
Increased
respect &
understanding
for people of
different
cultures
Learned
nothing
new
(already
knew)
Understand &
express
sense of
responsibility
to
community &
world
Understand
& express
identity in
terms of
their
diversity
Awareness
of self in
relationship
to own
background
& others
Compassion
&
commitment
to justice
Code 1
Code 2
Code 3
Code 4
Code 5
Code 6
Code 7
Code 8
32
57
27
12
10
29
47
29
2006/2007 Workshops
41
63
18
180 student participants
9
3
39
47
31
2007-08 Goals
Recommendations
• Ambassador workshops to occur in all freshmen
seminars
• Use the most effective exercises in the workshops
Assessment Goal
• A greater # of students will report Learning
Outcomes:
1. Understand and express an increased understanding
and respect for people of different cultures
2. Demonstrate both compassion toward others and a
commitment to justice
3. Understand and express their sense of responsibility to
their community and the world
2007
Assessment Symposium
Learning Together
May 22, 2007