Transcript Using Measurable Outcomes to Evaluate Tutor Programs

Innovative Educators Webinar
December 1, 2010
Jan Norton, Presenter








Terminology
The Black Box
Timetables
Data/Measurement
Data/Measurement Cautions
Program Outcomes & Examples
Student Outcomes & Examples
Closing & Questions








Visions
Missions
Values
Goals
Objectives
Outcomes
Program vs. Student Outcomes
Learning Outcomes
.
Something
Happens





Actions leading to specific, observable results
What students can expect to learn
How students or programs will be different as
a result of what happens in the box
What knowledge, skills, and abilities will be
improved
What measurable changes occur in tutoring
programs

Scope of assessment
 ≥ 6 years
 2 years
 1 year
 1 semester
 1 course
 1 assignment
 1 tutoring session

Planning for Assessment
 Continual processes
 Multiple times per semester
 Once or twice per semester
 Two or three times per year
 Once per year
 Every two years
 Every 4 years

Why Measure

What to measure
 Before (Benchmarks)
 During
 After (Outcomes)

How to measure
 Program-generated data
 Course-specific data
 Campus data
 Campus surveys
 Standardized tools
 Locally developed tools

More about locally developed tools
 Surveys
 Likert-type scales
 Explicated scales
 Behavioral rubrics
 Self assessment by student
 Tutor assessment
 Observer assessment






Direct vs. Indirect Measures
Data access and consistency
Realistic limitations
Cooperation
The Baseline Challenge
Time & money
(and staff and comfort and ……)

Possible Areas of Assessment Focus
 Establishing baselines
 Usage
 Satisfaction or quality
 Level/variety of services
 Staff development
 Reaching certification or meeting standards

Outcome Statement
 As a result of the new marketing strategy, the
number of students attending group tutoring
sessions will increase by 10% in Spring 2011.

Timetable
 One semester

Starting Measure (Benchmark)
 Average of past attendance

Ending Measure
 Workshop attendance records

Outcome Statement
 During 2010-11, the number of tutors who earn
CRLA Level I certification will increase by 5%.

Timetable
 One academic year

Starting Measure (Benchmark)
 Number of tutors achieving Level I in previous
year(s)

Ending Measure
 Number of tutors achieving Level I in 2010-11

Outcome Statement
 Outside reviewers using the NADE self-study
process will score the tutoring program at 85% or
above on all sections.

Timetable
 One year

Starting Measure (Benchmark)
 Internal review & self-study

Ending Measure
 External review

Outcome Statement
 As a result of tutor training sessions, tutors will
demonstrate an improved understanding of Bloom’s
Taxonomy.

Timetable
 One semester or year, depending on training schedule

Starting Measure (Benchmark)
 Pre-test

Ending Measure
 Post-test
TYPES


Affective
MEASURES

Qualitative
 Build confidence
 Interviews & focus groups
 Improve attitude
 Opinion surveys
 Reduce anxiety
 Observations or self reports
Cognitive

Quantitative
 Mastery of material
 Assignment & course grades
 Conscious strategies use
 Instrument scores
 Observation rubrics

Outcome Statement
 After being tutored for at least one hour, clients will
be more confident in their ability to accomplish the
academic task they focused on in tutoring.

Timetable
 One tutoring session

Starting Measure (Benchmark)
 Client self-report

Ending Measure
 Client self report

Outcome Statement
 Students who work with the study skills tutors
will demonstrate an increased ability to use
memorization strategies.

Timetable
 One tutoring session

Starting Measure (Benchmark)
 Tutor or observer checklist

Ending Measure
 Tutor or observer checklist

Outcome Statement
 75% of the students who work with a reading
tutor for at least two sessions will score at least
75% on the main ideas reading test.

Timetable
 Within three weeks

Starting Measure (Benchmark)
 Reading quiz at start of tutoring

Ending Measure
 Chapter reading test in class

Outcome Statement
 Students who attend at least 10 group tutoring
sessions for Algebra Review 090 will earn an
average grade of B or above in the course.

Timetable
 One semester, length of course

Starting Measure (Benchmark)
 Placement score into course

Ending Measure
 Course grade

Outcome Statement
 After attending mandatory tutoring for all 5 of their
developmental education courses, students will be
retained until the start of their junior year at a rate at
least 25% higher than developmental students for
whom tutoring was not required.

Timetable
 Two years

Starting Measure (Benchmark)
 Developmental course placement

Ending Measure
 Enrollment level

Outcome Statement
 Students attending SI for World Religions will “[apply]
previously understood information, concepts, and
experiences to a new situation or setting.” (CAS)

Timetable
 Within one semester

Starting Measure (Benchmark)
 Pre-survey

Ending Measure
 Post-survey
Check with your institutional research or
assessment office
 Review early drafts of outcomes statements
with your supervisor
 Pre-test any locally created measurement tool
 Be prepared for outcomes that are not met or
wildly exceeded
 Remember the black box: we do not own or
control the box, and our ability to see into it is
very limited
