Insight Improvement Impact ®

IDEA Student Ratings of Instruction

University of Alabama Birmingham September 11, 2012 Shelley A. Chapman, PhD

Plan for this Session • What Makes IDEA Unique • Conditions for Good Use • Reflective Practice Framework • Student Learning Framework • Faculty Information Form • Interpreting Reports • Questions and Answers

Insight Improvement Impact ® • Individual Development and Educational Assessment • Kellogg Grant in 1975 • Non-profit Organization 2000 • Mission To help colleges and universities as they seek to improve teaching, learning, and leadership

What makes IDEA unique?

4.

5.

6.

1.

2.

3.

Focus on Student Learning Focus on Instructor’s Purpose Adjustments for Extraneous Influences Validity and Reliability Comparison Data Flexibility

Conditions for Good Use The instrument • Focuses on learning • Provides suggested action steps for teaching improvement

Conditions for Good Use The Faculty • Trust the process • Value student feedback • Are motivated to make improvements

Conditions for Good Use • • • Campus Culture Teaching excellence - high priority Resources to improve provided Student ratings - appropriate weight

Conditions for Good Use The Evaluation Process • 30-50% of evaluation of teaching • 6-8 classes, more if small (<10) • Not over-interpreted (3-5 performance categories)

Reflective Practice using Individual Reports

Try new ideas

Improve Collect Feedback

Online, Paper Talk with colleagues

Reflect & Discuss Interpret Results

What the reports say and what they mean

Read & Learn

IDEA resources that are keyed to reports

Student Learning Framework: 2 Assumptions Assumption 1: Types of learning must reflect the instructor’s purpose.

Student Diagnostic Form Assumption 2: Effectiveness determined by students ’ progress on objectives stressed by instructor

Student Learning Model Specific teaching behaviors are associated with certain types of student progress under certain circumstances.

Circumstances Teaching Behaviors Student Learning

Student Learning Model: Diagnostic Form

Teaching Behaviors

Items 1-20

Student Learning

Items 21-32 Circumstances Students: Items 36-39, 43 Course: Items 33-35 Summary Items: 40-42 Research Items: 44-47 Up to 20 extra items

Student Learning Model: Short Form

Teaching Behaviors Student Learning

Items 1-12 Circumstances Students: Items 13-15 Summary Measures: Items 16-18 Experimental Questions: Items 14 20 Additional Questions

Faculty Information Form (FIF)

FIF: Selecting Objectives • 3 5 as “Essential” or “Important” • Is it a significant part of the course?

• Do you do something specific to help students accomplish the objective?

• Does the student’s progress on the objective influence his or her grade?

Be true to your course.

The Average Number of Objectives Selected by UAB: Spring 2012

Common Misconception #1 Students are expected to make significant progress on all 12 learning objectives in a given course.

Common Misconception #2 Effective instructors need to successfully employ all 20 teaching methods in a given course.

Relationship of Learning Objectives to Teaching Methods

Common Misconception #3 Faculty Evaluation The 20 teaching methods items should be used to make an overall judgment about teaching effectiveness.

Course Description Items (FIF) • Used for research • Best answered toward end of term • Do

NOT

influence your results Bottom of Page 1 Top of page 2

IDEA Online

IDEA Online: FIF Delivery • Email delivery/reminders • Start/end dates determined by Institution • Access is unlimited while available • Questions can be added to student survey • Objectives can be copied from previously completed FIFs

Copying Objectives

IDEA Online: Student Survey Delivery • Email/Course embedded URL • Blackboard Building Block • Email reminders • Start/end dates Determined by Institution • Submission is confidential and restricted to one

Online Response Rates – Best Practices • Create value for student feedback • Monitor and Communicate through multiple modalities: • Twitter • Facebook • Other • Prepare Students • Talk about it • Syllabus

Example: Course Syllabus

IDEA Center Learning Objective Course Learning Outcomes Objective 3

: Learning to apply course material (to improve thinking, problem solving, and decisions) Students will be able to apply the methods, processes, and principles of earth science to understanding natural phenomena

Objective 8

: Developing skill in expressing myself orally or in writing Students will think more critically about the earth and environment Students will be able to present scientific results in written and oral forms

Reflective Practice with IDEA Improve Collect Feedback

IDEA Student Ratings of Instruction

Reflect & Discuss Read & Learn Interpret Results

Individual Reports Group Summary Reports Benchmarking Reports

Diagnostic Report Overview  Page 1 – Big Picture  How did I do?

 Page 2 – Learning Details  What did students learn?

 Page 3 – Diagnostic  What can I do differently?

 Page 4 – Statistical Detail  Any additional insights?

The Big Picture

Your Average

(5-point Scale)

Raw A. Progress on Relevant Objectives 1 Four

objectives were selected as relevant (Important or Essential —see page 2) 4.1

Adj.

1 If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average.

4.3

Progress On Relevant Objectives 4.3 + 4.3

4.1

4.2

3.6

4

5

Summary Evaluation: Five-Point Scale Report Page 1

A.

Progress on Relevant Objectives Four

objectives were selected as relevant (Important or Essential —see page 2)

Overall Ratings

B. Excellent Teacher C. Excellent Course

D. Average of B & C 50% 25% 25% Summary Evaluation (Average of A & D) Your Average Score (5-point scale) Raw Adj.

4.1

4.3

4.7

4.1

4.4

4.3

4.9

4.4

4.7

4.5

Adjusted Scores

Adjusted Scores • Student Work Habits (#43DF, #13SF) • Student Motivation (#39DF, #15SF) • Class Size (Enrollment, FIF)

Understanding Adjusted Scores

Impact of Extraneous Factors • Gaining Factual Knowledge – Average Progress Ratings Work Habits (Item 43) High High Avg.

Hig h 4.48

Student Motivation (Item 39) High Avg.

4.38

Avg.

4.28

Low Avg.

4.13

Low 4.04

4.38

4.29

4.14

3.96

3.76

Average Low Avg.

Low 4.28

4.15

4.11

4.14

4.05

3.96

4.01

3.88

3.78

3.83

3.70

3.58

3.64

3.51

3.38

Technical Report 12, page 40

Impact of Extraneous Factors • Gaining Factual Knowledge – Average Progress Ratings Work Habits (Item 43) Student Motivation (Item 39) High Hig h 4.48

High Avg.

4.38

Avg.

Low Avg.

Low High Avg.

4.38

4.29

Average Low Avg.

Low 4.01

3.70

3.58

3.51

3.38

Technical Report 12, page 40

When to Use Adjusted Scores for Personnel Decisions Are adjusted scores lower or higher than raw scores?

Lower Higher Do raw scores meet or exceed expectations?

* No Yes Use adjusted scores Use raw scores *Expectations defined by your unit.

Comparisons (Norms): Converted Averages • Able to compare scores on the same scale • Use T Scores • • Average = 50 Standard Deviation = 10 •

They are not percentiles or percentages

Comparisons (Norms): Converted Averages

Comparison Scores Distribution Gray Area 40% 10% Much Lower 20% Lower Similar 20% Higher 10% Much Higher

Comparison Scores

Using the Report to Improve Course Planning and Teaching

Page 2: What did students learn?

Suggested Action Steps

#16 #18 #19

POD-IDEA Notes IDEA Website

POD-IDEA Notes • Background • Helpful Hints • Application for online learning • Assessment Issues • References and Resources

References and Links to Helpful Resources are Provided

IDEA Papers • • Resources for Faculty Evaluation Faculty Development

Reflective Practice

Try something new

Improve Collect Feedback

Paper or Online Meet with colleagues to reflect

Reflect & Discuss Interpret Results

Interpret Reports

Read & Learn

POD-IDEA Notes IDEA Papers

Questions ?

www.theideacenter.org

Visit our IDEA Help Community!