Transcript What is IDEA? - Wilmington University

Understanding and Using Your
IDEA Evaluation Results
Nina Campanicki & Ernie Linsay
Faculty Development Day
March 21, 2009
What is IDEA?
INDIVIDUAL DEVELOPMENT
AND
EDUCATIONAL ASSESSMENT
IDEA EVALUATION COMPONENTS:
• Faculty Information Form
• Student Survey
–Diagnostic Form
• Individualized Diagnostic Report
FACULTY INFORMATION FORM
(FIF)
FACULTY INFORMATION FORM (FIF)
IDEA OBJECTIVES
FACULTY INFORMATION FORM (FIF)
IDEA OBJECTIVES:
• 3-5 “Essential” or “Important” should reflect
most significant aspects of course
• WU has specific IDEA objectives set in
advance for each course
FACULTY INFORMATION FORM (FIF)
CONTEXTUAL QUESTIONS
There are 6 Contextual Questions at the bottom of front page and the top of back page:
Please take a moment to fill in this information.
The IDEA Center uses this information to improve the interpretation of student ratings.
DIAGNOSTIC STUDENT SURVEY
LONG FORM
DIAGNOSTIC STUDENT SURVEY
• SPECIFIC TEACHING BEHAVIORS…
– Teaching Methods are Items: 1-20
• …INFLUENCE CERTAIN TYPES OF STUDENT
PROGRESS…
– IDEA Learning Objectives are Items: 21-32
• …UNDER CERTAIN CIRCUMSTANCES.
– Course Management/Content are Items: 33-35
– Student Characteristics are Items: 36-39, 43
• Other Data
– Summary Measures are Items: 40-42
– Experimental Questions are Items: 44-47
– Extra Questions are Items: 48-67
DIAGNOSTIC STUDENT SURVEY
FALSE ASSUMPTIONS
• Effective Instructors employ all 20 teaching
methods
• The 20 Teaching Methods Items are used to
make an overall judgment about teaching
effectiveness
• Students should make significant progress on
all 12 learning IDEA objectives
DIAGNOSTIC STUDENT SURVEY
BEST PRACTICES
• Overall: Students need to be informed about
the survey and the IDEA objectives should be
discussed throughout the course.
What Does Your IDEA Report Mean?
What were my students’ perceptions of the
course and their learning?
1. Overall, how effectively was this class taught?
2. How do I compare with other teachers?
3. Was I more successful in facilitating progress on
some IDEA objectives than on others?
What might I do to improve my teaching?
Info on Page 1
•
•
•
•
Number of students responding
Average scores – raw & adjusted
Converted scores
Comparison to data bases
Sample DF
Impact of Response Rates
• Number responding
<10
10-14
15-24
25-39
>40
Unreliable
Marginal reliability
Fairly reliable
Reliable
Highly reliable
• Representativeness of Results
<50%
50-64%
65-74%
75% +
inadequate to assure representativeness
may not be representative
probably representative
representative
Sample DF
Adjusted Scores Level Playing Field
• Fr/Soph/Gen Ed classes
– Students with lower motivation
– Adjusted scores > raw scores
• Jr/Sr/Grad classes
– Students are more motivated & better work habits
– Adjusted scores < raw scores
Sample DF
Adjusted Scores
•
•
•
•
•
Student Work Habits (#43)
Student Motivation (#39)
Class Size (Enrollment, FIF)
Student Effort (multiple items)
Course Difficulty (multiple items)
Sample DF
Impact of Extraneous Factors
• Gaining Factual Knowledge – Average Progress Ratings
Student Motivation (Item 39)
Work
Habits
(Item 43)
Low
Low
Avg.
Avg.
High
Avg.
High
Low
3.51
3.66
3.80
3.95
4.08
Low Avg.
3.60
3.76
3.91
4.05
4.07
Average
3.73
3.87
4.02
4.12
4.21
High Avg.
3.88
3.97
4.13
4.23
4.33
High
4.01
4.12
4.25
4.33
4.48
Technical
Report 12,
page 40
Converted Scores
• 5-point ratings converted to T-scores with “50”
being the middle
• Not percentiles
• Gray area (45-55T) represents 40% of faculty
Sample DF
Comparative Information
• All classes in IDEA database
– 128 institutions
– Updated only periodically
• IDEA discipline database
– Most recent 5 yrs
– Updated annually on Sept 1
– Minimum of 400 classes
• Institution (WU) database – all classes
– Most recent 5 yrs
– Updated annually on Sept 1
– Minimum of 400 classes
Sample DF
Example
IDEA Database All Classes
IDEA Discipline Database
Institution All Classes
5.0
5.0
5.0
4.9
4.9
4.9
4.8
4.8
4.8
4.6 = 63T
4.6 = 58T
4.6 = 55T
4.5
4.5
4.4
4.4
4.4
4.2
4.2
4.2
4.0
4.0
4.0
3.9
3.9
3.9
3.8
3.8
3.8
3.6
Info on Page 2
• Student ratings of learning on Essential and Important
IDEA objectives
• Description of course and students
Sample DF
Calculating “Progress on Relevant IDEA
Objectives”
Essential objectives have double weight of
Important objectives
Gaining factual…
Learning fundamental…
Learning to apply…
Learning how to find…
Sample DF
4.5 X 2 = 9.0
4.4 X 2 = 8.8
4.2 X 2 = 8.4
3.2 X 1 = 3.2
29.4/7 = 4.2
Info on Pages 3 & 4
Page 3
• Improving teaching effectiveness
• What might you do differently
Page 4
• Statistical details
Sample DF
Personal Comments
• IDEA results can spike up or down
• Grades do not influence IDEA results
• Knowledge Base in IDEA Center website has
great info about teaching & learning and
understanding the report
– www.theideacenter.org
– Item #3 in Web Resources for Teaching in Faculty
Forum under Faculty in WU home page
Sample DF
IDEA EVALUATION PROCESS
• How and When do you Receive the IDEA
Evaluations?
– You will receive a packet in your mailbox or via
email approximately 2 weeks before your course
ends.
IDEA EVALUATION PROCESS
• What is the Hard Copy Procedure?
– Select a student to proctor the evaluations.
• They will hand out and collect the Diagnostic and
Comment forms.
– Take your FIF out of the packet and…
• Write Your Name, Course Number and Time/Day
information on the Board for the students
• Tell the Student/Proctor where you will be and…
• Leave the room
IDEA EVALUATION PROCESS
• When the Student comes out of the
classroom
– Ask them to place your FIF into the envelope
– Direct the Student to the Nearest attended
Workstation or Drop-off Box
– You are Not Allowed to Handle the Packet after the
Students have Completed the Evaluations.
• Do not let the Student leave your Evaluation
Packet on an unattended desk
IDEA EVALUATION PROCESS
• What’s Next:
–All IDEA Evaluations are sent to New
Castle for processing
–IDEA Evaluations are grouped:
• By Division
• By Instructor (alpha-order)
IDEA EVALUATION PROCESS
• Then:
– They are packaged and sent to IDEA Center.
– Results returned from IDEA Center within 14 days
– Distribute results to the Deans within 3 days
– Deans and Program Coordinators review and send
you the Individual IDEA Reports
– You can discuss your IDEA Report with your
Program Coordinator or Faculty Development &
Support