Transcript Reading the Tea Leaves: IDEA Results and Student Learning

READING THE TEA LEAVES:
IDEA RESULTS AND STUDENT
LEARNING
GOALS AND OBJECTIVES
Goals: 1. To help faculty respond critically to
student generated course evaluation; 2. To help
faculty improve student learning.
 Outcomes: Faculty participants will

Better understand the relationship between chosen
objectives and the IDEA Diagnostic Form Report;
 Recognize the connection between objectives and
classroom teaching methods;
 Have access to resources that will help them achieve
greater student learning;
 Understand additional evaluation tools.


[A]reflective analysis on the part of the
instructor can be instrumental in
promoting instructional improvement
(Braskamp & Ory, 1994). In addition, the
instructor is the only person who can
supply certain kinds of information
needed by those charged with making
such evaluations, including information
about course objectives; readings,
assignments, and other learning
activities; the creation of instructional
materials or learning opportunities;
procedures for appraising student
achievement; results of, and course
modifications based on, classroom
research and other faculty efforts
directed to improving instructional skills.
ALL INFORMATION IN THIS PRESENTATION IS CREDITED TO THE IDEA CENTER
CHOOSING OBJECTIVES
Most instructors will be unable to address more
than 3-5 objectives
 Select with these three questions:

 Is
this a significant part of the course?
 Do I do something specific to help the students
accomplish this objective?
 Does the student’s progress on this objective affect
his or her grade?
ESSENTIAL/IMPORTANT AND
THE SCORE YOU GET
WHAT DO THE OBJECTIVES MEAN?
IDEA’S NOTE ON SELECTING OBJECTIVES
1. Gaining factual knowledge (terminology,
classifications, methods, trends)
 2. Learning fundamental principles,
generalizations, or theories
 These objectives are mainly cognitive in nature as
distinguished from affective objectives, which
focus on feelings or attitudes. They are closely
related. Both are concerned with the acquisition of
information or knowledge.
 Stresses learning at its most basic level.

APPLYING WHAT HAS BEEN LEARNED TO SOLVE PROBLEMS,
MAKE DECISIONS, OR PERFORM SPECIALIZED FUNCTIONS:
3. Learning to apply course materials (to improve rational thinking,
problem solving and decisions)
4. Developing specific skills, competencies and points of view
needed by professionals in the field most closely related to this
course
 The first of these emphasizes applications of principles, theories,
and concepts to solve a problem or arrive at a decision. Course
materials are employed to develop this general intellectual skill. It
is appropriate in courses where applications are intended to
develop problem-solving skills--analysis, synthesis, and
evaluation.
 Only in courses where a primary goal is to develop a generalized
skill in applying course materials to concerns or problems should
Objective 3 be selected as "Essential" or "Important."
EXPRESSIVENESS
6. Developing creative capacities (writing, inventing, designing,
performing in art, music drama, etc.)
8. Developing skills in expressing oneself orally or in writing
 Creativity connotes originality, imagination, and
expressiveness. While it is often associated with the fine arts
and literature, it is often relevant to aspects of science,
engineering, and other fields where design, research, and
innovation are required.
 For this objective to be classified as “Essential” or
“Important,” it should be a significant emphasis in the
course and there should be specific instruction or
assignments designed to promote its development.
INTELLECTUAL DEVELOPMENT
7. Gaining a broader understanding and appreciation of
intellectual-cultural activity (music, science, literature, etc.)
10. Developing a clearer understanding of, and commitment
to, personal values.
11. Learning to analyze and critically evaluate ideas,
arguments, and points of view.
 This objective should be selected as important or essential if
the focus of the course is on broadening intellectual
background and increasing the breadth of students’
interests and appreciations. Such courses are usually
directed to non-majors, although they may serve as
introductory courses for majors as well.
LIFELONG LEARNING
9. Learning how to find and use resources for
answering questions or solving problems.
12. Acquiring an interest in learning more by asking
questions and seeking answers.
 This objective is especially appropriate for courses
in which students have already acquired sufficient
background information and skill to permit them to
identify the trends or unresolved problems that are
likely to form the basis for future advances in the
field.
TEAM SKILLS
5. Acquiring skills in working with others as a member of a
team.
 Instructors who emphasize the development of team
skills will note the overlap with several other objectives
included on the IDEA list, including Objective 8
Communication Skills, Objective 3 Applications for
Problem Solving, and Objective 6 Creative Capacities. If
the expectation is that all of these skills will be
developed, together with skills involving effective
interpersonal relationships, then this objective should
be identified as “Important” or “Essential.”
LET’S CHOOSE SOME
OBJECTIVES
DURING THE SEMESTER
Post Objectives to your syllabus;
 Add objectives on each assignment and exam;
 Emphasize the connection between the class
discussion and the objectives and learning
outcomes;

PART 2: THE RESULTS
AN INTERPRETIVE GUIDE
CLICK FOR IDEA NOTES






An overall index of teaching effectiveness (PRO=Progress on Relevant
Objectives) combines ratings of progress on the objectives identified
by the instructor as Important (weighted “1”) or Essential (weighted
“2”) The IDEA Center regards this as its single best estimate of
teaching effectiveness.
Two additional overall measures of teaching effectiveness are shown
on the report. These are the average ratings of two items using a 5point scale
1. Overall, I rate this instructor an excellent
teacher.
2. Overall, I rate this course as excellent.
As an index of teaching effectiveness, the average of these two
ratings is commonly regarded as about equal in value to the
“Progress on Relevant Objectives” index described above.
IDEA DIAGNOSTIC FORM REPORT:
PAGE 1


Your report compares your average ratings to results for
three different groups of classes. The first comparison
group is with all classes in the standard IDEA database,
and is always reported. The other two are reported only
if enough classes were available to provide a stable
basis for comparison. These consist of (1) all classes in
the same discipline as the class in question and (2) all
classes at your institution.
Pay attention to the 5 point scale numbers. The
converted numbers are useful only insofar as they
measure your results against other faculty. They are not
necessarily a measure of progress on relevant
objectives.
ENGLISH 2325: FALL 2009
DIAGNOSTIC REPORT--PAGE 2




The first portion of Page 2 lists the 12 objectives included on the
IDEA form and summarizes student ratings on those you selected as
either Important or Essential. The main purpose is to help you focus
your improvement efforts.
Note the % does not include student’s who respond with a 3.
Students described the class by comparing it to other classes they
have taken in terms of (1) amount of reading, (2) amount of work in
non-reading assignments, and (3) difficulty. Average ratings are
compared with “All classes” in the IDEA database.
Students described their motivation by making self-ratings on the
three items listed at the bottom of Page 2. These characteristics have
been found to impact student ratings of progress.
DIAGNOSTIC REPORT—PAGE 2
DIAGNOSTIC REPORT--PAGE 3


The main purpose of instruction is to facilitate progress on objectives that
the instructor selects as Important or Essential.
The IDEA Center has conducted many studies that relate ratings on each of
these “methods” to ratings of student progress on the 12 learning
objectives. Through these studies, 7-10 methods that are most closely
related to progress on each of the 12 objectives for classes of different sizes
have been identified. Although there is some overlap, there are distinct
differences in the methods that facilitate progress on the 12 objectives;
there are also some differences among class sizes. The objectives for which
a given method is especially helpful in promoting learning are identified in
the column titled ”Relevant to Objectives.” The last column proposes an
action for you to take, depending on the relevance of the item and how
students rated it. If the rating for a relevant item was well above the IDEA
average, it is described as a “Strength to retain”; if the rating was well below
average, you are advised to “Consider increasing use”; and if it was in the
average range, it is suggested that you “Retain current use or consider
increasing.”
DIAGNOSTIC REPORT--PAGE 3
CONNECT PAGE 3 TO PAGE 2
Retain current use or consider increasing.
 Why? Only 59% of the students marked that I
“inspired” them to set and achieve goals. If I
focus on improving that element, I will improve
their progress on all selected relevant
objectives.
 IDEA recommends: IDEA Item #15

EXAMPLE 2
To improve their progress on Objective 27 and
31, I need to focus some attention on asking
students to share ideas.
 IDEA Recommends: IDEA Item #16
 What I find, based on IDEA data, is the two
issues are linked.

DIAGNOSTIC REPORT—PAGE 4
IDEA tells us that page 4 is for “those who want to
confirm scores reported on pages 1-3.
 We can also use page 4 to determine if our
Objectives correlate to the student experience.


I’m surprised my students focused on Developing
creative capacities. It’s not marked as an
Objective, but it is worth my time to examine the
course to see why they saw it as important.
DIAGNOSTIC REPORT--PAGE 4
HOW TO IMPROVE?
IDEA NOTES ON LEARNING
(CLICK TO LINK TO IDEA NOTES)
IDEA NOTES ON INSTRUCTION
(CLICK TO LINK TO IDEA NOTES)
ADDITIONAL EVALUATION TOOLS

Mid-semester evaluations (coming soon)
 Find

out in March so you prevent it by May;
Short Answer Response Questions
 Ask
your students specific questions related to
course organization, content, texts, etc.

Small group analysis
 Colleague
or CITR Director can visit a class to
discuss progress on objectives and learning
outcomes.
THE GOAL IS TO RESPOND CRITICALLY TO THE
NUMBERS, NOT USE THE NUMBERS TO BE
CRITICAL