Transcript Improving and Assessing Instructional Effectiveness Theodore Frick Department of Instructional Systems Technology School of Education Indiana University Bloomington Invited Lecture to IUB Medical Sciences Faculty October 14,

Improving and Assessing
Instructional Effectiveness
Theodore Frick
Department of Instructional Systems Technology
School of Education
Indiana University Bloomington
Invited Lecture to IUB Medical Sciences Faculty
October 14, 2002
Overview
Confounding of instructional
effectiveness by other factors
 Types of knowledge about education
 Type 2, 3 and 4 examples
 Recommendations

October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
2
The Logic of Effectiveness
Before instruction
After instruction
Conclusion
1. Student nonmastery
Student mastery
Instruction appears to be
effective enough, but …
2. Student nonmastery
Student nonmastery
Instruction apparently was not
effective enough
3. Student mastery
Student mastery
Can't tell if instruction is
effective, since student had
achieved the goal before
instruction began
4. Student mastery
Student nonmastery
??? Something wrong with the
content - factual errors,
incorrect procedures, bad
models ??? Bad luck ???
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
3
Confounding Factors
1.
2.
3.
4.
Learning occurs without intentional
instruction.
Learning occurs despite poor
instruction.
We’re not the only teachers a student
has.
Results of instruction may occur LONG
afterwards.
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
4
Conflation of Research Methods and
Outcomes
In addition to the confounding problem in
determining effectiveness of instruction,
there has been a lot of debate in
education about inquiry methods – e.g.,
quantitative vs. qualitative.
 Continuing problem of bridging research
and practice in education

October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
5
Consider this Debate
Imagine for a moment three carpenters arguing about
which tool is best.






Quanta: "Hammers and nails are clearly superior."
Qualia: "I disagree. Screwdrivers and screws are
much more effective."
Performa: "You're both wrong. Saws are best for
cutting wood."
Quanta: "Who said anything about cutting wood? I
thought we were talking about fastening wood
together."
Qualia: "Right. Who needs saws?"
Performa: "I do. I need to cut this board in half."
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
6
Premise
The kind of knowledge about
education that we create through
disciplined inquiry determines what
research methods are appropriate and
useful.
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
7
Types of Educology: Knowledge
about Education

Methodology of Theory Building (Elizabeth
Steiner, 1988) (link)
Non-axiological knowledge – what is
 Axiological knowledge – pertaining to values:




Instrumental value: good for – what is effective
Intrinsic value: good in itself – what is worthwhile
The Dependability of Behavioral
Measurements: Theory of Generalizability for
Scores and Profiles (Lee Cronbach, et al.,
1972)
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
8
Six Types of Knowledge (Frick)
Scope of Knowledge
Purpose of Inquiry
Unique
Generalizable
What is?
1
2
What is effective?
3
4
What is worthwhile?
5
6
See Frick: R690 Syllabus for further details
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
9
Six Types of Knowledge (Frick)
Scope of Knowledge
Purpose of Inquiry
Unique
Generalizable
What is?
1
2
What is effective?
3
4
What is worthwhile?
5
6
Example: Type 1 Knowledge Claim
‘Elizabeth Steiner is an educational philosopher.’
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
10
Six Types of Knowledge (Frick)
Scope of Knowledge
Purpose of Inquiry
Unique
Generalizable
What is?
1
2
What is effective?
3
4
What is worthwhile?
5
6
Example: Type 2 Knowledge Claim
‘Students who spend more time engaged successfully in
arithmetic and reading tasks score more highly on achievement
tests in those content areas.’ (Fisher, et al., 1976)
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
11
Six Types of Knowledge (Frick)
Scope of Knowledge
Purpose of Inquiry
Unique
Generalizable
What is?
1
2
What is effective?
3
4
What is worthwhile?
5
6
Example: Type 3 Knowledge Claim
‘The DISTAR program for teaching arithmetic and reading to
elementary students works.’
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
12
Six Types of Knowledge (Frick)
Scope of Knowledge
Purpose of Inquiry
Unique
Generalizable
What is?
1
2
What is effective?
3
4
What is worthwhile?
5
6
Example: Type 4 Knowledge Claim
‘Instruction is most effective when it is problem-based, activates prior
learner knowledge, demonstrates what is to be learned, provides
opportunities for guided practice, and encourages integration with
everyday life.’ (Merrill, 2001)
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
13
Six Types of Knowledge (Frick)
Scope of Knowledge
Purpose of Inquiry
Unique
Generalizable
What is?
1
2
What is effective?
3
4
What is worthwhile?
5
6
Example: Type 5 Knowledge Claim
‘The corporal punishment policy in Houston schools is a bad
policy.’
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
14
Six Types of Knowledge (Frick)
Scope of Knowledge
Purpose of Inquiry
Unique
Generalizable
What is?
1
2
What is effective?
3
4
What is worthwhile?
5
6
Example: Type 6 Knowledge Claim
‘Teachers and students should respect each other.’
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
15
Do not confuse ends and means
The kind of knowledge about education that we
create through disciplined inquiry determines
what research methods are appropriate and
useful.



Outcome of research is knowledge.
Inquiry methods are a means to that end.
Criteria for evaluating adequacy of research
methodology are NOT the same for each
knowledge type – e.g.,


October 14, 2002
criteria for statistical inference to a broad population do not
apply to Types 1, 3, and 5;
effectiveness not of concern in Types 1, 2, 5, and 6, but
main concern in 3 and 4.
Improving and Assessing Instructional
Effectiveness -- Ted Frick
16
For today…
Focus on a few examples of Types 2, 3
and 4 knowledge of education
Type 2: Academic learning time research
 Type 3 outcomes: Program/product
evaluation – Kirkpatrick’s 4 levels (1959)
 Type 4 process: Effective Web Instruction –
Frick & Boling (2002)
 Type 4: Principles of 5-Star Instruction –
David Merrill (2001)

October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
17
Six Types of Knowledge (Frick)
Scope of Knowledge
Purpose of Inquiry
Unique
Generalizable
What is?
1
2
What is effective?
3
4
What is worthwhile?
5
6
Type 2 Example: Relationship of academic
learning time and student achievement
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
18
Type 2: Academic Learning Time
Task Success
Engaged Time
Allocated Time
ALT is positively correlated with academic
achievement in the same content areas (Fisher,
et al., 1976; Rieth & Frick, 1983; Berliner; 1985).
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
19
Six Types of Knowledge (Frick)
Scope of Knowledge
Purpose of Inquiry
Unique
Generalizable
What is?
1
2
What is effective?
3
4
What is worthwhile?
5
6
Type 3: Evaluation of a particular instructional
program or product
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
20
Type 3: Program/Product
Evaluation – Kirkpatrick’s Levels
Techniques for Evaluating Training
Programs -- Donald Kirkpatrick (1959)
1.
2.
3.
4.
Reaction (satisfaction)
Learning (achievement)
Behavior (transfer to real context)
Results (impact on organization/context)
More on Kirkpatrick’s levels from Encyclopedia of Educational Technology
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
21
Six Types of Knowledge (Frick)
Scope of Knowledge
Purpose of Inquiry
Unique
Generalizable
What is?
1
2
What is effective?
3
4
What is worthwhile?
5
6
Type 4 Example: how to create effective
instructional products – design theory
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
22
Type 4: Effective Web Instruction: An
Inquiry-Based Process – Frick & Boling
More
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
23
Type 4: First Principles of
Instruction – David Merrill
Integration
Activation
Problem
Application
October 14, 2002
Demonstration
Improving and Assessing Instructional
Effectiveness -- Ted Frick
24
Type 4: First Principles of
Instruction (cont’d)
“Learning is facilitated when:
Learners are engaged in solving real-world
problems.
2.
Existing knowledge is activated as a foundation
for new knowledge.
3.
New knowledge is demonstrated to the learner.
4.
New knowledge is applied by the learner.
5.
New knowledge is integrated into the learner’s
world.” (Merrill, 2001, p. 2)
Does your instruction rate 5 stars? A rating scale
1.
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
25
Six Types of Knowledge (Frick)
Scope of Knowledge
Purpose of Inquiry
Unique
Generalizable
What is?
1
2
What is effective?
3
4
What is worthwhile?
5
6
Type 3: Indiana University physician education
program
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
26
Recommendations
In physician education, focus on Type 3
knowledge:
Apply Merrill’s 5 principles as criteria for
design. See video (requires RealPlayer).
 Use Kirkpatrick’s 4 levels of evaluation for
your programs at IU.

October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
27
Questions?

This presentation is available at:
http://www.indiana.edu/~tedfrick/improveinstruction.ppt

Contact: [email protected]
October 14, 2002
Improving and Assessing Instructional
Effectiveness -- Ted Frick
28