Transcript Document

The Driver Behind
Assessment: Intellectual
Curiosity about Research or
Study Questions that Lead
to Changes in Pedagogy
Peggy L. Maki
[email protected]
Assessment Consultant, Editor and Writer
Presented at University Faculty Conference
Pepperdine University
Hyatt Regency Newport Beach, October 3, 2008
1
Foci
 Research on Learning that Informs Teaching,
Learning and Assessment
 Case Studies that Illustrate How Assessment
Results Lead to Effective Changes in Pedagogy
and Educational Practices
 Research or Study Questions that Guide Inquiry
into Student Learning
2
 The Design or Selection of Direct and Indirect
Methods and Standards and Criteria of
Judgment
 Development of a Plan to Answer Your
Research or Study Question and Develop
Research-Based Curricula: A Plan Used in
Research
3
How People Actually Learn
“I reverted to what I learned about trigonometry
from how I learned trigonometry in my home
country. I could never follow what the American
faculty member was telling us to do —I learned it
differently.” (international student)
“I was supposed to diagnose a patient the way the
faculty member described, but that’s not how I
really did it at all. Yet I still was the only one in my
class to present the correct diagnosis. I never
diagnosed the way I was taught but always made
the correct diagnosis.” (neurologist)
4
 “I still use my fingers to count.” (student)
 “I never did well on memory tests about
dosages of medicine to prescribe because I
knew as a Veterinarian that I would be able to
look up the dosages. I would ask instead:
‘Observe me diagnosing an ailment, identifying
the treatment, and then looking up the dosage
needed.” (Veterinarian)
5
Some Things We Know about Learning That
Inform the Relationship among Teaching,
Learning, and Assessment
 Learning is a complex process of interpretation-
-not a linear process
 Learners create meaning as opposed to
receive meaning
 Knowledge is socially constructed (importance
of peer-to-peer interaction in high impact
practices such as learning communities and
service learning)
6
 People learn differently—prefer certain ways of
learning (learning inventories, such as Solomon
and Felder:
http://www.engr.ncsu.edu/learningstyles/ilsweb.
html ; Teaching Style Inventory, such as Pratt:
http://www.teachingperspectives.com/html/tpi_fr
ames.htm)
 Deep learning occurs over time—transference
7
Approaches to Learning
 Surface Learning
 Deep Learning
8
 Meta-cognitive processes are a significant
means of reinforcing learning (thinking about
one’s thinking)
 Learning involves creating relationships between
short-term and long-term memory
9
 Transfer of new knowledge into different contexts
is important to deepen understanding

NRC. 2001. Knowing What Students Know: The Science and Design of
Educational Assessment. Washington, D.C.
10
What lines of inquiry can we explore to
design the next generation of curricula-cocurricula design that responds to what we
are learning and can learn about student
learning to improve student achievement?
11
The four case studies in front of you
illustrate various ways in which faculty
have changed pedagogy, instructional
design, and strategies to improve student
learning based on students’ performance
in assigned work or after agreed upon
times to capture students’ learning along
the chronology of students’ studies.
Collectively identifying a problem in
student learning, faculty pursued the
reason for the problems they identified and
then developed alternative ways to
improve student learning.
12
At your tables, assign various people to read
these four case studies; then through
collaborative discussion at your table identify
problems you may already consistently seen in
student work or identify how you might work
together to identify common problems through
your assessment of student learning that would
promote collaborative discussion about
improving teaching and learning.
13
Research or Study Questions
that Guide Inquiry into
Student Learning
Couple and align learning outcome statements
with a research or study question about the
efficacy of educational practices along the
chronology of students’ learning.
14
Levels of Learning Outcome
Statements
Institution-level Outcome Statements (including GE or Core)
Department-,Program-, Certificate-level Outcome Statements
Course/Service/Educational Experience Outcome Statements
15
Characteristics of Outcome
Statements
 Describe learning desired within a context
 Rely on active verbs (create, compose,
calculate, construct, apply, for example with a
focus on the highest levels by the time
students’ graduate)
 Emerge from our collective intentions
over time
16
 Can be assessed quantitatively or
qualitatively during students’ undergraduate
and graduate careers
 Can be mapped to curricular and co-curricular
practices (ample, multiple and varied
opportunities to learn over time)
 Are written at a course, program, or the
institution-level
17
Distinguishing between
Objectives and Outcomes
 Objectives state overarching expectations
such as-Students will develop effective oral
communication skills.
OR
Students will understand different
economic principles.
18
Quantitative Literate Graduates
according to MAA Should be Able to:
1.
Interpret mathematical models such as formulas,
graphs, tables, and schematics, and draw
inferences from them.
2. Represent mathematical information symbolically,
visually, numerically, and verbally.
3. Use arithmetical, algebraic, geometric, and
statistical methods to solve problems.
19
4.
Estimate and check answers to mathematical
problems in order to determine reasonableness,
identify alternatives, and select optimal results.
5.
Recognize that mathematical and statistical
methods have limits.
(http://www.ma.org/pubs/books/grs.html) The Mathematics Association of
America (Quantitative Reasoning for College Graduates: A
Complement to the Standards, 1996).
20
Ethics—Students should be able
to…(institution-level)
 Identify and analyze real world ethical problems or
dilemmas, and identify those affected by the
dilemma.
 Describe and analyze the complexity and importance
of choices that are available to the decision-makers
concerned with this dilemma
21
 Articulate and acknowledge their own deeply held
beliefs and assumptions as part of a conscious value
system
 Describe and analyze their own and others’
perceptions and ethical frameworks for decisionmaking
 Consider and use multiple choices, beliefs, and diverse
ethical frameworks when making decisions to respond
to ethical dilemmas or problems.
California State University Monterey Bay: University Learning Requirements,
2002
22
Ways to Articulate Outcomes
 Adapt from professional organizations
 Derive from mission of
institution/program/department/service
 Derive from students’ work
23
 Derive from ethnographic process
 Derive from exercise focused on listing one or
two outcomes “you attend to”
 Draw from taxonomies, such as Bloom’s
24
How well does your outcome statement
meet characteristics of a good
statement? (Refer to pages 16-17)
Ask the person next to you to apply the
characteristics of a good outcome statement
to your outcome statement(s); then discuss
that person’s assessment of your statements.
How might each of you improve your
statements?
25
External Validation
 Advisory boards
 Recent alums
 Survey of individuals in a field
 Developments in Professional Organizations such
as AAC&U
26
Sample Research or Study
Questions that You Can Join to
Your Outcome Statements…
 How do students…

come to know material that we teach?

represent their learning to themselves?

initially construct meaning in a field, discipline, or even a
course?

create mental models?
27
 integrate
 store
new learning into previous learning?
and draw on previous learning?
 reposition
 develop
 reuse
or expand their understanding?
dispositions to learn over time?
or apply stored learning or transfer it?
28

build layers of complexity in their learning—
conceptual complexity, for example?

reposition or modify or change altogether long-held
understanding, misunderstanding, or beliefs?

learn or don’t learn as a result of demands of
time or “coverage”?

develop spiritual behaviors, actions, attitudes, values?
29
How well do your students…
 Integrate
 Transfer
 Apply or re-apply
 Re-use
 Synthesize
 Re-position their understanding of their GE
outcomes or outcomes in their major program of
study?
30
 Within a course
 Along the chronology of students’ studies
and educational experiences
 From one discipline or topic or focus to
another
 From one context or situation to another,
such as from courses to co-curriculum
31
Integrated Learning….
Cognitive
Psychomotor
Affective
Expressive
32
Questions about Pedagogy or
Other Educational Practices in
promoting….
 Recall and recognition
 Transfer
 Integration
 Synthesis
 Application and re-application
 Use and re-use
 Change in perspective or understanding
 Sustained learning

33
What Do You Want to Discover about
Teaching and Learning? Discovery
Questions
 Efficacy of kinds of pedagogy (problem-
based, experiential, didactic) that promote
complex problem solving in a discipline
 Efficacy of theory behind your teaching and
instructional design
 Efficacy of curricular or relevant course(s)
design or co-curricular design
34
 Efficacy of the use of educational experiences—
service learning, learning communities, for example
 Efficacy of intentional scaffolding through on-line or
face-to-face instruction along the curriculum
 Efficacy of the use of out-of-course assistance, such
as tutorials or software programs
 Efficacy of instructional design (computer-based, for
example)
35
 What strategies enable students’ to develop strong
conclusions (use of graphic organizers, for example)
 What kinds of representational models develop complex
conceptual understanding. Or--What kinds of
representations are conducive to learning in your field?
(Physics)
 What are the relationships between students’ study
habits and deep learning?
36
 What’s the extent to which students engage and
develop higher order thinking skills and critical
reflection in a discipline or across GE?
 What strategies enable students to transition from
thinking arithmetically to thinking algebraically?
 How do students’ beliefs affect conceptual
development?
37
 What strategies enable students to overcome
learning barriers or obstacles
 How do students’ levels of cognition affect their
conceptual development?
 How do educators’ epistemological views in
their fields, translated into instructional design,
foster enduring student learning?
38
 How well do students transfer their early
learning in a discipline or profession into their
later learning?
 How well do students transfer learning from
GE courses into their major program of study?
 How well do students transfer their GE
learning or major program learning into the life
outside of the class such as in community
service?
39
 How well do digital dialogue games or
other forms of technology stimulate
students’ reasoning or conceptual change?
 When students reposition their
understanding, is it based on a belief
revision or conceptual change and
restructured knowledge (talk alouds)?
 How effective are hypermedia technologies
in fostering complex problem solving?
40
 What strategies do students use to restructure
naïve or intuitive theories?
 How well do students build their own
knowledge based on the use of instructional
multi-media designs?
 What strategies do successful students use to
read and interpret texts, visuals, maps?
41
 What barriers do students face when they read
and interpret texts, etc. What strategies help
them overcome those barriers (vocabulary,
discourse organization, comprehension, math?)
(Philosophy example)
 How well do interactive discussions help
students construct knowledge?
42
What Is the Question You Want to
Answer about one of Your GE or
Program-level Outcomes?
 What’s your study question?
Or
 What’s your research question?
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------43
What Other Data Might You Need
to Answer Your Question?
 Baseline exercises, such as concept
inventories used in Physics, case studies used
over time, or simulations used over time
 Maps or inventories of practice
 Surveys or interviews with students about their
learning
44
 Transcript analyses of course-taking patterns
 Participation in co-curricular programs
 Educator interviews
45
 SALG results
 Syllabi analyses about kinds of in-class
assessments or methods of
teaching/learning
 Student think alouds
46
Think Alouds…
Quellmalz and Haydel (2003) found in cognitive analyses of
think-alouds that “ students were more likely to use
schematic and strategic knowledge on performance
assessments than on multiple-choice items.” Assessment
approaches that require students to construct and explain
thinking as they solve problems can measure distinct
components of inquiry and problem solving, including
stating research questions, posing hypotheses, planning
and conducting investigations, gathering evidence,
analyzing data, considering disconfirming evidence, and
communicating interpretations.
http://serc.carleton.edu/files/NAGTworkshops/Assess/QuellmalzEssay/pdf
47
The Design or Selection of Direct and
Indirect Methods and Standards and
Criteria of Judgment
“Every assessment is also based on a set of
beliefs about the kinds of tasks or situations
that will prompt students to say, do, or create
something that demonstrates important
knowledge and skills. The tasks to which
students are asked to respond on an
assessment are not arbitrary.“
National Research Council. Knowing what students know: The
science and design of educational assessment . Washington,
D.C.: National Academy Press, 2001, p. 47.
48
Assumptions Underlying
Teaching
Actual Practices
Assumptions Underlying
Assessment Tasks
Actual Tasks
49
Shulman on Assessment
Methods
 “… the first lesson regarding an assessment is
to take responsibility for locating its unavoidable
insufficiencies in relation to what it claims it can
measure….We do not seek one perfect
measurement instrument, but an array of
indicators that can be understood in relation to
one another.”
 Lee Shulman. “Principles for the Uses of Assessment in Policy and
Practice.”
50
What Tasks Elicit Learning
You Desire?
 Tasks that require students to select among
possible answers (multiple choice test)?
 Tasks that require students to construct
answers (students’ problem-solving and
interdisciplinary thinking abilities)?
51
When Will or Do You Seek
Evidence?
 Formative—along the way?
For example, to ascertain progress
or development against pedagogy
 Summative—at the end?
For example, to ascertain level of final
achievement
52
Direct Methods
 Focus on how students represent or
demonstrate their learning (meaning making)
 Align with students’ learning and assessment
experiences
 Align with curricular-and co-curricular design
verified through mapping
53
Possible Assessment Methods Higher
Education can Use to Learn More about How
Students Learn (See also handout on
methods)
 Student Assessment of Learning Gains (SALG)—ask
students to identify ways they actually learned across
components or elements of a lesson or course. Could be
extended across the program of study.
 Online journals that record how students make
meaning/solve problems
 Wikis (knowledge building sites)
 Classroom Response System (CRS)—clickers
54
 Assessment checkpoints based on layers of learning in
vertical themes (skill layers, factual layers, theoretical
layers, conceptual layers, interpretive layers, knowledge
layers, logic layers, methods layers, reasoning layers)
 Online discussion boards
 Small Group Instructional Diagnosis (SGID) conducted by
someone other than faculty teaching a course
http://www.ntff.com/ntff/sgid.doc
55
 Resulting patterns from engagement with interactive
computer simulated tasks that provide data on patterns
of actions, decisions, etc., and branch students forward
or backward
 Knowledge, decision, or procedural maps
http://classes.aces.uiuc.edu/aces100/ mind/c_m2.html
Spider Concept Map
56
What Criteria Will Be Applied to Student
Achievement so That You Can Make
Inferences about Students’ Achievement of
Your Outcome?

Skills

Knowledge

Habits of mind (disciplinary or interdisciplinary
habits of mind)

Ways of knowing
57

Dispositions—Spiritual?

Research strategies/approaches

Disciplinary conventions

Ways of problem solving (including
increasingly complex problems)
58
How Well Do These Criteria
Align with…
 Teaching practices
 Learning practices (how we position students to learn)
 Frequency of feedback
 Students’ learning histories
 Design and coherence of curriculum and co-curriculum
(multiple and diverse opportunities to learn)
59
Development of a Plan to Answer Your
Research or Study Question and
Develop Research-Based Curricula: A
Plan Used in Research
60
Contributions from Research on Student
Learning Based on…
 Deconstruction of a unit or course or the curriculum into
layers or components or elements
 Experimentation with pedagogy based on assumptions
about how students learn layers or components or
elements
 Assessment of student learning after each layer,
component, or element to ascertain the efficacy of specific
kinds of pedagogy for each layer
61
 Element-based, component based, or layer-based
student-directed questions
 Think alouds that ask students to construct and
explain thinking
62
Example: Deconstruct the curriculum based on
vertical themes, such as in a medical program:











Nutrition
Pain
Disability
Life cycle
Personal development
Communication
Evidence-based practice
Ethics & legal responsibilities
Psychological aspects of clinical practice
Pharmacology and therapeutics
Public health
63
 Deconstruct themes into elements or layers or
components across the curriculum
 Identify chronological pedagogy or forms of instruction
along those layers
 Develop assessment methods that align with pedagogy
or instructional design in each layer or component
 Use focus groups or surveys of students’ responses to
the pedagogy related to each layer or element
64
Example: VaNTh ERC (VanderbiltNorthwestern-Texas- Harvard/MIT
Engineering Research Center
Focuses on “real life” challenges in professional
education:
Turner and Thomas argue in Clear and Simple as the Truth
that writing skills are most successfully taught when they
are integrated with genuine (rather than contrived) activities
that build on past learning, create a real need for the new
skills, and offer an opportunity to learn those skills. As they
explain: “Intellectual activities lead to skills, but skills do not
generate intellectual activities” (p.4)
65
According to Hirsch, et als, “Instead of writing essays,
papers, and exams, students write to faculty and clients to
communicate important information about their projects:
for example, they write mission statements, report on
client meetings, synthesize the results of research,
prepare progress reports, and create slides for PowerPoint
presentations. Thus, as a communication course, EDC
sends a strong, clear message to students:
communication is an integral part of the design enterprise,
not merely a superficial matter of editing. Clear
communication advances creative problem-solving, the
heart of engineering design.” (Hirsch, et als., p. 4)
66
Results..
 Request student performance analysis that can
be aggregated and disaggregated according to
your research or study question, such as
performance based on students’ course taking
patterns, different pedagogies, different
contexts for learning
 Request narrative interpretation of student
performance (recall Case 4)
67
68
69
“It is always possible to defend the inspirational lecturer, the
importance of academic individuality, the value of
pressuring students to work independently, but we cannot
defend a mode of operation that actively undermines a
professional approach to teaching. Teachers need to know
more than just their subject. They need to know the ways it
can come to be understood, the ways it can be
misunderstood, what counts as understanding; they need to
know how individuals experience the subject. But they are
neither required nor enabled to know these things” (Diana
Laurillard, 6)
70
Selected Resources
Hirsch, P. Kelso, D., Shwom, B.,Troy, J. Walsh, J. Redefining
Communication Education for Engineers: How the NSF/VaNTH ERC is
Experimenting with a New Approach. Northwestern University,
Session 2261 (copy available at www.vanth/docs/016_2001.pdf)
Holbert,N. (February, 2008). “Shooting Aliens: The Gamer's
Guide to Thinking.” Educational Leadership. Vol 65. No.5.
Laurillard, D. (1993). Rethinking University Thinking: A
Framework for the Effective Use of Educational Technology.
London: Routledge
71
Maki, P. 2004. Assessing for Learning: Building a
Sustainable Commitment across the Institution. VA:
Stylus Publishing, LLC. (to be revised in 2009)
National Research Council. 2001.Knowing What
Students Know: The Science and Design of
Educational Assessment. Washington, D.C.
Physics Education Technology Project
(http://phetcolorado.edu/webpages/publicatons/phet_aapt-04pd
72
Quellmalz and Haydel. 2003. Center for Technology in Learning at
SRI International. Available at: http://sercc.carleton.edu/files/NAGT
workshops/Assess/QuellmalzEssay/pdf.
Shulman. L. 2006. “Principles for The Uses of Assessment in Policy
and Practice.” President’s Report to the Board of Trustees of the
Carnegie Foundation for the Advancement of Teaching. CA:
Stanford. Available at:
www.teaglefoundation.org/learning/resources.aspx#assessment
Material presented in this workshop will be integrated into Maki’s 2009
2nd Edition of Assessing for Learning. Stylus Publishing, VA
73