Transcript Slide 1

Learning by Design: An Iterative
Mapping Strategy for Improved
Curriculum Design and
Assessment
By Arvid
Roach
Dr. Donald L. McEachron
Coordinator for Academic Assessment and Quality Improvement
School of Biomedical Engineering, Science and Health Systems
Drexel University
Goal: Properly sequenced and
implemented series of educational
experiences which transform
students into productive members
of a global society
Mechanism: Assessment-driven
and evidence-based curriculum
design
What is meant by productive
members of a global society?
• Combination of common and program-specific
elements
• Alumni activities and achievement
• Two major questions
– How to determine?
– How to measure?
From: http://www.aprilmoralba.com/projects/dslp.html
Example - Program Educational
Objectives
• Measures of Alumni Success (ABET, Inc.)
• Sources/Stakeholders
– Immediate/Short-term Perspective = Students
– Moderate/Long-term Perspective = External
Advisory Board
– Long-term Perspective = Faculty
• Iterative Development of PEOs
Example of PEO from Biomedical
Engineering
Leadership in Research, Innovation and Design
As a result, within a few years of graduation, the
graduate will have made significant or meaningful
contributions in his or her chosen field, either thorough
research publications and/or presentations, the
development of a product or process, obtaining patents
for new products and/or processes, or other evidence of
contributing to the advancement of knowledge,
particularly in fields integrating engineering and the life
sciences.
From: http://www.aprilmoralba.com/projects/dslp.html
How to Measure?
• Traditional method – Alumni Surveys
– Limited response rate
• Small Sample Size
• Volunteer Bias
• Potential Alternatives
– Social Networking – LinkedIn, etc.
– Internet Search/Data Mining
Goals Established – Design Can Begin
Guiding Principles
• Alignment-Centered (Quality Matters Initiative): each
aspect of the curriculum supports student learning in
order to achieve program goals
• Assessment-Driven: Assessments guide development of
educational methodologies
• Evidence-Based: Alumni and student performance guide
re-design and quality improvement
• Alignment refers to each element of the
curriculum supporting overall program goals
• To achieve program educational objectives
(PEOs), develop a set of student learning
outcomes (SLOs) such that success on these
outcomes predicts/supports success on the
PEOs.
• Examples of SLOs
– Certain Drexel Student Learning Priorities (DSLPs)
– ABET a-k criteria
What is an SLO exactly?
• SLOs are characteristics of
students at the time of
graduation. These characteristics
can include knowledge, skills,
attitudes, etc.
• Understand accounting and business terminology used in business
scenarios, and be proficient with commonly used office software
programs (Butte College, Business Education Program
http://www.butte.edu/departments/careertech/businessed/slos.html)
• To think philosophically about our existence in the world and to
demonstrate a philosophic approach to ethical issues (Seattle
University http://www.seattleu.edu/assessment/SLO.asp)
• The ability to analyze and evaluate artwork from various
perspectives and to receive responsively suggestions about and
criticisms of his or her own work from others. (Dickinson State
University Bachelor of Arts degree in Art
http://www.dsu.nodak.edu/Catalog/fine_arts/art_majors_minors.htm)
• Recognize the relationship between structure and function at all
levels: molecular, cellular, and organismal. (University of San
Francisco Bachelor of Science degree in Biology
http://www.usfca.edu/biology/outcomes.htm)
Technological
Skills
Professional
Presence
Problem-Solving
Abilities
Innovation &
Design
Ensure
Alignment
Between
PEOs and
SLOs through
Mapping
Workforce
Skills
Research
Abilities
Leadership
SLOs
PEOs
Leadership
Communication
Communication
Human
Resources and
Interactions
Personal
Engagement
Engagement
Ethics and
Professionalism
Ethical
Reasoning,
Behavior
Characteristics of SLOs
• SLOs are often Broad and thus Difficult to
Measure
• Need to be Decomposed into Measureable
Components for Assessment and
Evaluation
• Learning Indicators (LIs) (also known as
performance criteria or performance
indicators)
Next Step: Using SLOs, LIs and Rubrics to
Design Curriculum
• Assessment-Driven Performance Mapping
• Why Assessment-Driven?
• When you investigate how to measure an SLO or
LI, you must understand the fundamental nature
of the student outcome – what is the student
supposed to be able to know, understand, apply,
evaluate, etc.
Performance Mapping
• What are We Assessing?
• Measurable Components of SLOs = Learning
Indicators (LIs)
• Associate LIs with Developmentally
Appropriate Educational Experiences
• How? By Mapping LIs to Educational
Experiences
Performance Mapping is Iterative
Courtesy of Dr. Fred Allen
Map 1 – Learning Indicators vs. Educational
Experiences
• Set Performance Levels:
• Introduce; Reinforce; Emphasize
• Introduce; Practice; Review; Utilize
• Not Critical – Refine Later
• Associate Learning Indicators with Activity
• Small group (ex. Curriculum Committee)
• Use syllabi to associate criteria with course
• After initial analysis, sent to faculty for review
Utility of Map 1
• Reveals Uneven Distributions of
Performance Criteria
• Indicates Gaps or Over-Emphasis
• Begins to Reveal True Nature of Current
Curriculum
Map 2 – Educational Activities vs. Learning
Indicators
• First Map: Learning Indicators vs. Course
– Reveals Relative Importance of Various Performance
Criteria in the Curriculum
– Not Enough
• Second Map: Course vs. Learning Indicator
– Show Relative Roles of Courses in Curriculum
– Reveals Disconnects between Course Requirements
and Performance Criteria
– Uncovers ‘Core’ or ‘Gateway’ courses
Map 3 – Curriculum Sequence
• Place Performance on Learning Indicators in
a Temporal Context
• Look at When Learning Indicators are
covered as well as Where
• Can Reveal Additional Disconnects
– Learning as a Developmental Process
– Does the Curriculum Develop Learning Properly?
Mapping the Changes in Learning Development through
Increasing Expectations
Term 1
Term 5
Further Alignments and Iterations
• Each course and educational activity now
defined in context
• So far, mapping has proceeded from up from
educational experience (ex. course syllabus) to
align with program goals and objectives (SLOs
and PEOs)
• Now, chance to burrow down into the course
level to ensure alignment
Possible Flow Chart for SLO-Driven Course
Design
Curriculum
Map
Assign
LIs
No
No
No
Correct?
Adequate?
Yes
Develop
Performance
Standards
Develop
Assessment
Strategies
Develop
Instruc onal/Learning
Strategies
Evaluate
Performance
Yes
Refinements and Benefits
• Translate Levels into Bloom’s Taxonomy
– Reinforce the Developmental Aspects of
Learning
• Place Additional Resources into Key
Courses
– Many Learning Indicators
– Transitional Timing
Mapping with AEFIS
• Mapping can be done manually – not
recommended
• AEFIS provides for mapping and pivots for
all three types of maps
• Assessments can be incorporated into the
mapping
Enter or Import Required SLOs
(ABET, DSLPs, etc.)into AEFIS
ABET k
ABET a
Technological
Skills
Technology
Use
Problem-Solving
Abilities
Creative &
Critical Thinking
ABET e
What Are the
Targets? – Student
Learning Outcomes
(SLOs):
School Developed 9
Intellectual Pathways
Incorporating
both ABET (external
accreditor) criteria
and Drexel Student
Learning Priorities
ABET c
Innovation &
Design
Research and
Scholarship
ABET b
Research
Abilities
ABET i
Information
Literacy
Professional
Practice
ABET Program
Leadership
ABET g
Communication
ABET d
Human
Resources and
Interactions
Leadership
Communication
Self-Directed
Learning
Responsible
Citizenship
ABET j
Engagement
ABET h
ABET f
Ethical
Reasoning,
Behavior
Global
Competence
Ethical
Reasoning
Example - Communication
• SLO - Communication – The graduate employs an
understanding of audience, purpose and context to
communicate effectively in a range of situations using
appropriate media while displaying a significant aptitude
for presenting scientific and technical materials to
diverse audiences.
• Supported By
– ABET g Ability to communicate effectively
– Drexel Learning Priority - Communication: Employ an
understanding of audience, purpose and context to
communicate effectively in a range of situations using
appropriate media
From: http://www.aprilmoralba.com/projects/dslp.html
Align Required SLOs to Program
Level SLOs
Create Measurable Sub-elements Capable of being Mapped
into Curriculum (Performance criteria; Performance
Indicators; Learning Indicators, etc.)
Add Learning Indicators/Rubrics to AEFIS
Map SLOs or Subcomponents to Courses
Collect Direct Assessment Data
Display Direct Assessment Data
Assessment Data is
Evidence
• Collect, Analyze, Evaluate
• Determine appropriate course of action
• Re-design curriculum
• Collect, Analyze, Evaluate to determine the
effectiveness of the changes
How Much Assessment?
Incoming
Forma ve Assessments
Students
Educa onal
Ac vity 1
Educa onal
Ac vity 2
Educa onal
Ac vity 3
Educa onal
Ac vity 6
Educa onal
Ac vity 5
Educa onal
Ac vity 4
Pre-Assessments
Capstone
Ac vity
Gradua ng Seniors
Forma ve Assessments
Summa ve Assessments
Co-Operative Education
Extra-Curricular Activities
By Arvid Roach
Comments?
Questions?
Thank You!
Contact Information
Dr. Donald L. McEachron
3141 Chestnut Street
Philadelphia, PA 19104
Phone: 215-895-1382
Fax: 215-895-4983
Email: [email protected]