Transcript Two Problems with Most Pedagogical Reform Initiatives

Improving Undergraduate Education
through the Assessment of Student Learning
SACS-COC Institute on Quality Enhancement and
Accreditation
July 30, 2006
Peter T. Ewell
National Center for Higher Education Management Systems
Looking Back: Origins of the Assessment
Movement Two Decades Ago
• Undergraduate Reform Reports of 1985-86
• Internal Stimulus: Call for More Coherent
Teaching/Learning Approaches and Information
for Improvement
• External Stimulus: Stakeholder Demands for
Information on “Return on Investment”
• Tensions in Motive and Message Ever Since
Why Didn’t Assessment Go Away?
• Pressure to Produce Evidence of Student Learning
Outcomes Never Let Up
• By Early 1990s, Accreditors Replace States as
Primary External Stimulus to Get Started
• Intermittent Federal Interest in Assessment as an
Element of National Accountability
• But Resulting Faculty Ambivalence About a Process
Seen as “External” and “Administrative”
Looking Back: What’s Been Accomplished?
• Assessment Is for the Most Part Perceived as
Inevitable and Legitimate
• Vast Majority of Institutions Have Statements of
Learning Outcomes (General and Programmatic)
• A “Semi-Profession” of Folks Involved in
Assessment
• Steadily Growing Sophistication with Respect to
Methods of Gathering Evidence
Looking Back: What Hasn’t Happened?
• Authentic Integration of Assessment into Faculty
Cultures and Behaviors
• Assessment Activities Still Largely “Added On” to
the Curriculum Instead of Being Embedded In It
• Systematic and Widespread Use of Assessment
Results for Institutional and Curricular Improvement
• Proactive and Sincere Institutional Engagement with
Accrediting Organizations Around Topics of
Assessment
Assessment as a “Perpetual Movement?”
• Most Social Movements…
– Fade as Fads, or
– Go Away Because Core Ideas are Mainstreamed
• The “Assessment Movement” Has Done Neither...
– External Requirements Keep it Alive
– But External Pressures also Constrain Faculty Buy-In
and Meaningful Institutional Use
• Moving Beyond a “Perpetual Movement” Will Demand...
– Making Assessment Real to Faculty by Connecting It
to the Actual Practice of Teaching and Learning
– Re-Focusing Accountability on the Authentic Student
Abilities that Society Says it Needs
Some Prominent Changes in Higher Education’s
Operating Environment
• Doing More with Less
- Need for Curricula that are Effective and Efficient
- Need Information About Curricular Functioning to Enable
Effective Action
• Changes in Instructional Delivery
- Competencies and “Deep Learning”
- Student Engagement and Role of Technology
• New Views of Accountability
- “Quality”— From Inputs/Processes to Results
- Stakeholder Voices — Students and Employers
Challenges to the Academy:
The Internal Dimension
• Changing Paradigm of Teaching and Learning
• Resulting Changes in Academic Roles,
Behaviors and Structures
– Nature and Role of “Faculty” in Instruction
– Patterns of Student Enrollment
• [Technology as a “Wild Card”]
A Changing Paradigm of
Teaching and Learning
• From “Faculty Teaching” to “Students
Learning”
• Students “Make Their Own Paths” through
Multiple Learning Opportunities
• Explicit Designs for Learning Based on
Research
Changing Paradigm:
Possible Lines of Response
• Demonstrated Achievement Becomes
Paramount, not “Seat Time”
• Students “Teach” One Another
• Individual (and Asynchronous) Paths
• Technology Seen as Opportunity to ReThink [not as a “Solution”]
Some Implications for Student
Assessment Processes
• Assessments Reinforce Common Standards
for Learning Across Curricula and Classes
• “Seamless” Assessments Become an
Integral Part of Curriculum and Pedagogy
• Assessments Emphasize Connections and
Longitudinal Development, not Just
Attainment
Assessment Approaches:
A Resulting Shift in Emphasis
• Accountability-Based: Assessments Added
onto Instruction to “Check Up” on the
System in the Aggregate
• Scholarship and Improvement:
Assessments Built into the System to
Simultaneously Assure Standards and
Provide Feedback on Collective
Performance
Kinds of Information Needed
• Alignment of Key Learning Outcomes Across
Units, Sequences, and Courses
• Match Between Curricular Design, Delivery,
and Student Experience
• Match Between Instruction and Needs of
Diverse “Student Bodies”
• Effectiveness of Particular Innovations and
Interventions
Changing Nature of the Faculty Role
• Dis-aggregation of Instructional Roles
• More Things for Faculty to Do
• New Potential Career Patterns and Paths
New Faculty Roles:
Possible Lines of Response
• Emphasize Peer Support and Collaboration in
Faculty Development
• Recognize and Regularize Alternative Career
Paths
• Recognize “Mentorship” as the One Thing
You Can’t Responsibly Outsource
Kinds of Information Needed
• Capturing and Re-Aggregating Data About
Discrete Instructional Functions
• Accounting Technology-Based Costs
• Tracking Faculty “Assets”
Changing Patterns of Student Attendance
• Increased Levels of Multi-Institutional
Attendance
• Increased Complexity in Course-Taking
Behavior within Institutions
• Greater (and Unpredictable) Time Lapses
Between Instructional Encounters
Changing Patterns of Attendance:
Possible Lines of Response
• Coherence Based on Common Practices [and
Outcomes], not Common Content
• Stress Ways for Students to Exploit and
Reflect on Their Own Experiences
• Establish Clear Transition Points at which to
Assess Student Mastery of Key Concepts
Kinds of Information Needed
• Relationships Between Particular Institutional
Experiences and Particular Outcomes
• Tracking Student Learning Styles and
Individual Paths of Development
• “Episode-based” [as opposed to time-based]
Data Structures
Challenges to the Academy:
The External Dimension
• Increased Accountability
• Changing Expectations Regarding What
Students Know and Can Do
• The “New Competition”
What Are States Doing?
• Forces Influencing State Approaches
– Decreased Agency Capacity Due to Funding Cuts
– Momentum of K-12 Reform (NCLB)
– Political Uncertainty and Instability
• Types of State Policy Responses
– P-16 Alignment
– Performance Measures and “Report Cards”
– Achievement Testing (Driven by K-12)
What’s Shaping Accreditation?
• Forces Influencing Accreditors
– Pressure for Specific Performance from Federal
Government
– Demands from Institutions to “Add Value”
– New Models from Other Sectors and Abroad
• “New Looks” in Accreditation
–
–
–
–
Focus on Outcomes and Effectiveness
Presenting Evidence [e.g. “Portfolios”]
Review Approaches [e.g. “Academic Audits”]
Connection to Institutional Planning [e.g. QEP]
Learning Outcomes:
What Employers Expect
• Higher-Order “Literacies” as Well as Specific
Skills
• Framed in Terms of “Practice” [not
“Knowledge”]
• As Much About Attitudes as Academics
[“Soft Skills”]
External Forces:
Some Resulting Influences on the Academy
• Increased Emphasis on Credentialing
• Modularity and Acceleration to Increase
Accessibility
• Pressure to Respond to “Students as
Customers”
• Accountability “Superstructures” that
Divert Attention and Information Resources
Kinds of Information Needed
• Outcomes and Performance Measures
• Data on What Experiences/Services Students
Can Expect
• Peer Comparisons [Increasingly Outside the
Academy] and Comparative Performance
• Needs and Satisfaction of External Stakeholders
Attributes of a Meaningful
“Culture of Evidence”
• Shared Recognition That Many (But Not All)
Things Are Knowable
• An Accessible Store of Information About
Organizational Condition and Performance
• An Attitude Toward Problem-Solving that
Minimizes “Finger-Pointing”
• Clear Follow-Through On Decisions Made
and Why They Were Taken
Cultures of Evidence: Success Factors
• Visible Metaphor of Scholarship
• Beginning with Real Problems and Processes,
not with “Method”
• Consistent Messages from Leadership
• Periodically “Re-Socializing” the Community
• “Closing the Loop” with Action
Cultures of Evidence: Inhibiting Factors
• Either Excessive or Non-Existent
Consequences
• Alien Language and “Management Culture”
• Excessive Complexity
• Burnout and “Committee Fatigue”
Meaningful Assessment is More About
Mindset than Method
• Questions About Learning are not Just
Matters of Opinion
– What Information Might We Collect?
– What Might We Expect to Find?
– What Difference Would Finding Out Make?
• Assessment is About Improving Practice
– What are We Trying to Fix?
– How Good is Good Enough?
– What Changes are Implied?
The Bottom Line
• For Internal Management, “Seat of the Pants”
Decision-making is no Longer Sufficient
– Information Used Openly, Consistently, and Continuously to
Inform Academic Decisions
• For External Constituencies, “Trust Me” is no Longer
Sufficient
– Need Clear, Understandable Evidence of Student Academic
Attainment
• Be Vigilant about the Information You Choose and the
Signals it Sends
– Make Sure that What You Measure is What You Value
– Harness the Accreditation Process to Make it Happen!