MEASURING THE IMPACT OF A PEDAGOGICAL CHANGE OR INNOVATION S T R A T E G I E S F O R D I S S E M I N A T I O N Anastassis Kozanitis Educational consultant Polytechnique Montréal

DO YOU KNOW ANYONE IN THIS ROOM WHO HAS INNOVATED?

• • • 1- no 2- yes 3- I’m not sure

HAVE YOU INNOVATED IN THE PAST 2 YEARS?

• • • • 1- no 2- yes 3- I’m not sure 4- I would of liked to, but I lack the ressources

WHY DID YOU (NOT) INNOVATE OR WHY DO YOU (DON’T) WANT TO INNOVATE?

• Activity 1: Think-pair-share

SUPPORTING ELEMENTS? INSTITUTIONAL OR DEPARTMENTAL LEVELS

• • • • • • • • Pedagogical culture Pedagogical training for teachers Educational support center Valuing teaching (support of the institution or department) Tangible recognitions (awards, $, career progress) Educational leadership of the authorities (Dean, Head of department, etc.) Support (human, material and financial resources) Work collaboratively within a program approach

ELEMENTS THAT INHIBIT OR PUSH BACK

• • • • • Doubt that it will work. Negative reaction from students.

Sensing that there was no added value. Material, financial or contextual constraints (e.g. group size), lack of time or work load too large.

Unsuccessful blind trial and error.

SUPPORTING ELEMENTS? INDIVIDUAL LEVEL

• • • Strength of your personal beliefs Desire to make your work more interesting Professionalization of teaching (3 levels of intensity) • • • 1) Reflective practice (Schön, 1983) 2) Scholarly teaching (Richlin, 2001) 3) Scholarship of teaching and learning (Potter & Kustra, 2011)

OUTLINE AND GOALS

• • • • Choose what elements to measure as a result of a pedagogical innovation implementation Identify how to measure and analyze its impact on students Develop strategies to disseminate the findings and motivate colleagues to do the same. Commit to practice a scholarly teaching

CIRCLE OF INNOVATION

Inicial situation Innovation Effects or results

PEDAGOGICAL INNOVATION DEFINITION:

• A deliberate activity that aims to introduce novelty in a given context, and seeks to substantially improve the learning of students in a situation of interaction or interactivity (Béchard & Pelletier, 2002)

• • •

INITIATIVES FOR INNOVATIONS WITH OR WITHOUT ICT

Institutional

• Changes in software platform(Moodle) • Curricular changes (competency-based curriculum)

Departmental

• Curricular changes(program-approach) • Structural changes (course fusion)

Individual

• Changes to active learning methods

Limited time (minutes) Small scale activities

Puzzle (Jigsaw) Role playing Discussions, debates

Limited or prolonged time (minutes-weeks)

Peer instruction Case study Webquest

Prolonged time (weeks or months) large scale act.

Problem/project-based learning Blended learning, serious games Seminary

THE ACTIVE LEARNING CONTINUUM*

Interactive lectures Instructor centered Limited time or small scale activities Prolonged time or large scale activities Student centered Think-pair-share Role playing Case study

*Adapted from Michael Prince (2011). Active cooperative learning. Bucknell University. http://www.asee.org/documents/conferences/annual/2011/plenary-michael-prince.pdf

Project-based learning

TEACHING METHODS

Instructor centered Student centered

• • • • Lecturing Podcasting Video-conferencing Demonstration • • • • • • • • • • • Problem-based learning Guided discovery Case study Project-based learning Role playing Simulation Tournament Peer instruction Team-based learning Laboratory Discussions

IS THERE SOMETHING YOU WISH TO CHANGE IN YOUR TEACHING?

• • What problems have you observed concerning your students’ learning?

What solutions have you tried and what were the results?

• Example: implementing a new method • Problem-based learning • Project-based learning • • • Case studies Team-based learning Other?

RELEVANT PRE-ACTIVE QUESTIONS

• • • • • • • How can I fix my mistakes from last semester? How can I get my students’ attention? What learning activities should I provide to help students understand the subject? How can I arouse their interest and motivation? What content-specific resources do I need?

How will I consider student heterogeneity and their different ways of learning? What are my expectations about student productions?

ACTIVITY 2 CURRENT SITUATION – DESIRED SITUATION

• • Step 1.

Identify, in your practice, a specific situation you want to improve or a recurring problem you wish to solve.

• • • Step 2.

Using one or two words, post your situation or problem on a piece of paper for everyone to see.

For example: • • • • Classroom participation Motivation Critical thinking Student attitude

SCHOLARSHIP OF TEACHING AND LEARNING (SOTL)

• • Definition A careful analysis from the teacher about students’ learning, before and/or after a systematic and intentional change of pedagogy, with an evaluation of the results of this change, and dissemination of these results (Potter y Kustra, 2011).

STEPS TO SOTL

• 1 – write a research question 2 - design the study 3 - collect data 4 - analyze the data 5 - present and publish the results

SPECIFIC ASPECTS TO BE MEASURED

• For example, if a teacher says he has problems with teaching large groups, you may look at the following indicators : • • • • • • Absenteeism rate Verbal or non-verbal behavior of students in the classroom Lack of student participation during activities Grades on tests and assignments Results of teaching evaluations Student comments

ACTIVITY 3 CHOOSING THE SPECIFIC ASPECTS TO MEASURE OR ASSESS

• • • Identify the specific aspects of the general theme you want to measure.

You have to be very accurate. For example, it is not enough to say I want to measure motivation. But rather what aspect of motivation (interest in the course or learning activity, perception of usefulness of what they are learning, amount of time or effort invested in the course, the perception of self-efficacy, etc..).

You can measure multiple aspects, however make sure that the information you seek is: available, valid, accurate, and true.

ACTIVITY 4 IDENTIFY HOW TO MEASURE OR ASSESS THE SPECIFIC ASPECTS

• • • It is the procedural or operational part What instrument or process do you need in order to measure the specific aspects 2 options: 1) You can choose and adapt an existing instrument 2) You can build or create an instrument • In both cases you must validate the instrument • How will you do the validation? • Who are your sources?

• With how many participants are going to validate the instrument?

METHODS OR TOOLS FOR MEASURING THE IMPACT OF THE PEDAGOGICAL CHANGE

• Quantitative methods • Qualitative methods • • • • • • Quasi-experimental design Likert type questionnaire Existing tools like the NSSE Comparing means (ANOVA, t-test) Regressions SEM o Path analysis • • • • • Case study Semi-structured interviews (NVivo) Focus group Quality circle In situ observations using a form

EXAMPLES OF METHODS

• • • • • • A valid questionnaire on motivation; Some open-ended questions regarding opinions; A test or an examination of knowledge or skills; An observation form of classroom behavior; Self-assessments, peer-assessments, student evaluations; Etc.

DISSEMINATION

• • • • How to get your colleagues involved?

How to convince them to change?

How can you teach them what you have learned?

What are the usual arguments for inaction?

DISSEMINATION

cognitive affective

FACTORS RELATED WITH PEDAGOGICAL INNOVATION DEVELOPMENT (BÉCHARD & PELLETIER, 2002) •

4 categories :

• •

External environment Institution

• •

Department Classroom

External environment Actors

-government -universities -professional associations -civil society -companies

Social context Institution

ESTRATEGIAS

-innovator types -T-L conception -motivations -source of inspiration

Construction actors

-categories of collaborators -possible roles

Institutional Department Departmental Classroom Teaching actors

-epistemic orientations -relation towards innovation -conception teaching & learning

Implementation

-accountability -technological transformations -pedagogical transformations

External strategies

-financial, human, and material resources -programs -type of management

Professional development strategies

-legislation -financing -competition -accreditation -cooperation -pedagogical training -financing -awards -accompanying -load relief -structure -atmosphere -leadership of the head of department

Characteristics of innovations

-experimental site -students’ ability

Nature of learning

-goals of innovation -nature of innovation -ICT -desired clientele -declarative and procedural knowledge -transversal competencies -technological literacy

OPPORTUNITIES AND THREATS

External environment

-Increasing pressure to offer distance learning (Mooc) -budget reduction -national reform -funding from private foundations

Institution

-scarcity of resources -increase in enrollment -monetary incentives

Department

-cost of equipment

Classroom

-students need to be more active

STRATEGIES FOR DISSEMINATION

• • • • • • Organizing an event for presentation of best practices (institutional or department level) Formal and informal pedagogical training Mentoring – coaching Teamwork with a colleague giving the same course Participating in educational conferences (STLHE, AERA, CHEP, Lilly conference, ISSOTL, etc.) Funding pedagogical initiatives (specific criteria)

WHAT CAN LEAD TO INNOVATIONS?

1 - Personal, informal initiatives 2 - Evaluation of teaching 3 – Work in collegiality to prepare assessment activities 4 - Work in collegially to prepare course material 5 - Promotion of educational initiatives (funds, special events e.g. Teaching day) 6 - Scholarship of teaching and learning (SoTL) 7 - Training and pedagogical support for teachers 8 - Paradigm shift (from teaching to learning) 9 - International collaboration projects

ACTIVITY 7

• • Share with a neighbor what other strategies are adequate given your context.

Think of any resistance you anticipate.

CONCLUSION

• • • Degree of satisfaction with the work accomplished?

Degree of satisfaction with the help?

Any questions remain?

REFERENCES

• • • • • • • • • Béchar, J-P. & Pelletier, P. (2002). Dynamique des innovations pédagogiques en enseignement supérieur: à la recherche d’un cadre théorique. Available online at : http://neumann.hec.ca/oipg/fichiers/2002-002_ _Dynamique_des_innovations_pedagogiques_en_enseignement_superieur.pdf

Cornet, J. & Voz, G. (2010). «Comment former de futurs étudiants réflexif?» Education and formation, [En ligne], octobre, e-294, http://ute3.umh.ac.be/revues/include/download.php?idRevue=10&idRes=81 . Hannan, A., English, S. & Silver, H. (1999). Why innovate? Some preliminary findings from a research project on “Innovations in teaching and learning in higher education”. Studies in Higher Education, 24(3), 279-289.

Kapur, M. (2012). Productive failure. Available online at: http://www.isls.org/icls2012/downloads/K2Kapur.pdf

Lenoir, Y. (2012). Analyse réflexive : un outil de questionnement sur la gestion des activités d’enseignement-apprentissage, Outil 2. Vivre le primaire, 25(4). O’Brien, M. (2008). Navigating the SoTL Landscape: A Compass, Map and Some Tools for Getting Started. International Journal for the Scholarship of Teaching and Learning, 2(2), 1-20.

Perrenoud, P. (2001). Développer la pratique réflexive dans le métier d'enseignant: professionnalisation et raison pédagogique: ESF éditeur.

Potter, M. K., & Kustra, E. (2011). The relationship between scholarly teaching and SoTL: Models, distinctions, and clarifications. International Journal for the Scholarship of Teaching and Learning.

Schön, D.A. (1983). The Reflective Practitioner. San Francisco: Jossey-Bass.