Transcript The Function of Formative Evaluation

Resident Evaluation of a Web-based Integrative
Medicine Curriculum: The Function of
Formative Evaluation
Ben Kligler, M.D., Patricia Lebensohn, M.D.,
Sally Dodds, Ph.D., Raymond Teets, M.D., &
Victoria Maizes, M.D.
Presentation Objectives
The objectives of this presentation are to:
▫ Describe the function of formative evaluation for curriculum
development.
▫ Review the results from learners’ evaluation of the Integrative
Medicine in Residency curriculum.
▫ Describe the procedures used in applying feedback from learners’
evaluations to guide curriculum changes and revisions.
Integrative Medicine in Residency (IMR) is…
•
•
•
•
•
•
•
Competency-based, online, 200-hour, curriculum.
In-depth training in Integrative Medicine.
Incorporated through all 3 years of Family Medicine residency.
Piloted at 8 residencies nationwide.
Seamless, online evaluation of the curriculum and the residents.
Responds to ACGME competency requirements.
Evaluation developed simultaneously with the curriculum.
IMR Program Locations
Maine-Dartmouth
Maine Medical Center
Hennepin County
University of
Connecticut
Beth Israel
Carolinas Medical Center
University of Arizona
University of Texas
Medical Branch
IMR Curriculum Units
Prevention
and
Wellness
Acute
Care
Tools in
Integrative
Medicine
Pediatrics
Chronic
Illness
Women’s
Health
Units and Courses
Introduction to Integrative Medicine
Pediatric Topics:
Prevention and Wellness:
•ADD/ADHD
•Chronic Pain Syndrome
•Asthma and Allergies
•U.S. Preventive Health Services
•Nutrition and Diet
•Supplements for Prevention
•Physical Activity
•Sleep
•Stress and Mind-Body Medicine
•Spirituality
•Clinical Integration
Tools in Integrative Medicine:
•Integrative Medicine Intake and Care Plan
•Botanicals
•Mind-Body Medicine
•Manual Medicine
•Introduction to Energy Medicine and Whole Systems
•Practice Management
•Motivational Interviewing for Behavioral Change
Acute Care:
•Acute Back Pain,
•Urinary Tract Infection,
•Gastroenteritis,
•Otitis Media,
•Vaginitis,
•Atypical Chest Pain,
•Upper Respiratory Infection
Women’s Health Topics:
• PMS/PMDD
• Dysmenorrhea
• Menopause
• Fibromyalgia
• Osteoporosis
•Depression
•Eating Disorders
•Pregnancy and Lactation
Chronic Illness:
•Cardiovascular Disease
•Type II Diabetes
•Osteoarthritis
•Rheumatoid Arthritis
•Obesity
•Irritable Bowel Syndrome
•Chronic Back Pain
Special Topics:
•HIV/AIDS
•Cancer Survivorship
•Environmental Medicine
IMR: Educational Methods
 Needs assessment informed curriculum design.
 Web-based curriculum written and edited by Integrative
Medicine educators.
 Competencies aligned with the ACGME Outcomes Project.
 Flexible modular format to meet the needs of residency
schedules.
 Case-based, interactive learning and streaming video.
 Experiential exercises and process-oriented group
activities at the residency sites.
 A community of learners through online dialogues with
faculty and peers.
 Emphasis on teaching and promoting physician wellbeing and self-care.
IMR User Interface
Reflections
Flexible modular format
Direct Observation Checklists
Online portfolios
Resources and Links
What is Formative Evaluation?
“When the cook tastes the soup, that’s
formative; when the guests taste the soup,
that’s summative.” - Robert Stakes
Formative Evaluation…
• Helps form and strengthen programs by:
▫ Examining the need for them,
▫ Their delivery or technology
▫ The quality of their implementation
▫ Assessing their organizational contexts, procedures, and resources
(Scriven, 1991).
• Methods emphasize data collection and analysis prior to
completion.
Why Use Formative Evaluation in the IMR?
• Medical knowledge in Integrative Medicine is dynamic, evolving
rapidly; revisions are largely driven by new information.
• Distributed, web-based curricula requires feedback from learners to
be successful.
• In the IMR, formative methods include:
1. Preliminary needs assessment
2. Feedback from resident evaluation of courses
1. IMR Needs Assessment
(Benn, Maizes, Guerrera, Sierpina, Cook, & Lebensohn, 2009)
• Methods
▫ 222 faculty and residents from 8 family medicine programs
(60.2% response rate).
▫ Online survey w/ structured and open-ended questions.
• Results
▫ Preferred IM be woven throughout all curriculum areas (67%).
▫ Top topics: Nutrition, Supplements, and Physician Wellness.
▫ Top curricular areas for IM enhancement: Chronic illness,
behavioral health, and outpatient medicine
▫ Viewed IM central to family medicine training, patient care, and
the field of family medicine (84%).
▫ Top challenges: Limitations in time, resources, and acceptance.
2. IMR Course Evaluations
• Measures assess learners’ ratings of the course in:
▫ Meeting course objectives
▫ Clinical utility of the course
▫ Time needed to complete the course
▫ Functionality of the online technology
 Analyzed when 50% of pilot residents complete a course.
▫ Ratings < 8o% and open-ended comments targeted for review.
 Review of 01 pilot group suggested Likert-type response categories be
changed from 4 to 5 points to increase precision and variability.
Resident Demographics
2011 (n = 67)
2012 (n = 64)
Controls (n = 31)
64%
36%
64%
36%
45%
55%
32
(27 – 43)
32
(24 – 56)
30
(26 – 38)
Marital Status
Married
Live w/ partner
Single
Unk.
45%
13%
27%
15%
27%
6%
34%
33%
58%
10%
32%
0%
Graduate Status
US MD
US DO
FMG
Unk.
51%
18%
27%
4%
41%
26%
28%
5%
55%
16%
16%
13%
Sex
Female
Male
Mean Age
range
Did Course Meet Learning Objectives?
Year 01 Content
Very definitely/Definitely
2011
92%92%
94%94%
96%
94%
2012
95%
91% 91%
82%
*Currently, data are available for 4 courses for the 2012 class.
95%
94%
89%
Clinical Utility of the Course?
Year 01 Content
Very Useful/Useful
2011
95%96% 93%97% 99%
78%74%
2012
96%
86% 89%
93%
91%
47%
*Currently, indicators are available for 4 courses for the 2012 class.
How Smooth was the Online Technology?
Year 01 Content
Very Smoothly/Smoothly
2011
78%82%
91%
95%94% 97%
83%
2012
94%94%
90%
*Currently, indicators are available for 4 courses for the 2012 class.
93%
81%
85%
Year 02 Course Evaluations
Courses currently at 50% threshold
Meet Learning Objectives?
Clinical Utility?
Very Useful/Useful
Very Definitely/Definitely
97%
98%
100%
100%
100%
98%
100%
94%
Year 02 Course Evaluations
Courses currently at 50% threshold
Smooth Online technology?
Very smooth/Smooth
100%
92%
82%
77%
Length of Time Spent
Year 01 &Year 02 Content
Class
Year
1
2011
Course
Introduction
2012
2011
Supplements
2012
2011
Nutrition & Diet
2012
2011
Physical Activity
2012
Year
2
< 1 hr
1-2 hr
2-3 hr
3-4 hr
4-5 hr
34%
44%
13%
5%
4%
34%
36%
26%
4%
32%
35%
12%
12%
50%
36%
9%
13%
25%
31%
21%
42%
29%
61%
38%
2%
66%
28%
7%
2011
Stress & Mind Body
34%
43%
17%
2011
Spirituality
72%
26%
2%
2011
Sleep & Health
94%
6%
2011
Preventive Services
59%
32%
7%
2011
Clinical Integration
17%
38%
2011
Allergy/Asthma
11%
43%
2011
ADHD
48%
48%
2011
Pediatric Pain
55%
27%
14%
2011
PMD/PMDD
53%
40%
7%
5%
5-6 hr
3%
5%
16%
6%
7%
4%
4%
3%
2%
21%
8%
9%
8%
27%
11%
3%
5%
4%
4%
What Residents Say…
I enjoyed the interactive nature of the
modules.
The tools are amazing!
Did an excellent job of relating spirituality
to health care.
I think sleep is one thing most neglected in
health care topics. This course has rightly
identified it.
The ability to self reflect as well as think
about our patients.
I loved the video lecture – very engaging
and interesting.
This is a very comprehensive topic.
Great links and resources.
The seven cases. Great way to pull all of
these courses together!
The buttons aren’t working.
There was a lot of data in this module
[Nutrition and Diet], but not enough
interactive learning.
Too lengthy and too in-depth.
Some links were not available.
Too many resources were given and I did
not have enough time to thoroughly
review them.
Too much reading, too many links, and too
many videos.
Discussion
Acknowledgements
University of Arizona
Pilot Sites
Emily Sherbrooke
Victor Sierpina, M.D.
Paula Cook
Selma Sroka, M.D.
Rhonda Hallquist
Mary Guerrera, M.D.
Tieraona Low Dog, M.D.
Dael Waxman, M.D.
Craig Schneider, M.D.
John Woytowicz, M.D.