Transcript and the Next Accreditation System 10 year Self Study

Program Evaluation Committees
(PEC), Annual Evaluations, and
the 10 year Self Study and Visit in
the Next Accreditation System
Cuc Mai MD FACP
August 2014
Disclaimers
• No part of my salary is supported by ACGME.
• I am only a messenger and also a program
director.
• Information may change in the Next Best
Accreditation System.
Goals and Objectives
• Review the goals of the accreditation process in
the Next Accreditation System for Maintenance
of Accreditation
• Define current description of the Annual Program
Evaluation, the 10 year self study, and 10 year self
study visit
• Define the requirements for the Program
Evaluation Committee
• Describe some actions programs should consider
for the 10 year self study
The Old Days…
• Internal Reviews
• Program Information Form
• Duration between Site Visits based on Results
of Accreditation Visit
Maintenance of Accreditation in the Next
Accreditation System
Annual Program Evaluation (PR-V.C.)
 Resident performance
 Faculty development
 Graduate performance
 Program quality
 Documented improvement plan
SelfStudy
SelfStudy
visit
Ongoing Improvement
Yr 0
Yr 1
Yr 2
AE
AE
Focused assessments
Yr 7at anyYrtime
Yr 9
8 based
Yr possible
6
Yr 5
Yr 3
Yr 4
AEfrom AE.
AE
AE
AE
AEon data
AE
AE
Full site visit at any time if
data from AE eggregious.
AE: Annual Program Evaluation
Yr 10
AE
Aims of the Next Accreditation System
 Enhance the ability of the peer-review system to
prepare physicians for practice in the 21st
Century
 To accelerate the movement of the ACGME
toward accreditation on the basis of educational
outcomes
 Reduce the burden associated with the current
structure and process-based approach
© 2013 Accreditation Council for Graduate Medical Education
Information Current as of December 2, 2013
10 Self-Study and Self-Study Visit
 Self-Study
 Conducted by the
program
 Annual Program
Evaluation
 Review of program
goals and
improvement
efforts
 Self-Study Visit
 Conducted by ACGME
Field Staff members
Annual Data Review Elements
Policy 17.61 Review of Annual Data
 Continuous Data Collection/Review








ADS Annual Update
Resident Survey
Faculty Survey
Milestone data
Certification examination performance
Case Log data/Clinical experience
Hospital accreditation data
Faculty member and resident scholarly activity and
productivity
 Other: ACGME complaints, Institutional Quality Metrics
© 2013 Accreditation Council for Graduate Medical Education
Information Current as of December 2, 2013
The Next Accreditation System
Continuous
Observations
Assess
Program
Improvement(s)
Promote
Innovation
Program
Makes
Improvement(s)
Identify Opportunities
for Improvement
Next Accreditation System: Ten-Year Self-Study
and Self-Study Visit
SelfStudy
visit
Ongoing Improvement
Yr 0
Yr 1
Yr 2
Yr 3
Yr 4
Yr 5
Yr 6
Yr 7
Yr 8
Yr 9
Yr 10
AE
AE
AE
AE
AE
AE
AE
AE
AE
AE
AE: Annual Program Evaluation
Objectives of the Ten Year Self-Study Visit
• A comprehensive review of the program
– Using the Annual Program Evaluation and 10 year
self study
– Information on how the program creates an
effective learning and working environment
– How this leads to desired educational outcomes
Objectives of the Ten Year Self Study Visit
• Judge program success at Continuous Quality
Improvement
• Learn future goals of program
• Will verify compliance with CORE
requirements
• Core and subspecialty programs will be
reviewed together
Core and Subspecialty Programs
Reviewed Together
•
Needs of core residency taken into account
• When fellowships are started
• In decisions made regarding finite resources
• Coordination of curriculum and program resources
• Subspecialties can access to core resources
• Local oversight of fellowships
• Coordinated Self-Study of core & subspecialty programs:
• Assess common strengths and areas for improvement
• Action plans for areas for improvement
• Efficient Self-Study Visit
• Less time and resources spent, coordinated collection
and review of data
© 2014 Accreditation Council for Graduate Medical Education
Next Accreditation System: Ten-Year Self-Study
and Self-Study Visit
SelfStudy
Ongoing Improvement
Yr 0
Yr 1
Yr 2
Yr 3
Yr 4
Yr 5
Yr 6
Yr 7
Yr 8
Yr 9
Yr 10
AE
AE
AE
AE
AE
AE
AE
AE
AE
AE
AE: Annual Program Evaluation
Objectives of the 10 year Self Study
• Conducted one year before self study visit
• Assess ongoing compliance and improvement in
all dimensions, specifically on outcomes
• Consider program aims and
strengths/weakness/opportunities/threats
(SWOT) of program
• Uses annual program evaluations and ACGME
data to review program improvement activities,
successes achieved, and areas still in need of
improvement
Self-Study: Key Processes
• Data Gathering
• Annual Program Evaluations, ACGME Annual Data
• Focus on data gathering as a learning exercise
• Evaluate strengths and areas for improvement
• Explore opportunities and threats
• Reflect stakeholder (residents, faculty, and relevant
others) participation, input and perspective
• Offer evidence to support conclusions
• Data gathering begins in 2014 for programs with
Self-Study Visit scheduled for 2015
© 2014 Accreditation Council for Graduate Medical Education
The 10-year Self-Study: Timeline
Time prior to Self
Study Visit
11-12 months
ACGME Actions
1. Sends summary of
actions/follow-up
from Annual Data
Review
6-11 months
4 months
Program Actions
1. Aggregates data
from Annual Program
Evaluations
1. Conducts Self Study
1. Sets FINAL SelfStudy Visit Date and
informs program
10 days
© 2014 Accreditation Council for Graduate Medical Education
1. Completes ADS data
update
2. Uploads Self-Study
summary to ADS
New Areas: Program Aims
and Opportunities and Threats
• Program aims
• Evaluate effectiveness in meeting aims
• Assess relevant initiatives and outcomes achieved
• Should be realistic
• Opportunities and Threats
• Assess how factors and contexts external to the
program (eg, institutional, local, regional and
national) that affect the program
• Opportunities: Factors that favor the program, that
the program may take advantage of
• Threats: Factors that pose risks
© 2014 Accreditation Council for Graduate Medical Education
Defining Program Aims
• Set aims as part of self-identified annual
improvement process
• Who are our residents/fellows?
• What do we prepare them for?
• Fellowship
• Academic practice
• Leadership and other roles
• Who are the patients/populations we care for?
© 2014 Accreditation Council for Graduate Medical Education
Examples of Program Aims
• Provide a comprehensive 3 year curriculum to enable
residents to learn tertiary, secondary, and primary care
skills in all settings.
• Educate residents to be excellent practitioners of
medically directed anesthesiology in an Anesthesia care
team model.
• Train individuals with expertise in population health
and serving medically underserved.
• Produce excellent, independent practitioners who will
be local and national leaders, and for academic
careers.
© 2014 Accreditation Council for Graduate Medical Education
Enhancing the Definition of
“Areas for Improvement”
• Negative aspects that detract from the value of the
program or place it at a disadvantage.
• You need to address these areas to compete with
your best competitor.
• What factors within your control detract from the ability to
maintain a high-performing program?
• What areas need improvement to accomplish objectives
or enhance or supplement your existing strengths?
• What does your program lack (expertise in a certain
subspecialty, a type of technology, access to a particular
patient population, faculty with interest and skill in
research)?
• Is there a lack of some types of resources; is the
institution constrained in its capacity to provide support?
© 2014 Accreditation Council for Graduate Medical Education
DefiningOp
“ po rtunities”
• Opportunities are external attractive factors that, if
acted upon, will contribute to the program
flourishing.
• What are your capabilities for further evolving the
program; how can you capitalize on them?
• Has there been recent change in your immediate
context that that creates an opportunity for your
program?
• Are these opportunities ongoing, or is there a narrow
window for them? How critical is the timing?
© 2014 Accreditation Council for Graduate Medical Education
DefiningThr
“ e ats”
• Threats include external factors that affect the program.
• While the program cannot control them, beneficial to
have plans to address them if they occur.
• What factors beyond your control place your program
e
at risk?
What are changes in rs idents’
p
sec ialty choice, regulation, or
other factors that may affect the future success of your program?
• Are there challenges or unfavorable trends in your
immediate context that may affect your program?
• E.g. faculty burdened with heavy clinical load that prevents
effective teaching and mentorship
© 2014 Accreditation Council for Graduate Medical Education
Who Should Organize and
Conduct the Self-Study?
• Not defined
• Members of the Program Evaluation Committee?
• Natural extension of improvement process through the
Annual Program Evaluation
• PEC requirements1
• The PEC must be composed of at least 2 faculty
members and at least 1 resident(core);
• must have a written description of responsibilities(core);
1
ACGME Common Program Requirements, Effective July 2013
© 2014 Accreditation Council for Graduate Medical Education
Role of the PEC
• The PEC should participate actively in:
• planning, developing, implementing, and evaluating
educational activities;(detail)
• reviewing and making recommendations for revision of
competency-based curriculum goals and objectives;
• addressing areas of non-compliance with ACGME
standards;(detail)
• reviewing the program annually using evaluations of
faculty, residents, and others, as specified below(detail)
1
ACGME Common Program Requirements, Effective July 2013
© 2014 Accreditation Council for Graduate Medical Education
Role of the PEC (2)
• PEC prepares a written plan of action
• To document initiatives to improve performance
in the outlined areas ,
(core)
• To delineate how they will be measured and
monitored .
(core)
• The action plan should be reviewed and
approved by the teaching faculty and
documented in meeting minutes .
(detail)
1
ACGME Common Program Requirements, Effective July 2013
© 2014 Accreditation Council for Graduate Medical Education
Next Accreditation System: Ten-Year Self-Study
and Self-Study Visit
Annual Program Evaluation (PR-V.C.)
 Resident performance
 Faculty development
 Graduate performance
 Program quality
 Documented improvement plan
SelfStudy
visit
SelfStudy
Ongoing Improvement
Yr 0
Yr 1
Yr 2
Yr 3
Yr 4
Yr 5
Yr 6
Yr 7
Yr 8
Yr 9
Yr 10
AE
AE
AE
AE
AE
AE
AE
AE
AE
AE
AE: Annual Program Evaluation
Annual Program Evaluation
• Annual but may be more frequent
• Done by Program Evaluation Committee
• Document Review of Program Outcomes from
Annual Data
• Document Review of Program Achievement of
Aims, Program Strengths, Program Threats,
and Program Opportunities
• Develop Continuous Program Improvement
Projects and Track/Document Follow-up
Sample Improvement Plan
Area for
Improvement
Dissemination
of Goals and
Objectives
Issue(s)
Improvement Plan
Posted on
Educate residents
Intranet (5
and faculty
clicks to reach)
More prominent
Not
placement on Intranet
accessed
(1-click)
Not known
how this is
utilized by
residents and
faculty
Make accessible/
viewable in every
setting
Integrate with
resident formative
evaluations
© 2014 Accreditation Council for Graduate Medical Education
Group
Responsible
2 residents
and 1
faculty
member
(names) –
give them
credit for
work
Target
Completion
Date
Follow-up
June XXXX for
Quarterly
implementation
survey
at start of new
regarding
academic year
effectiveness
of new
approach
Spot check
Maintenance of Accreditation in the Next
Accreditation System
Annual Program Evaluation (PR-V.C.)
 Resident performance
 Faculty development
 Graduate performance
 Program quality
 Documented improvement plan
SelfStudy
SelfStudy
visit
Ongoing Improvement
Yr 0
Yr 1
Yr 2
AE
AE
Focused assessments
Yr 7at anyYrtime
Yr 9
8 based
Yr possible
6
Yr 5
Yr 3
Yr 4
AEfrom AE.
AE
AE
AE
AEon data
AE
AE
Full site visit at any time if
data from AE eggregious.
AE: Annual Program Evaluation
Yr 10
AE
Program Considerations for the 10
Year Self Study Visit
• How are you and your subspecialty programs
working together?
Program Considerations for the 10
year Self Study
• Do you have defined program aims?
• Have you ever analyzed your program with a
SWOT analysis?
Program Considerations for the Annual
Evaluation?
• Do you have a Program Evaluation Committee
and policy?
• Are you reviewing all the Program Outcomes
evaluated by the ACGME and looking for areas
of program improvements?
• Do you create, follow through on, and
document program improvement projects?
The Next Accreditation System
Continuous
Observations
Assess
Program
Improvement(s)
Promote
Innovation
Program
Makes
Improvement(s)
Identify Opportunities
for Improvement
ACGME Webinars and Other Resources

ACGME webinars are available at:
http://www.acgme.org/acgmeweb/tabid/431/ProgramandInstitutionalAccreditation/NextAc
creditationSystem/Webinars.aspx


 CLER
 Overview of Next Accreditation System
 Milestones, Evaluation, CCCs
 Specialty-Specific Webinars (Phase I)
 Phase I Coordinator Webinars (surgical and non-surgical)
 Specialty-specific Webinars (Phase II): Nov 2013-Dec 2013
Slide presentations for distribution to the GME community: NAS, CCC, Milestones,
Annual Program Evaluation/PEC, Updates on Policy– December 2013
Upcoming
 Specialty-specific Webinars (Phase II): Jan 2014-May 2014
 CLER
 Self-Study (what programs do)
 Self-Study Visit (what ACGME site visitors do)