Site Visit 102: DIO*s Role for Program Site Visits

Transcript Site Visit 102: DIO*s Role for Program Site Visits

Site Visit 102: The DIO’s Role for Program Site Visits

Ann M. Dohn, M.A.

DIO Stanford

Disclaimer

 No Conflicts of Interest to Report

DIO Competencies

  Maintaining the Institution’s Residency Program’s ACGME accreditation Improving the Institution’s education program  Developing and supporting residency program directors

DIO Vision

 “Your job is not a powerful one…your job is to empower others.” i.e. program directors – Helen Rantz, 1984

Session Objectives

 To understand the role of the DIO in ACGME program site visits – Using ACGME annual resident and faculty surveys – Using internal reviews and annual program evaluation in preparation for site visits Using other potential measures of program quality

Session Objectives

 A time of change… – Living in two worlds-starting 7/1/2013  “PIF-less Site visits / NAS  Site visits with PIFs ACGME OLD NEW

Then and Now

 1984 site visits vs. 2012 site visits

Start with the New and “unknown”

 Unknown is scary

New Model: What Do We Know?

 Longer Cycles  Annual WebADS Updates 

ACGME Resident and Faculty Surveys

 End of the PIF

End of the PIF

 DIOs will have more recreational reading time!

“New” Model

 Effective 7/1/2013 – “Early adaptors”= RRCs deciding to go “PIF-less”  Emergency Medicine  Internal Medicine  Neurosurgery  Orthopedic Surgery  Pediatrics  Radiology (Diagnostic)  Urology – and their subs

New Accreditation System (NAS)

 The NAS requires categorization of the ACGME’s Common Program Requirements to clearly identify: –

Core Requirements

–

Detail Requirements

–

Outcome Requirements

New Accreditation System (NAS)



Core Requirements

– Statements that define structure, resource, or process elements essential to every graduate medical educational program. 

Duty Hours



Sponsoring Institution’s responsibility for Program Accreditation



Change in program director



Qualifications of the program director

New Accreditation System (NAS)



Detail Requirements:

– Statements that describe a specific structure, resource, or process for achieving compliance with a Core Requirement.

Examples of “Detail” Requirements

 Detail Requirements include particular educational approaches and learning experiences  a given number of lectures or hours of bedside teaching experiences  Ensuring compliance with grievance and due process procedures as set forth in the Institutional Requirements and implemented by the sponsoring institution

New Accreditation System (NAS)



Outcome Requirements:

– Statements that specify expected measurable or observable attributes (knowledge, abilities, skills, or attitudes) of residents or fellows at key stages of their graduate medical education.

– Programs in substantial compliance with the Outcome Requirements may utilize alternative or innovative approaches to meet Core Requirements.

Example of “Outcome” Requirements

 Outcome requirements include: – Milestones – Residents must be able to competently perform all medical, diagnostic, and surgical procedures considered essential for the area of practice.

Common Program Requirements

Neurosurgery – new Accreditation Example…

New Model-Tracer Model Inverted Model

 Increases value on resident input – ACGME survey – Resident interviews  Focuses on PROGRAM rather than a description of the program –

The Tolstoy novel will no longer be written!!

New Model-PIF Questions

 Small number of questions from PIF now in annual WebADS update  Remaining information assessed by the field staff

Tracer Method

 Allows for the selection of particular elements to be evaluated: – – Citations Complaints from residents/faculty – – Non-compliance reported on the ACGME resident survey Duty hour and learning environment standards

Tracer Method

 Focuses on: – Program’s actual operations – Implementation of policies and procedures

Tracer Method – Interactive – A Key Change

 Allows for discussion of important topics related to the new duty hour standards  Provides opportunities for site visitors to educate program leaders and share innovative practices

Tracer Method

 Longitudinal information on: – New standards – – Prior citations/corrective actions Resident survey identified areas of non-compliance  Areas identified during preparation for the Site Visit

Tracer Method

 Will focus on relevant high priority issues for the given program  Planned for two-four areas/topics per program – Done during the regular interview (no walkarounds)

Tracer Method

 Enhancing resident input – Residents to directly respond to the site visitor with a single, confidential “consensus list”    strengths areas for improvements starts with the trainees

PIF-Less Site Visit: The Stanford Experience

 August 2011  Three Programs  Visit done in lieu of mid-cycle internal review  Data not used for accreditation  Focus was on program requirements  Used comprehensive data system (no paper)

PIF-Less Site Visit: The Stanford Experience (2)

 Individuals interviewed = regular site visit  ACGME provided comprehensive reports  Extensive debriefing with the site visitors and program participants  Very positive experience for PDs. PCs, and the DIO and GME staff

PIF-Less Site Visit: Stanford Findings

 PIF-Less (inverted) site visit report finding – Comparable to standard internal review – Note: our protocol always starts with the residents/fellows

PIF-Less Site Visit: ACGME Findings

 Findings: – Residents feel more engaged – – PDs love it Early ID of Program Themes – – Improves continuity and integration of data Seamless process for follow-up – Increased opportunity for reconciliation of discrepant issues – Transformative Model - rather than additive

PIF-Less Site Visit: ACGME Findings (2)

Findings Continued  Comparable to visit with PIF for citation review  Superior for review of program improvements and positive program elements  Variability with other elements – Dependent on data system used by the institution/program

Upcoming Changes in the Site Visit

 Broader implementation of the inverted Site Visit – Team site visits (two site visitors)  Programs  Institutions

Yikes! ACGME is coming….

ACGME Use of Survey Data during the Site Visit

 Use resident data to validate/clarify program information  Compare resident data to program director and faculty interviews  Validate resident data to program documentation  Compare duty hours responses to institutional data

DIO Use of Data for Program Site Visits

 ACGME resident and faculty surveys  APRs  Internal Reviews  Other measures of program quality

Annual ACGME Surveys 2012

Annual ACGME Surveys 2009

ACGME Concerns with the Resident Survey

 ACGME concerns with the resident survey: – “Honesty” in reporting – ? Residents being coached ?

– Amount of time spent by residents completing the survey = 20 seconds per question – – Survey questions being sold to companies who want to ‘help’ programs prepare Survey fatigue

ACGME Annual Resident Survey

 Possible solutions to using the survey – 2012- Aggregation of individual survey questions into groupings – 2012-use of alternate forms of the survey – Educating residents about the vital nature of the survey

Annual Program Reviews: Summary

Annual Program Reviews: Data Elements

Internal Reviews

Monitoring Internal Reviews

Other Possible Sources of Data

 GME Annual House Staff Survey  Duty Hours Reports  Evaluations of programs, faculty, and house staff  Alumni surveys  Board and in-training exam scores

GME Annual House Staff Survey

 Provides feedback on eight (8) key areas of Graduate Medical Education: – – Overall Experience Program Curriculum – Program Faculty – Evaluation and Feedback – Training Environment – Personal Wellness – Quality Improvements – – Communication and Patient Perceptions Other areas….(anything we missed)

GME Annual House Staff Survey

 Provides a level of detail not available from the ACGME survey – Such as on call room issues, fear/intimidation issues, harassment issues, faculty – Allows for text answers, comments Warning: Don’t do a survey unless you are willing to accept and act on the input

GME Annual House Staff Survey

Duty Hour Reports

DIO’s Role

 Continuous monitoring of all programs – Look at all of the data – Take to GMEC

Tools

 ACGME surveys  GME house staff survey  APRs  Duty hour reports  ACGME letters  Internal reviews  Institutional report card

“The Report Card”

Site Visit (or Self Study) Preparation

 Ongoing   Best done continuously Recommend not “last minute”   Takes time and resources Needs your best “detail oriented” staff

Where to start?

Internal Scheduling/Tracking

Plan ahead

 Look at the ACGME schedule  Monitor annually APRs  Monitor annually program policies  Use the ACGME survey data!!

 Use resident/program director feedback  Update PIFs annually/monitor annual updates

Truly A Juggling Act!

Teach….Teach…

 Monthly program director & coordinator sessions  Reach out to programs  Assign an education specialist

Faculty development for program directors (& coordinators)

 Monthly 1 hour meetings Site visit preparation PIF writing

GME: Faculty Development - AY 2011 2012

Faculty Development

 Workshops – Fundamentals of curriculum writing – – Evaluation Policies – Feedback

How We Teach

 Preparing for a site visit

Accreditation & Maintenance of Program Quality: A Continuous Process

Preparing for an ACGME Site Visit For Program Directors, Assistant Directors and Coordinators

TIMELINE: Preparing for an ACGME Site Visit

 Receive ACGME site visit letter 100-110 days prior to visit  APR Annually  Annual ACGME Resident Survey Jan to May  Annual GME House Staff Survey March to May  Mid-Cycle Internal Review – Response to Mid-Cycle Review between Site Visits

RRC Site visit prep:

 Outline approach to PIF  Strategize with PC/GME Midpoint 3-4 months prior 3-4 months prior  Submit draft to DIO  Submit final draft to GME/DIO  Final PIF to site visitor   Meet with faculty Meet with residents/fellows 4-6 weeks prior 3-4 weeks prior 2 weeks prior 2-4 weeks prior 2-4 weeks prior

Site Visitor – ACGME Field Staff

 Two types: Field representatives and Specialist Site Visitors. (Field reps do 1940 of the 2000 annual visits e.g. 125 visits per year each – Specialist Site Visitors do approximately 60 visits) – Field Representatives are employed by the ACGME, and are expected to be knowledgeable in the Program Requirements and on-site review of the medical specialty and subspecialty areas accredited by the ACGME.

 Bios on ACGME web site: 31 individuals: 24 MDs, 6 PhDs, one EdD  Role: produce report that verifies and clarifies the information provided in the PIF and from the ACGME resident survey  Will interview program director, faculty, residents/fellows, and DIO 

SITE VISITOR IS A FACT-FINDER for the RRC, NOT THE DECISION-MAKER

Prerequisites for Accreditation

 I. Institutional Requirements  II. Common Program Requirements  III. Specialty-Specific Program Requirements

Read your requirements…

Understanding Requirements

 Must: Present without fail  Should: Absence justified by sound rationale  Desirable: Viewed as enhancement to program

Understanding Requirements

 Must: Duty hours must be limited to 80 hours per week, averaged over a 4-week period.

 2) Must/Should: Adequate time for rest and personal activities must be provided. This

must

consist of an 8-hour time period and between duty periods and

should

consist of a 10-hour time period between duty hour periods.

Understanding Requirements

 Old common requirements: – Assessment should include …. evaluation by faculty, patients, peers, self, and other professional staff.  New common requirements: – The program must… use multiple evaluators (e.g., faculty, peers, patients, self, and other professional staff);

Know your requirements…

Read your PIF…

Program Information Forms (PIF)

 Common PIF – Accreditation information – Sites – Faculty – Resident/Fellow appointments – Evaluation – Duty Hours

Program Information Forms (PIF)



Specialty Specific PIF

– Section for each competency – Duration and scope of education – General institutional information – Physician wellness – Faculty supervision and research projects – Duty hours and work conditions – – Best practices Comments – Goals and objectives

PIF Strategy for Program Directors

 Download the PIF a year in advance  Start working on one section at a time  Do not expect your coordinator, other faculty or your mother to write the PIF  You know your program the best of anyone!

Program Information Form Preparation (PIF)

 Review old PIF, internal review report, ACGME letters and responses (if any)  Cannot just update your old PIF – PIF format has changed!

 Prepare PIF with current program requirements open side by side!

Who are we?

The Stanford University Programs sponsored by Stanford Hospital and Clinics

How do we explain ourselves?

 A master affiliation agreement exists between Stanford University School of Medicine and Stanford Hospital and Clinics. – LPCH has also signed this agreement but they do not have their own programs

Master Affiliation Agreements

 GME Department renews annually with major affiliates – VAH – SCVMC – UCSF – Kaiser

Program Letters of Agreement (PLAs)

 Formerly educational agreements  Program responsibility  Needed between program and affiliate site director  Must have program competency based goals and objectives  Must be signed and dated

Single Site Sponsor

 Does not apply to us!

 We have 87+ programs!

PIF Preparation Summary

         Write the PIF Clear, concise, in order, no missing information Beware of cut and paste errors – can not always use cut and paste tool Follow instructions!

Third parties to proof Must be accurate and concordant with resident feedback to visitor Agreements are current, signed and available Faculty CVs current, especially scholarly activity Coordinate with your DIO (who must sign) as to when (and where) to mail the PIF

Common Errors

 Document not spell-checked  Questions not answered  Blanks (Medical School, License Expiration, etc.)  Statistics not added correctly  % of time for faculty inconsistent within PIF  Pages not numbered  Type font does not match  CVs list more than requested years of publications

The Questions!!!

 Be sure to answer the question not market your program

Remember

 Answer the question concisely  Only provide documentation requested  Check and double check the PIF  Have someone else who knows your program read the PIF

Finally

 A messy or incomplete PIF can lead to a disaster  Some of the site visitors are non MDs and some are MDs from other specialties.

The PIF is Done, Now What?

 Preparing the Files, Forms,  Policies, and People  Preparation before the big day  The day of the visit

ORGANIZATION!!!

Preparation for the Day

Site Visitor

     Letter from ACGME Make contact immediately Be very accommodating Site Visitor will make specific scheduling requests – – – Who should attend Timeline for the day Additional information that should be provided Offer: – – – – Help with travel plans Parking at SHC Detailed directions and location information Breakfast/coffee/lunch

Preparation for the Day

Schedule

 Date is not usually flexible  All people requested by Site Visitor must be present   – Inform him/her immediately if someone is unavailable Clear, precise, organized – email to Site Visitor Once it’s set don’t change…especially on the day of the visit  Schedule one conference room for entire day  Arrange for tours of key facilities as requested by the site visitor

Site Visit Preparation Logistics

   Schedule Chair, Faculty, Residents/Fellows (peer selected), Administrators (including DIO) Review resident/fellow files (internal audit): application, educational plans (goals and objectives), rotation schedule and call schedules, conference attendance, evaluations Group meetings: – – Key faculty: review program requirements and PIF Residents/fellows:  Review program requirements and how you are meeting them.

 Review ACGME and GME resident surveys if available

Preparing the People

    Residents/fellows and participating faculty Distribute the PIF – should be read several times prior to the visit Review: – ACGME Resident Survey if available (4 or >) – – – – Policies Evaluation process ACGME competencies Goals and objectives – Etc.

Prepare everyone, but don’t tell them what to say!

Engaging Residents and Fellows

•Site visitor will want to meet with 10 12 (all if small program) •Must be peer selected, with representation from all years •Review requirements, citations, surveys; fatigue/sleep deprivation education; duty hours guidelines; competencies; rotation goals and objectives; evaluation of residents, faculty, and program, and

core competencies

Engaging Residents and Fellows

 “What would you like to see improved?”  Consider mock session with other PD/DIO/GME Resident Representative  Arrange coverage, exclude those post call, hand off pagers  Site visit is not optimal time to have new grievances aired for first time!

Preparation for the Day

Preparing the Documentation

 Two types: – Resident/Clinical Fellow personnel files – Program Documentation  Organized – No loose paper (binders or folders) – Be able to find things quickly when requested  Make sure all documentation matches what was reported in the PIF!

 Prepare one location with all required documents for review

Preparation for the Day

Program Documentation

       Binders and dividers Documentation of Internal Review (not full report) Program educational goals Competency-based goals and objectives (by rotation and PGY) Program Letters of Agreement (PLAs) Conference schedules and attendance Duty hour reports Compliance Summary ONLY      PIF Program policies, including: – – – – Supervision Duty hours Moonlighting Etc.

Evaluations (templates) – – – – – – Of trainees for each rotation Semi-annual of trainee by Program Director Summative evaluations of trainee completing program Of faculty by trainees Of program by trainees Of program by faculty Rotation and call schedules Faculty CVs

Preparation for the Day

Resident/Clinical Fellow Files

   Personnel files – one file per trainee Can organize by year, type of document, or both Transfers: include copies of correspondence between Program Directors  Include: – – – Appointment paperwork (each year) Evaluations of trainee  Divided by competency and complete set    Final evaluation for program graduates 360 evaluations 6-month evaluations by Program Director – Other educational information, including:     Procedure logs PowerPoint presentations In-service exam scores Conference attendance     Rotation schedules Published articles and other scholarly activities Records of disciplinary actions Moonlighting records

Day of Visit

      Conference room – – – – – – Private and quiet Make sure room is clean Sign on door Refreshments Computer and phone access if possible All binders and files should be in room, ready upon request (back table) Page/email reminders for all participants – 1 week prior, day before, and/or morning of Site Visit Program Coordinator should participate – Additional help for Coordinator to serve as a “runner” 3 copies of PIF (exactly what was sent) Stick to schedule – be on time Again, be as accommodating as possible

1-2 days Prior

 Give everyone a copy of the schedule  Review files with coordinator  Ensure the residency office, residency conference room, sleep rooms, are clean and orderly  Prepare the conference room – Do not disturb sign

Site Visit Day

 Very early – Check conference room and ensure all notebooks are in place  Text page everyone 1-2 hours prior  Additional staff standing by  Have everything at fingertips/credenza behind table (15 second rule) – – – 3 copies of PIF (identical to that sent) Resident files Other binders, other information

The RRC Decision

 Letter sent within 90 days of the RRC meeting where your PIF and Site Visit Findings are reviewed.

 Citations/areas of noncompliance delineated  Accreditation action – Provisional (new application) – Continued full accreditation – Adverse decisions (subject to appeal):  Probation (proposed)  Withdraw (proposed)  Withhold (proposed)

Progress Reports and Letters

 RRC may request a response to major concerns  Use tone of program improvement, not defensive  Must be reviewed by GMEC and DIO, sent in triplicate, sent on time  Only send responses to RRC when requested  Written response to other concerns sent to GMEC

What can Programs Do Between Cycles to Improve?

 Annual program review and follow-up plans for improvement (a “must”)  Internal review and follow-up recommendations (also a “must”)   GMEC meetings, program coordinator meetings, workshops Program Directors’ Annual Retreat (Sponsored by GME)  National meetings

Summary

   PLAN: Allocate enough time Call DIO/GME with any questions Learn (and have others learn): – – The Institutional, Common and Program specific requirements The Core Competencies    Write the PIF side by side with the requirements Anticipate the unanticipated during the site visit Engage residents/fellows in the process early

Lessons from a Dinosaur….

 The “typical” site visit  How to answer site visitor questions…..

  How to evaluate the site visitor And the inevitable…..”horror” stories

Gentle Words of Wisdom

 “I’m only a good teacher because I have made all of the mistakes before…myself.” Norman Shumway, M.D.

The Site Visitor

 Reach out via email to the site visitor to welcome – Ask if they have any special requests – Announce your availability

The Site Visitor

 Answer the questions – Be terse – – Be tight Avoid “we used to …” “we will”

Gentle Words of Wisdom

 Keep notes – Likes and dislikes of site visitors – – – Who did what programs Any correlation between site visitors and accreditation cycle Don’t hesitate to call ACGME with problems!!!!

Horror Stories…..

   Oops…sent the wrong version of the PIF Oops…forgot to download the correct version of the PIF Oops…the Golf Course  And the lavender dress vs. the golden retriever

Questions

 And that’s all folks…

Enjoy!

Questions?

 Ann M. Dohn  [email protected]

DIOs Role in Teaching

 PIFMANSHIP for – Program directors – Coordinators

How We Teach

 PIF writing

PIFmanship 101

“I see a Site Visit coming in your future…”

PIF Writing 101

 Gentle words of wisdom… – The first impression of your program for the site visitor will be your PIF.

  Be accurate Answer the questions (be terse…be tight)  Have documentation to back up your answers  Start early-it takes months to write a good PIF  This is not something your mother, spouse, best friend, or admin asst. can do for you!

General Information

    Glossary of ACGME terms : http://www.acgme.org/acWebsite/ab out/ab_ACGMEglossary.pdf

Examples of “common information” required in the PIFs we provide to our Program Directors & Coordinators: 1) Hospital Statistics: – 613 Licensed Beds at SHC – 272 Beds at LPCH  52 Obstetric  220 Pediatric

2. Major Participating Institutions (Affiliates)

 Children's Health Council [058191]  Kaiser Permanente Medical Center (Santa Clara) [050571]  Kaiser Santa Teresa [058092]  Lucile Salter Packard Children's Hospital at Stanford [050572]  San Mateo Medical Center [050585]  Santa Clara Valley Medical Center [050438]

3. Library Resources

 http://lane.stanford.edu/services/ resourcecore.html

Background

 There are actually 2 PIFS to complete: – – The Common PIF  Addresses the program’s compliance with the Common Program Requirements common to all GME programs.

 The common PIF is electronically generated through the Web Accreditation Data System (WebADS) The Specialty PIF  Addresses compliance with the specialty specific program requirements  Word document downloaded from the ACGME Site

Common PIF

 Common Program Information Form  www.acgme.org

 Use your ACGME assigned User ID & password  Go to the PIF preparation section (left hand menu)  Update the Common PIF data.  Most of the data should be updated annually by your coordinator.

 The common PIF contains questions regarding participating sites, faculty CVs, evaluation, and duty hours.  Some information is entered only at site visit time.  Once all of the data is entered and VALIDATED  Print

Common PIF –1

Common PIF

 Data feeds from ACGME WebADS  TWO Original Signatures Required – Program Director – DIO

Common PIF – 2

Common PIF

 Respond to previous citations – Data feeds from ACGME WebADS – If you had a progress report due, the data remains in WebADS  Automatically populated into your next PIF

Common PIF

 Major Changes are: – Only since last site visit – Program Format, e.g.,  Have you gone from 3 to 4 years?

– Change in resident complement?

– – Change in Program Leadership?

Change in participating sites?

Common PIF – 3

Common PIF

 Participating Sites – Feeds from ACGME WebADS – Stanford is NOT a single limited site sponsor – – Only required sites appear on the PIF  Required means all residents rotate to that site Length of rotation  Make sure the length of rotations adds up to 12 months per year across all sites

Common PIF - 4

Common PIF

 Participating Sites – Program Letters of Agreement (PLAs)  We provide a PLA template on our GME Website for Programs to use and educate that at Stanford, e.g., a separate PLA is required for the Children’s Hospital, because the Faculty is Stanford University-i.e..,  The PLA may be between you and yourself…

Common PIF - 5

Common PIF

 Program Director MUST be able to approve the teaching faculty  The Program Director MUST evaluate the faculty and approve continuation as teaching faculty  Program Directors MUST comply with the university’s or medical center’s policies on selection, evaluation, promotion, disciplinary action and supervision.

– – e.g., at Stanford we refer them to: The GME House staff Policies and Procedures – on the GME Website http://med.stanford.edu/gme/policy/

Common PIF

 The Program Director MUST be familiar with ACGME Policies and Procedures – – Institutional Requirements Common Requirements – Program Specific Requirements

Common PIF

 Physician Faculty Roster – List Alphabetically and by Site – Faculty who spend at least 10 hrs. per week in resident education – Make sure the data matches their CVs

Common PIF

 Faculty Teaching Staff – – Hours/week devoted to education  Should be realistic Board certification  If double boarded include both boards  If you are sub-specialty program director or faculty, ensure primary boards are included

Common PIF- 6

Common PIF

 Potential Area for a HUGE number of citations for Programs: – Must be totally completed (no BLANKS) – – Ensure certification is valid Ensure license is current and hasn’t lapsed

Common PIF

– Concise Summary of Role in Program:  oversees the operations of the entire program; supervises trainees during patient-care activities, mentors trainees’ research/projects, provides professional training, tutorials (lectures and seminars); monitors duty hour compliance; coordinates evaluation of courses, rotations, trainees and faculty.

Common PIF

 Selected Bibliography – Strict limit of 10  – – Strict limit only the last five years “Publications from the last 5 years” excludes any before 2006… Publications should not be “in press” if submitted many years ago….

If not ABMS certified, explain…

Common PIF - 7

Common PIF - 8

Common PIF

 Number of ACGME Approved Positions – Note: If you have a resident making up a maternity leave, you must ask ACGME for an extra slot if you are over your cap  Actively enrolled residents – Other than interns, everyone should have prior years of GME

Common PIF - 9

Common PIF-10

Common PIF

  Transferred, Withdrawn or Dismissed Residents – – Residents who resign are NOT dismissed Must have summative evaluation on transferred trainees Evaluations – Yes residents are evaluated following each learning experience (i.e.. rotation) – Yes- evaluations are documented and state how  Electronically  Files

Common PIF

 Methods of Evaluation – – Assessment Method  Direct Observation  Simulation  OSCE  Etc.

Evaluator  Faculty  Nurses  Program Director  360 (required as of 2007)

Common PIF -11

Common PIF Evaluation

 Describe how evaluators are educated to use the assessment methods listed above so that residents are evaluated fairly and consistently.

– The Program Director meets with evaluators annually before new fellows start in the training program and reviews, answers questions and discusses the core competencies and core competency performance evaluations used in the program and assessment methods with the faculty. The electronic assessment system and the scales and their use is also reviewed and discussed in faculty meetings in order to ensure that faculty are fully educated and up to date with the assessment methods and processes.

Common PIF Evaluation

 Describe how residents are informed of the performance criteria on which they will be evaluated.

– At the beginning of each academic year, the Program Director conducts an orientation to address several key issues related to the residency including the performance criteria on which the residents will be evaluated. During this orientation session, the Program Director carefully details the specific evaluation methods to be used. Both the criteria and methods are also documented in the residency handbook. Additionally, at the beginning of each block rotation, the rotation director meets with the resident and carefully delineates the expectations and performance criteria on which the resident will be evaluated for that specific rotation.

Common PIF Evaluation

 Describe the system to ensure that faculty complete written evaluations of residents in a timely manner following each rotation or educational experience.

– At the end of each block rotation, our online resident data collection and tracking system, MedHub, sends an automated reminder to the service attending(s) to evaluate the resident(s) on that particular rotation. Using MedHub, the Program Coordinator tracks pending evaluations and follows up with faculty as needed to urge them to complete their evaluations on time. In the unlikely event that a faculty member is unresponsive to the coordinator's requests to complete evaluations, the Program Director contacts the faculty member and requests him/her to complete the evaluations.

Common PIF Evaluation

 Describe the process used to complete and document written semiannual resident evaluations, including the mechanism for reviewing results (e.g., who meets with the residents and how the results are documented in resident files).

– Residence performance is evaluated by the teaching faculty at the conclusion of each rotation using an electronic evaluation form (MedHub). Additionally, each faculty mentor meets with their resident advisee quarterly and documents a summary of the meeting to be placed in the resident’s file. The Program Director meets with each resident on an individual basis semiannually to review the accumulated performance evaluations and mentor notes, provide feedback, and update the resident’s learning plan as appropriate. A summary of these meetings is documented and placed in the resident’s file. The residents are free to review the contents of their records at any time.

Common PIF Evaluation

 Describe the system used by the residents to provide annual confidential written evaluations of the teaching faculty (have examples and forms available for review by site visitor).

– – Residents annually confidentially evaluate the teaching faculty using . The electronically submitted . evaluation forms are anonymous. Paper copies of completed evaluations without personal identifiers are printed by the Program Coordinator, reviewed by the Program Director, and presented to individual faculty members for review and consideration. Written evaluations are completed electronically and anonymously by using a centralized system used throughout the hospital to provide annual aggregated data. Fellows are sent e-mail reminders about completing evaluations in a timely manner by the Fellowship Coordinator. The program coordinator follows up, aggregates the results and forwards them to the Program Director. The Program Director reviews the results and takes any actions appropriate with individual program faculty.

Common PIF Evaluation

 Describe the program's (or Department's, if applicable) system for evaluating and providing feedback to the teaching faculty.

– Residents annually evaluate the teaching faculty using the . The electronically submitted evaluation forms are anonymous. Paper copies of completed evaluations without personal identifiers are printed by the Program Coordinator, reviewed by the Program Director, and presented to individual faculty members for review and consideration. If and when the evaluations reveal particular issues with the teaching faculty, the Program Director meets with the particular faculty more urgently to address those issues and concerns. Additionally, the Program Director meets with the all faculty on an annual basis to review resident feedback and to effect any necessary changes.

Common PIF Program Evaluation

 Describe the approach used for program evaluation.

– Residents and faculty annually evaluate the program using the . The electronically submitted evaluation forms are anonymous. Residents and faculty are also encouraged to provide feedback to the Program Director whenever any issue arises or as they see opportunities for improvement. Additionally, residents and faculty participate in an Annual Program Improvement Meeting led by the Program Director. Aggregated data including the most recent ACGME survey results and the resident/faculty program evaluation results are reviewed and used at this meeting to improve the program. The Program Coordinator keeps minutes during this annual meeting and documents any plans to address areas requiring improvement. Action plan progress is monitored and documented by the Program Director.

Common PIF Program Evaluation

 Describe one example how the program used the aggregated results of residents' performance and/or other program evaluation results to improve the program (have the written plan of action available for review by the site visitor).

– – Annual fellowship retreats including fellows and faculty are conducted each year in May or June (Whenever yours is held) ….the agenda for these meetings includes review of all the aforementioned aggregated data as well an open discussion about educational quality and compliance with the program requirements and suggestions for improving the program for incoming residents. (See GME Website for more information on this and examples of written action plans) http://med.stanford.edu/gme/program_directo rs/eval_prog.html

Common PIF -12

Common PIF

 Describe the improvement efforts currently undertaken based on feedback from the ACGME Resident Survey.

– You MUST review your ACGME Annual Survey each year, bring the results to your Annual Program Retreat, address each area of concern (more than 20%) or any Duty Hour negative response

Common PIF

 DUTY HOURS – Use the summary from your tracking system reporting function – Describe

Common PIF

 Briefly describe how the faculty provides appropriate supervision of residents in patient care activities.

– Faculty discusses all patient care history and physical findings with resident. Faculty then confirms both the history and physical findings of the patient. The patient is then discussed in detail, education given regarding the patient illness or complaint, education materials are provided to the resident and finally, a care plan is developed and provided by the resident and confirmed by the faculty.

Common PIF

 Other Questions that may be included in you Common or Specialty PIF

SPECIALTY PIF

 The specialty specific PIF is located on the ACGME website under your Residency Review Committee.

– http://acgme.org/acWebsite/navPages/ nav_PDcoord.asp

 The specialty PIF contains the questions regarding the ACGME core competencies, block diagrams for your program, a narrative description of the program, scholarly activity, and often case logs.

Specialty PIF-Practice Based Learning & Improvement

 Describe one learning activity in which residents engage to identify strengths, deficiencies, and limits in their knowledge and expertise (self reflection and self-assessment); set learning and improvement goals; identify and perform appropriate learning activities to achieve self identified goals (life-long learning).

– Programs may use a structured process for reflection in which a faculty advisor guides the resident in using feedback and evaluations to inform the self assessment process. Documentation of the semi-annual evaluation meetings in which this process is demonstrated would provide evidence that this requirement is being addressed.

Specialty PIF-Practice Based Learning & Improvement

 Describe one example of a learning activity in which residents engage to develop the skills needed to use information technology to locate, appraise, and assimilate evidence from scientific studies and apply it to their patients’ health problems. The description should include: a) locating information, b) using information technology, c) appraising information, d) assimilating evidence information (from scientific studies), e) applying information to patient care.

– An appropriate learning activity could be a structured activity such as a journal club presentation, critical appraisal of a topic, or educational prescription with appropriate faculty oversight and formal assessment of skills. Additional documentation would be the written goals and objectives for this learning activity and how residents are assessed.

Specialty PIF Professionalism

 Describe at least one learning activity, other than lecture, by which residents develop a commitment to carrying out professional responsibilities and an adherence to ethical principles.

– This activity should be structured, should demonstrate active faculty involvement (not just passive role modeling) and timely feedback to residents, and should include a mechanism for collecting evaluations (including routine multi-source assessment).

Specialty Systems-Based Practice

 Describe an activity that fulfills the requirement for experiential learning in identifying system errors. – – Important elements may include identified faculty to guide the activity, mechanism to ensure active engagement by each resident, and evidence of experiential learning (not just passive presence at conferences and meetings) in which residents participate in identifying a system problem or error and contribute to a potential solution.

Additional documentation could include written goals and objectives for this learning activity and how residents are assessed. Aggregated resident outcomes may be in the form of percentage of residents that completed a patient safety or other Systems Based Practice project by the end of training, annual list of improvements that resulted from such projects, etc.

PIF

 UPDATE-REVISE Briefly describe how the faculty provides appropriate supervision of residents in patient care activities.

Common Errors

 Table of Contents Inaccurate  Pages not numbered or not numbered correctly  Document not spell-checked  Question asked not answered  Statistics not added correctly  % of time for faculty inconsistent within the PIF  Type font does not match

Putting It All Together

     REVISE---One complete PIF (common and specialty) is due to the site visitor at least 14 days before the site visit.

Stanford’s GME needs the complete PIF 14-30 days before it is send to the site visitor. (the math means that you should have the completed document ready 6 weeks BEFORE your site visit) Stanford’s GME usually has 3 site visits during a week. This translates to 3 PIFS to review. Help the poor folks out….and be EARLY!!!! The more time you give GME…the more time for editing, comments, and changes. You really cannot have too many eyes look at one PIF!!

Save time in the PIF Site Visit/Internal Review Process

 Leverage Inexpensive Technology Applications e.g., – PDF – PIF Review Functionality – Microsoft Word Track Changes  Documents revisions that are required  Electronic copies can easily be uploaded to internal web sites for easy access and documentation  Enables team reviews – from “anywhere”  Provides Program Directors easy review of recommendations  Provides Program Coordinators a streamlined way to make many of the required changes

The Land Mines

 As you write your PIF… remember to read the question carefully.

– – – – One example does NOT mean several “How” does not mean “we do” No “will’s” No “see attached – “see below”