Transcript Enhancing Competency Based Training Through the Screening

Enhancing Competency Based
Training Through the Screening,
Selection, and Performance
Evaluation of Psychology Interns
Paul Robins, Ph.D1,2., Thomas Power,
Ph.D1,2., Mary Rourke,Ph.D1., Melissa
Alderfer, Ph.D1,2., Cathi Grus, Ph.D3., &
Marley Watkins, Ph.D4
1The
Children’s Hospital of Philadelphia, 2University of
Pennsylvania, 3APA, 4Arizona State University
Competency Based Training
Forms a basis for defining and measuring
trainee learning outcomes
Competencies conceptualized as elements or
components of competence,
and consist of
l
discrete knowledge, skills, and attitudes (Kaslow
et al., 2004).
Cube model: twelve core competencies
conceptualized as either foundational or
functional competencies (Rodolfa, et. al, 2005)
Today’s Presentation
1.
2.
3.
4.
5.
Describe process of developing a blueprint of
foundational and functional competencies
Describe competency based intern evaluation
instrument
Describe process of developing an intern
screening tool that reflects core mission
priorities
Present reliability and validity data of the intern
screening tool
Future directions/learnings
Developing Core Competencies
Foundational documents
–
–
–
–
Hospital mission
Behavioral Health Center mission
Internship program mission
Funding priorities: GPE and LEND missions
Translate documents to core training domains
– Iterative process
– Responsive to multiple missions and our
understanding of training psychologists as leaders in
the field
Foundational Documents:
Hospital Mission
The Children’s Hospital of Philadelphia, the
oldest hospital in the United States
dedicated exclusively to pediatrics, strives
to be the world leader in the advancement
of health care for children by integrating
excellent patient care, innovative research
and quality professional education into all
of its programs.
Behavioral Health Center Mission
The Children’s Hospital of Philadelphia’s Behavioral Health
Center (BHC) strives to be among the nation’s top
leaders in children’s behavioral health. The BHC
provides an inter-disciplinary approach to improving the
behavioral health and development of children and their
families and advancing knowledge through excellent
clinical care, research, training and advocacy. The BHC
promotes innovative research and the integration of
science and practice, focusing on prevention and
intervention. We partner with others within the hospital,
regionally, and nationally to increase the availability of
evidence-based behavioral services for all patients at
CHOP. The BHC is committed to outstanding scientistpractitioner training and to supporting the competence of
caregivers and staff. Our endeavors integrate disciplines
and systems of care in children’s health.
Internship Program Mission
The Internship Program aims to prepare advanced
doctoral-level students to take leadership roles
in developing, providing, evaluating, and
disseminating effective psychological services
for the diverse population of children and
families with health and/or mental health needs.
The Program is committed to preparing
psychologists as leaders in the field, engaged in
clinical practice, advocacy, interdisciplinary
collaboration, and scientific investigation.
Funding Agencies
HRSA: BHP (GPE) & MCHB (LEND)
Vision
The Health Resources and Services Administration
(HRSA) envisions optimal health for all, supported by a
health care system that assures access to
comprehensive, culturally competent, quality care.
Mission
HRSA provides national leadership, program resources
and services needed to improve access to culturally
competent, quality health care.
Development of Core
Competencies
Functional Competency Domains (6)
1.
2.
3.
4.
5.
6.
Assessment
Intervention/prevention/advocacy
Consultation
Research
Professional development
Diversity effectiveness
Identify Unique Strengths of
Program—the “Big Five”
Foundational Competency Domains
1. Prevention/advocacy (BHP/GPE, future)
2. Ecological/systems underpinning (our
unique history, PCGC)
3. Leadership training (Hospital, MCHB/LEND)
4. Interdisciplinary collaboration (Hospital—
family centered care, systems)
5. Community based, medically underserved
populations (BHP/GPE)
Content Validity--Blueprint
6 Functional Competency Domains X 5
Foundational Competency Domains—55
items
Provided blueprint of core skills and
outcomes (content validity)
Formed basis of required intern skills and
outcomes
Assessing Intern Competencies
Core competencies rated by rotation
supervisors along a 3 point scale (scaling
being revised to reflect APA CoA standards)
“Based upon end of the internship performance
expectations, rate this intern on each item using the
following scale:
n/a = not applicable
1 = does not yet meet expectations
2 = meets expectations
3 = exceeds expectations (second semester only)
Assessing Intern Competencies
Item Examples
Assessment
– Demonstrates effective clinical interviewing
skills with children and families
– Demonstrates ability to effectively collect and
integrate data from multiple sources and/or
disciplines
Assessing Intern Competencies
Research
– Effectively searches for and applies most
relevant empirical findings to inform clinical
practice
– Demonstrates skills in developing research
designs
Intern Selection
Develop screening method/instrument
which assists us in identifying interns who
are a strong match based on our mission
and values
– Across functional and foundational
competency domains, our blueprint
Description of Screening
Instrument
Iterative process, revisions, & training
13 items total, anchored Likert scale (1-4)
Breath of clinical training (5 items), across:
– Systems of care (e.g., clinic, hospital, community, school)
– Treatment modalities (e.g., individual, group, family)
– Age/demographics (e.g., infants, toddlers, school age,
adolescents)
– Presenting diagnoses (e.g., mental health, medical)
– Clinical activities (e.g., assessment, intervention,
consultation)
Description of Screening Tool
Depth of clinical training (3 items)
– Amount/type of clinical supervision
– Direct treatment hours (adults and children)
– Number of comprehensive assessment
reports (adults and children)
Description of Screening Tool
“Big Five” items (Foundational)
– Developmental-ecological model
– Diversity effectiveness
– Interdisciplinary collaboration
– Research potential
– Leadership potential
1.
The Children’s Hospital of Philadelphia
Psychology Internship Training Program
Pre-Doctoral Internship Selection – Screening Form
(Class of 2008-2009)
Rate breadth of training…across multiple systems of
care (e.g.,mental health, medical, school, and community
systems)
3
2
1 or more training experiences within 4 systems of care
1 or more training experiences within 3 systems of care
1 or more training experiences 2 systems of care
1
1 training experience within 1 system of care
4
with regard to various treatment modalities (e.g., individual
therapy, family therapy, multiple family therapy, parent training,
child group intervention {i.e. classroom-based} and group therapy)
4
Training experience in 4 or more treatment modalities
3
Training experience in 3 different treatment modalities
2
Training experience in 2 different treatment modalities
1
Training experience in only 1 treatment modality
6. Rate level of ethnic, linguistic, and cultural diversity awareness and
effectiveness, taking into consideration experience in and commitment to learn
in this area as demonstrated through the following domains: research,
coursework, teaching experiences, seeking practicum placements, case
conceptualization, and committee membership.
4
Diversity effectiveness is a clear, defining priority, guiding graduate
training, as evidenced in more than 3 of the above domains
3
Diversity effectiveness informs graduate training and is important as
demonstrated in at least 3 of the above domains
2
Diversity effectiveness is a part of graduate training as
demonstrated by awareness and exposure in at least 2 of the above
domains
1
Diversity effectiveness is reflected in only 1 domain or is not evident
in training
8. Rate research potential in child-oriented psychology.
4
Extramural pre-doctoral grant and 1+ first author peerreviewed journal publication (excluding submitted for
publication)
2
1 first author publication (excluding submitted for
publication) or a pre-doctoral grant; plus at least 2
national poster or paper presentations as a first author
2 1 or more publications (including in press) on which
applicant is not the first author; plus 1 or more national
poster or paper presentations
1 No publications; may have some national poster or
paper presentations
Screening Process
Administrative screening: ~210
applications/year
Each of ~190 applications reviewed by 2
psychology staff using screening
instrument (early Nov-mid Dec)
Total score obtained
Preliminary rank order determined
Hand reviewed by TD
35 on -site interviews
Psychometrics:
Construct Validity
Factor Analysis—Reviewer 1
–
–
–
–
N = 382
2 factor solution
Factor 1 alpha = .69 (acceptable)
Factor 2 alpha = .70 (acceptable)
Factor Analysis—Replication (Reviewer 2)
– N = 382
– 2 factor solution
– Factor alphas .69 and .72, respectively
Factor 1 & 2 congruence coefficients = .94 & .98
(good to excellent factor similarity between sets of
raters)
F1 and F2 r = .44 (moderate)
Pattern Matrix
Item
1
2
Interdisciplinary collaboration
.70
-.08
Developmental ecological model
.68
.01
Diversity effectiveness
.50
-.03
Leadership effectiveness
.48
-.00
Research potential
.42
-.01
Practicum hours
.39
.10
Supervision quality
.17
.13
Number assessment reports
.15
.02
Number intervention cases
-.13
.69
Breath across age groups
-.00
.65
Breath across treatment modalities
.02
.62
Breath of clinical activities
.27
.41
Systems of care training
.22
.24
Reliability
Alphas:
– Factor 1 (Reviewers 1 & 2) = .69 &.72
– Factor 2 (Reviewers 1 & 2) = .70 & .70
– “acceptable”
Inter-rater reliability, factor level
– F1 = .62
– F2 = .61
– “marginal”
Results suggest:
•
Two factors makes sense—capture
breath of clinical experience and
foundational items (but still only capture
27.4% of variance)
•
Use inter-rater item correlations
Foundational or Functional?
Conclusions
Attempt to operationalize core
competencies—step in the right direction
Reflects our mission, our unique identity
Ongoing process, iterative
Screening tool:
Clarify items: better anchor response choices
Increase training (e.g., develop scoring manual with
examples)