Brief Critique of IES Student mentoring evaluation

Download Report

Transcript Brief Critique of IES Student mentoring evaluation

+

Jean Rhodes National Mentoring Partnership Professor Minnesota Mentoring Partnership October 24, 2010

For Which Youth is Mentoring Most Effective?

+

    How effective is youth mentoring for all youth?

For which youth is it most effective?

Under what conditions is it most effective?

What are the implications for policy, practice, and research?

+

Recent Meta-Analysis of Youth Mentoring

 Encompassed 73 independent evaluations (1999-2010).  The overall effect size was .21*, collapsing across studies and outcomes  The average follow-up effect size across the studies was .17.

DuBois, Portillo, Rhodes, Silverthorn, & Valentine (in press). Psychological

Science in the Public Interest.

+

Comparison of mean post-tx effects with school and community interventions with children/teens

Moderator

Attitudinal/Motivational

Mentor Meta-analysis Other Meta-analyses

0.19

0.23- 0.25

Social/Relational l Psychological/Emotiona Conduct Problem Academic/School Attitudes Grades Achievment tests Physical Health 0.17

0.15

0.21

0.21

0.19

0.24

0.18

0.06

0.15-0.26

0.10-.0.24

.02-0.41

0.11-0.27

0.14

0.22

0.11-0.24

0.08-0.41

+

Effect sizes

0.5

0.4

0.3

0.2

0.1

0 -0.1

Medium Effect Small Effect 0 1 2 3 4 5 6 7

Number of Practices

8 9 10 11 Empirically Based Practices Theory-Based Practices

+

Study level variables (moderators) associated with different effects

Youth, Mentor, Program Characteristics Effect Size

Problem Behavior Involvement Youth Gender Individual/Environmental Risk Evidence-based Mentor Training Yes: .29

No: .20

>50% Male: .25

<50% Male: .18

Low/High: .33

High/Low: .31

Below median: .19

Above median: .24

Mentor role function: Advocacy Yes: .26

No: .20

Matching based on shared interests Yes: .44

!!!

No: .21

+

    How effective is youth mentoring for all youth?

For which youth is it most effective?

Under what conditions is it most effective?

What are the implications for policy, practice, and research?

+

Stronger effects when…

Youth with  With moderate personal/environmental risk   Who are male satisfactory, but not strong baseline relationships.

+

Effects of Mentoring on Youth with Different Relational Profiles

BASELINE Relationship s

Overall Academics Prosocial Effort Self-Esteem

Poor Relationships

.00

Satisfactory but not Strong Strong Relationships

.21*** .05

.04

.05

-.04

.19* .18* .07

.04

.00

-.01

(Schwartz, Rhodes, & Chan (2010). Developmental Psychology

+

Stronger effects when…

Mentors who   Fit of background/ training with program goals Play an active, advocacy role    Are sensitive to socioeconomic & cultural influences Have higher self-efficacy

Hold positive attitudes toward youth

+

Measuring mentors ’ attitudes Grossman et al., 2007  The scale asked mentors to rate how many “ kids in your community ” could be characterized by indicators of youth development:  work hard at school  respect adults  are trouble-makers  are fun to be around  expect things to be handed to them  try to do their best  are interested in learning

+

Mentor attitudes and youth outcomes Karcher, Rhodes, Herrera, & Davidson (2010). Applied

Developmental Science

 Vulnerable mentees who were paired with high school mentors with positive attitudes negative mentors about youth were more emotionally engaged with mentor than those paired with more  Those who were paired with high school mentors with negative attitudes about youth were less emotionally engaged with and showed some negative outcomes.

Stronger effects when…

Relationships characterized by   consistency closeness    structure

appropriate meeting times duration

+

Does meeting time matter?…

Mentoring by Meeting Time

Afterschool/Lunch Academic Achievement Math Science Social Studies Reading Language Unexcused Absences Grades Scholastic Efficacy

*Note ES= Effect Size

After School During School .19* -.09

.18

.14

-.42** -.19

.11

.27* .21* -.35

-.20

.12

-.16

-.02

-.05

-.08

-.15

.13

Interaction Effect -.33* -.60** -.36* -.29

-.32

-.33* .21

-.51

.03

The role of duration

Grossman & Rhodes ( 2002). American Journal of Community Psychology

+

Re-matching?

+

Test of Time 2: Results

 Only youth in matches lasting 24 wks or more benefited academically  All mentored youth were less likely to skip school, regardless of match length  After controlling for selection bias:  Positive academic impacts observed only for youth with intact matches   No academic impact for youth with early terminations

Negative academic impacts for rematched youth

 Grossman, Chan, Schwartz, & Rhodes (in press). American Journal of Community

Psychology.

+

Stronger effects when…

Programs characterized by  careful recruitment   training monitoring   Multi-modal

matching on interest

+

    How effective is youth mentoring?

When are programs most beneficial?

How does mentoring promote positive youth development?

What are the implications for policy, practice, and research?

+

    How effective is youth mentoring for all youth?

For which youth is it most effective?

Under what conditions is it most effective?

What are the implications for policy, practice, and research?

Pathways of mentor influence

Mentor

Mutuality Trust Empathy

Relationship mediator Parental/peer relationships Social-emotional development Identity development Positive Outcomes e.g

., reduced health risk, better psych.

outcomes Cognitive development Interpersonal history, social competencies, relationship duration, developmental stage, family and community context

Pathways of mentor influence

Scholastic Competence .25

.26

.08

.29

Grades Mentoring .22

-.28

Quality of Parental relationship .25

.26

Self-worth .19

.18

.09

School value .11

Skipping School Child Development, (2002), 1662-1671

Pathways of mentor influence

Mentoring .23

Quality of Parental relationship .18

.10

-.46

Self-worth -.04

.14

-.08

Substance Use Quality of Peer relationships Rhodes, Reddy, & Grossman (2005) Applied Development Science

Pathways of mentor influence

Academic Attitudes .25

.78

Quality of Mentoring .27

.13

Quality of Teacher relationship .53

.32

-.28

Quality of Parent Relationship Self-worth .27

.09

Grades .18

School Behavior Chan,Rhodes, Schwartz, & Lowe (2011)

+

   How effective is youth mentoring for all youth?

For which youth is it most effective?

Under what conditions is it most effective?

What are the implications for policy, practice, and research?

+

Recommendations

 Promote  evidence-based practice   include rigorous evaluation measured replication and dissemination  Reward sustainability and quality over growth

+

Research to practice

Research

Implementation • Youth Serving Organizations Distribution • MENTOR/NMP Product Development.

+

Action steps

 Develop and improve training and support around relationships  Improve mentor retention  Employee retention, qualitative studies  Export to other youth-serving settings

+

Give Psychology Away

+

Credits

Big Brothers Big Sisters of America WT Grant Foundation Edna McConnell Clark Foundatin NICHD