Dr John Bamber - National Youth Council of Ireland

Transcript Dr John Bamber - National Youth Council of Ireland

Demonstrating outcomes in youth work: challenges and possibilities

Dr John Bamber

Aims for the input

1. To explain the concept of

evidence

context in a youth work 2. To distinguish between the terms ‘evidence-based’ and ‘evidence-informed ‘ 3. To discuss the possibilities afforded by an evidence informed approach to youth work, involving a theory of change, and a logic model 4. To set out the challenges involved for different actors 1

Why evidence?

The Age of RESULTS and ACCOUNTABILITY Key question: What difference does youth work make?

Is there a hierarchy of evidence?

http://www.campbellcollaboration.org/ 6.

10.

11.

12.

13.

14.

Systematic reviews Objective?

Randomised control trials proof of ‘what works’?

Quasi-experimental study ‘Before’ and ‘after’ designs Retrospective designs (no baseline) Independent large scale evaluations Traditional literature review Independent small scale evaluations Independent single studies Opinion pieces – peer reviewed journals Expert panel/committee Internal evaluation – large or small scale Practice wisdom – policies, standards, manuals Personal experience – reflective practice Subjective?

Evidence based programmes

http://www.colorado.edu/cspv/index.html

http://www.promisingpractices.net/ http://whatworks.uwex.edu/Pages/2evidenceregistries.html

An evidence informed approach

Professor Sandra Nutley, University of Edinburgh

CES Practice Seminar Series, October 2010 Keynote address Evidence-informed practice: Using research to improve services for children and young people

http://www.effectiveservices.org/ces-projects/p012 seminar-series-practice-issues 5

Knowledge required is more than ‘what works’

• • • • Know-about (problems): e.g. the nature, formation, and interrelations of social problems.

Know-why (requirements of action): explaining the relationship between values and policy and practice.

Know-how (to put into practice): e.g. pragmatic knowledge about service and programme implementation.

Know-who (to involve): e.g. care teams; building alliances for action. Sandra Nutley (2010) 6

Evidence informed is...

An approach that helps people and organisations make well-informed decisions by putting the best available evidence at the heart of practice development and service delivery.

Sandra Nutley (2010) 7

Good outcomes depend on effective action

Local decision-making, and the harnessing of local initiative, imagination and adaptation can be pre-eminent, but are much more likely to lead to effective action when combined with a broad collection of accessible knowledge that is drawn from practice and theory as well as research (Schorr, L., 2003: 21-21).

http://www.lisbethschorr.org/4.html

http://www.theoryofchange.org/about/what-is-theory-of-change/

Constructing a theory of change

Consultation Practice Research and Theory Policy

http://www.indefenceofyouthwork.org.uk/wordpress/wpcontent/uploads/2010/11/20252-Youth-stories report-2011_4th-1.pdf

Evidence from practice

• • • • •

Value led and educative Dialogical and empowering Deliberative and responsive Critical and reflective Process and results focussed

Evidence from theory – a selection

Psychology: Erikson

Psychosocial development

Communication: Berne

Transactional analysis

Political theory: Marx Class struggle Education: Piaget

Schemas, stage theory

Multiple intelligences Gardner Constructivism Bandura Reciprocal determinism Theory of knowledge: Habermas Communicative action Experiential Learning: Kolb

Learning cycle Learning styles

Action learning Revans L=P+Q Social change: Freire Praxis

Institute of Education, London. (March, 2012) International review of research literature on youth development . 2012. Commissioned by Centre for Effective Services on behalf of Department of Children and Youth Affairs, Dublin.

Evidence from research…a selection

McKee, V., Oldfield, C. and Poultney, J. (2010) The Benefits of Youth Work. Lifelong Learning UK and Unite.

Wiggins, M., Bonell, C., Burchett, H., Sawtell, M., Austerberry, H., Allen,E. and Strange, V. (2008) Young People’s Development Programme Evaluation: Final Report. Social Science Research Unit, Institute of Education and University of London with the London School of Hygiene and Tropical Medicine.

Bielby, G., Purdon, S., Agur, M., Gardiner, C., George, N., Golden, S. and Taylor, E. (2009) Empowering Young People Pilots (EYPP) Evaluation, Final Report. National Foundation for Educational Research.

Lewis-Charp, H. Hanh C.Y., Sengouvanh S. and Lacoe, J. (2003) Extending the Reach of Youth Development Through Civic Activism: Outcomes of the Youth Leadership for Development Initiative. Innovation Center for Community and Youth Development.

Eccles, J. and Appleton Gootman, J. Editors (2002) Community Programs to Promote Youth Development Committee on Community-Level Programs for Youth, National Research Council and Institute of Medicine. Washington, DC: National Academy Press.

Fullerton, D. And Burtney, E. (2010) Young People’s Health Community Based Approaches, A Rapid Review. Report to the Sexual Health Centre Cork. Insights Health and Social Research.

Evidence from policy

A common problem is that activities and strategies often do not lead to the desired outcomes. “I think you should be more

explicit here in Step Two.”

This is where evidence from consultation comes in…

Evidence from consultation

Goals Values Needs and issues Consultation: participants peers experts What are the needs, issues and interests?

What changes would people like to see?

What difference have we made, and has anybody benefitted?

Outcomes

Adapted from: Devlin, M. and Gunning, A. (2009) The Purpose and Outcomes of Youth Work Report to the Youth Services Interagency Group. Dublin: Irish youth Work Press

Proximal outcomes

Outcome Area

Attitudes, beliefs Knowledge Skills Behavior

Sample outcome statements

Participants are more ready to take on new and more diverse experiences Enhanced confidence, self-esteem, awareness (personal and social) More open to people from diverse backgrounds Increased knowledge of local area, of society Increased knowledge of rights, social issues, politics More informed about health, sexuality, the law, careers and formal education Enhanced capacity regarding: public speaking , problem solving, self-efficacy, making decisions, critical thinking Enhanced interpersonal abilities in relation to: teamwork, group work, communications In creased abilities in arts and creativity regarding: music, dance, drama, writing Increase in physical competence in relation to: sports, games, outdoor activities More engagement in structured and constructive activities Increased involvement in decision-making Enhanced positive and pro-social behaviour and diminishing negative and anti-social behaviour 16

Adapted from: Devlin, M. and Gunning, A. (2009) The Purpose and Outcomes of Youth Work Report to the Youth Services Interagency Group. Dublin: Irish youth Work Press

Distal outcomes

Outcome Area

Education Relationships with adults Health Social conditions Economic conditions Safety Service provision

Sample outcome statements

Deeper engagement with formal education and alternative routes to qualifications Improved communications and relationships between young people and adults within communities Safer and more enjoyable sex, reduction in alcohol and substance use Increased community cohesion, improvement in local conditions and amenities, developing local leadership Increased self-sufficiency Reduction in violence, vandalism, offending rates Increased support for homeless young people, Improved cooperation and coordination between local services 17

Chain of outcomes - example

Short

Young people recognise the benefits of engaging in informal learning, and spend more time in structured developmental activities

Medium

Enhanced sense of capabilities regarding community involvement, education, training

Long-Term

Learning skills acquired and transferred to other areas of life eg work, family life, politics Young men more able to articulate their needs and wants Enhanced political skills through their involvement in decision making in youth work settings and the wider community Young people friendly environment , and safer community for residents 18

Well defined information which shows whether something is happening.

Outcomes and indicators - examples

Outcome

Young people in the diversion programme less likely to offend.

Participants better able to manage conflict in relationships Improved relationships and closer family ties

Indicators

#,% Youth who will not be charged with a new crime 1-6 months after successfully completing the program. #,% Youth who will not be charged with a new crime 7-12 months after successfully completing the program.

#,% Participants demonstrate better knowledge of anger management in pre and post program surveys. #,% Parents and young people indicate improved communication at three month follow-up.

Outcomes and indicators - sources

http://www.urban.org/center/cnp/Projects/outcomeindicators.cfm

http://www.ces-vol.org.uk/ http://www.ceni.org/publications/research.asp

Goals Values Needs and issues Strategies Inputs Process Monitoring and evaluation Outputs Outcomes I N D I C A T O R S Evidence

Situation Analysis Goals derive from

Aim, mission, vision, values, ideology

Goals inform analysis

Populations and target groups Problems and causes, Situations and issues Current provision: strengths, weaknesses, gaps

Socio-economic and organisational context affect outcomes

Policy, legislation, funding, politics, economy, culture, local conditions

Strategies distil and give focus to goals

Outline of main ways to achieve outcomes

Logical Chain of Connections

Showing What the Program is to Accomplish Monitoring and Evaluation

Collecting and interpreting information about implementation and results, and reporting findings These processes take external factors into account and inform analysis, planning and implementation Targets and Indicators that can provide signs of progress or achievement, may derive from standards and benchmarks

Implementation Resources Activities Inputs (matched to strategies)

Funding Buildings Technology Partners Staff and volunteers

Process

Value led and educative Dialogical and empowering Deliberative and responsive Critical and reflective Process and results focussed

Outputs outcomes are key areas of work that enable desired

Who we will reach What we will do Where, when and how it will happen Targets and numbers to be reached Frequency of activities Standards to be achieved Benchmarks

Outcomes are cumulative changes Short term

Gains in knowledge, skills, awareness Changes in attitudes Changes in individual and group behaviour – social action

Medium Term

Organisational development Change in local situation and circumstances Change in policy, decision making, practice

Long term

Social change – health, education, civic, environment...

I N D I C A T O R S Evidence informs all aspects of the logic model

Taking ideas from social scientific research, literature, practice wisdom, policy and consultation processes, to inform understandings of problems, situations and issues, as well as ideas about work that can enable desired outcomes and ways of monitoring and evaluating the work 22 22

Intentions Organisational purpose

To develop young people personally, socially, educationally and recreationally in a high-quality, safe environment that respects its users.

Analysis (includes consultation)

Issues to be addressed: 1. Young people’s psycho-social development 2. Social change 3. Sexual health

Strategies

1. Adventure activities 2. Social action project 3. Information, advice, training

Sample logic model

Monitoring and Evaluation Implementation Resources Activities Inputs (matched to strategies)

Staff Volunteers Funding Buildings Technology Partners

Process

Value led and educative Dialogical and empowering Deliberative and responsive Critical and reflective Process and results focussed

Outputs Adventure:

8 linked weekend challenge courses over 18 months (includes personal and group learning inventory) for 20 participants

Social action project:

Over 12 months on an issue of importance to young people Series of monthly films, talks and discussions on social issues

Information, advice, training:

Trial (x2) Teen Talk programme (2 evenings per week for 2-3 hours, over 6 weeks) Plan for further work

Results Short Medium Long term Outcomes

• • Participants are more ready to take on new and more diverse experiences (S) Enhanced interpersonal skills (M) • Learning skills acquired and transferred to other areas of life: work, family life, politics (LT) • • Increased involvement in decision making processes (S) More informed decision-making (M) • Local services for young people are more user friendly (LT) • • More aware of harmful behaviours to self and others (S) Risk factors reduced (M) • Safer sexual practices (LT)

Evidence

Challenges of an evidence informed approach

•

Youth workers

•

Managers

•

Policy makers

•

Researchers

Challenge for youth workers

• • • •

Assumptions underlie much of what we do Faulty assumptions are often the reason for poor results – e.g. ‘process is all there is’ Evidence helps us make our assumptions explicit so that we can check and test them: espoused versus theory-in-use Reflective practice (Schön, 1983)

http://www.infed.org/thinkers/et-schon.htm

Challenge for line managers

• • • • •

Trust versus command and control Monitoring and reporting versus support, training and development Creating spaces for reflection Promoting a culture of evidence Underpinning own practice with evidence

Fixsen, D. et al (2011)

http://www.implementationconference.org/materials/plenary

The challenge for policy makers

Performance Assessment Training Selection Coaching Systems Intervention Facilitative Administration

Core Implementation Components

Leadership

Decision Support Data System Technical Adaptive

Challenge for researchers

• • • •

Providing policy makers with usable ideas and information that addresses current imperatives Developing user friendly frameworks and tools to evaluate the process and the outcomes Collaborating with practitioners in addressing pressing issues and problems Offering a critical and objective view that addresses the realities of practice

Wisconsin templates Routes to evidence in youth work?

ActKnowledge

Information about theories of change

SAMSHA resources Learning for sustainability Centre for Effective Services AECF Guide

Information about logic models

Blueprint model programmes CyferNet Search University of Arizona Campbell Collaboration Wisconsin online

Information about programmes in Ireland and Northern Ireland

Archways CDI Tallaght Together 4 all youngballymun Lifestart Foundation

Information about evidence-based and informed programmes

AECF Community Change EPPI Centre Preparing for Life

Information about evidence informed approaches

PROSPER Partnerships Evidence Network W.K. Kellog Foundation Centre for Evidence-Based Intervention

John Bamber Principal Fellow – Community Development Centre for Effective Services

[email protected]

0035314160513 www.effectiveservices.org