Transcript Part process and part outcomes evaluation of the World

by Eleanor Hazell
24 June 2010





Background
Evaluation design & methodology
WCA leadership programme theory
Evaluation findings
Recommendations to WCA
South African NPO, founded 2002
Vision –to develop the minds, hearts and spirits of people
for life success
Mission -to empower people for success through character
building and values based life skills training
Strategies
1)
Life Skills Training
2)
Leadership training for emerging leaders
3)
Mentoring & Life Coaching
4)
Networking students with people of influence and
opportunities
5)
Imparting positive values through training and modelling
Target groups

◦ unemployed youth,
◦ community leaders,
◦ high school students






Life Skills & Leadership programmes initiated
2002 for unemployed youth & community leaders
Programmes for high school students initiated in
2003
Life Skills =4 week course run in communities
Leadership = 11 week course run at WCA
headquarters, with outreach in communities
Life Skills provides students for leadership;
leadership students support Life Skills during
outreach
68 leadership courses run & 1930 students
graduated since August 2002.





Plethora of leadership programmes, wide range of
fields, different target groups
Considered difficult to evaluate;
◦ Individuals participate, but change is expected on broader
levels (e.g. organisational/community/society…)
◦ Links between change on different levels not well established
◦ Difficult to evaluate change on organisational/community/
society levels
◦ Influenced by external factors difficult to control
◦ Leadership = journey/process which develops/deepens over
time
Few leadership programmes have been rigorously
evaluated;
Few leadership evaluation tools exist
Most leadership evaluations done in USA
Nov-Dec 2009
 Gather background information
 Document review
 Meetings with WCA
 Develop logic model
 Design evaluation
Dec 2009
 Collect data (observation & interviews)
Jan-April 2009
 Transcription & data analysis
May-July 2009
 Report writing & presentation
A part process & part outcomes evaluation of the WCA leadership programme for
unemployed youth
Process evaluation will investigate:
 The number of courses run
 Number of students enrolled
 Where students come from
 The completion rate
 Course materials and
 Course structure
To uncover who is benefitting from the course, what students think about the
structure of the course and whether the course is being implemented as planned
The outcomes evaluation will investigate:
 The extent to which course facilitators model good/positive behaviours
 The extent to which students are able to describe and demonstrate improved
leadership skills in identified areas at the end of the programme
 Which behaviours/changes are difficult to sustain and why
 What graduates go on to do afterwards
 Significant changes which have occurred in graduates lives, which they attribute to
the leadership programme.








How many leadership courses are being run? How many students are
enrolled and how many complete the course? (Document review)
Where do the leadership students come from? How did they hear about
the programme? Have they completed the life skills course? (Interviews)
Did life skills ‘lay the foundation’ for the leadership programme? What
are the key differences between the life skills and leadership
programme’s? What components of the leadership programme do
students like/dislike? (Interviews & observation)
Is the leadership programme being implemented as planned? Residential
and practical components? Is the course being effectively monitored and
evaluated? (Document review, interviews and observation)
Are particular problems being encountered or specific barriers
experienced in the implementation? (Interviews)
Are the course facilitators (former leadership students) modelling
good/positive behaviours? (Interviews & observation)
In what ways are leadership students developing/improving leadership
skills in defined key areas (see logic model)? (Interviews)
What significant changes have occurred in leadership students which
they attribute to the leadership programme? (Interviews)
Interviews with key informants (4) and
leadership programme students (6)
Key informants
◦ WCA programmes manager
◦ WCA leadership programme manager
◦ WCA leadership programme co-ordinators (x2)
Leadership programme students selected at
random
5 males, 5 females; aged 20-37
Goal

To develop young leaders who will contribute positively to their
organisations, communities, society, countries, and change the world.
Objectives:
1.
To develop the hearts, minds and spirits of students from
disadvantaged KZN communities for life success (focus = individual).
2.
To develop and enhance the leadership skills, effectiveness and
impact of existing and potential leaders from impoverished KZN
communities (focus = individual impacting on
organisation/community/society).
3.
To expand the reach of WCA by developing a leadership programme
model, promoting and sharing it widely, and developing strategic
alliances in KZN, South Africa and internationally (focus = expand
reach and impact).
Target group(s):
 Primary = Unemployed youth (18-35) from disadvantaged communities
in eThekwini, Pietermaritzberg, Ugu district and Northern Zululand.
 Secondary – Grassroots community leaders.
 Long term - Impoverished KZN communities.
Summary
If disadvantaged youth complete WCA’s life skills (foundation) programme
and apply what they have learned they will develop emotionally, mentally
and spiritually such that their world within is changed resulting in more
positive personal outcomes (interpersonal relationships, academic and
professional success).
If they go on to complete WCA’s leadership (advanced) programme
and they consolidate and apply their learning practically, they will develop
skills and competencies which will enable them to develop as leaders and
impact positively on their organisations, communities and change the world
around them.
If graduates continue on their leadership journey, accept responsibility,
consolidate and apply their leadership skills and develop new ones, they will
motivate and inspire other disadvantaged youth, create opportunities and
contribute to the upliftment (economically, socially, psychologically and
spiritually) of disadvantaged communities.
See Logic Model handout
Evaluation findings
Jan-Dec 2009
Aug 2002-Dec 2009
Leadership courses (high
school)
6
32
Leadership courses
(unemployed youth)
7
36
Leadership courses All
13
68
Leadership graduates (high
school)
101
1014
Leadership graduates
(unemployed youth)
147
916
Leadership graduates (All)
248
1930
1) A range of KZN communities: South Coast, eThekwini, North Zululand
2) Students come from disadvantaged backgrounds and face an array of inter-related challenges:
Poverty: “I’m coming from the family that is not good financially” (R1)
“I wasn’t having enough money to pay for exams, I wasn’t have enough money for uniform and I just
drop out [of school]” (R4).
Unemployment: “Since 2003 I was searching for a job” (R6).
Crime/prison: “I learnt different ways of doing crime, and it was terrible. So I went to prison for 3
months . That was for stealing, for shop-lifting” (R7).
Drug & alcohol abuse: “I start thinking in others ways, start drinking alcohol, drug addiction, and
having boyfriends so that I can have money” (R3).
Teenage pregnancy: “ I end up even having a child at the age of 16 years” (R3).
Relationships/peer pressure: “As the youth we are involved in different relationships which are not
good, like we are involved with love relationships which we are not ready or mature enough to be
involved in and having friends who are doing wrong things, but we involve ourselves for the sake of
having a friend, and don’t know why”(R1).
“I drop out of school, and I listen to my friends rather than my parents” (R3).
Dropping out of school:
Family problems/loss of parents: “In my heart I was having the pain, because I lost my parent[s], my
mother she was sick and my father he was poked [stabbed]” (R4).
“I was looking for my father actually, he left me and my mum when I was young, so I grow up didn’t
know him in the age of 13 or 14 years, then I try to want to get connected with my father, cos the
situation at my mother’s house was not good” (R3).
Rape: “When I was young I was raped for three guys… afterwards when I went through to test and I test
HIV positive” (R4).
Depression/attempted suicide: “I tried to kill myself, I think about three times, I went to the road, I was
waiting for the car to crush me, but there was no car come” (R4).




All respondents completed life skills
(students in 2009, staff between 2002-2006)
8/9 found out about leadership via LS course
1/9 was recruited for the leadership course
Respondents found out about LS course via
◦ Word of mouth (8/9)
◦ Posters/pamphlets (4/9)


6/6 student respondents passed an interview
to attend leadership
4/6 student respondents paid R500, 1/6 paid
R200, 1/6 received a full scholarship
Student perspectives:
“It is like it’s the first step in changing life… [the] first step of a person who wants to change”
(R1).
“Life Skills developed, made me to see my vision, where I am going, where I am standing…
and it make them to be clear. If I want to be a change, [I must first make] the change my life
so that I can change other people’s life. That is when I started to be, and I would like to be a
leader that will change other people’s lives” (R2).
[During Life Skills] I’ve noticed that I was doing the wrong thing, like, like amabad
relationships, the way I talk with others, the way, when I done something wrong, I tend to
point fingers to someone… its helped me a lot, like to realise that I am responsible for my life”
(R6).
Staff perspectives:
‘”I think Life Skills is, does, provide a foundation, in that people come to the leadership with
an understanding of their identity, who they are and their vision and their purpose, what they
feel that they want to do in their lives… leadership then will build on top of what Life Skills
has already laid “ (R7).
“[Life Skills] more grills and builds up self-esteem, confidence. So when a person comes to a
stage of discovery and self-realisation, that’s when a person is more equipped and more
suited to, enrol in the leadership, because we believe that you cannot lead people if you are
still broken yourself, so you need to deal with yourself first, because anything that you say has
to emanate, emanate within you. It will be something that you have already gone through
yourself, the transition that happened to you” (R9).
Student perspectives:
Life Skills highlights; leadership makes things clearer and goes deeper
“Life Skills they just highlight things, they just take you step-by-step, but when you come to
leadership, they go deeper and deeper… they need you to lose your mind, to think, not just think
and make a decision, but think, think and think again, until you finally realise that it’s a decision
that you really want” (R5).
“Leadership is different, they are touching the real issues that can cause a human to not to succeed
in life” R6
Staff perspectives:
Leadership is community conscious
“Leadership is more community conscious, and Life Skills is more personal. At a Life Skills level, we
teach healing of the past, so that people will break free from whatever bondage or imprisonment
that they are in, for life success, and in leadership, what we do is teach healing of the past, with the
motive of them, imparting that to other people“ (R8).
“We are introducing them to a world that says, even if yes, we are saying take care of yourself, but
more so, are the people around you benefitting from what you are?... saying, I will be independent,
but again my independency is an interdependent kind of a dependency because I am, so therefore
people are, and they are, so therefore I am.” (R9),
Students are introduced to God
“Its where we introduce people to God, in leadership… you can have a vision, but if God is not part
of that vision, if God didn’t give you that vision, that means, there isn’t much of a vision, you don’t
have a vision, because you cannot have sight if God didn’t give you sight” (R9).



Overall, students have very positive things to say about the
leadership programme & WCA
Students liked the course structure (residential, practical,
feedback)
Highlights include:
◦
◦
◦
◦
◦
◦

Healing of the past
Motivational speakers
Developing communication skills
Varied facilitation techniques
Making friends/building a support network
Outreach
Dislikes were: (*see also problems/issues & suggested
improvements)
◦ Students have personal issues which affect participation*
◦ Too long spent watching movies
◦ Need for computers & computer skills
Residential (Phase 1, 4 weeks)

Facilitators use a range of techniques to design/tailor the course to
students needs

Motivational speakers are a key resource, but there are challenges
finding/retaining high quality speakers

Other resources = books, movies, the Bible, CD’s, students & life; there is
no manual

Leadership courses are run back-to-back throughout the year

Staff shortages mean the programme staff are over-stretched and can
become stressed
“This year, leadership was my secondary role. So if the students go out, I also go
out, I co-ordinate [Life Skills], but over the past couple of months, it became my
only role, because of shortages of staff… its either I’m co-ordinating [Life
Skills], if I’m not co-ordinating I’m a speaker in the Life Skills and still facilitating
another leadership, because when one leadership goes, another comes. So
there’s always people here for leadership… Its very hectic, it cracks your mind,
you come during the week I won’t be smiling, as much as I like to… Today, I’ve
gone already, I’ve gone to speak, when I come back at night I have to go and
facilitate the leadership and I have to prepare for the class tomorrow, so it really
becomes, really hectic” [R9]
Outreach (Phase 2, 6 weeks)

Students apply leadership skills practically via volunteering

Overall, students & staff find this component very worthwhile

Many students assist with Life Skills facilitation, when LS is not
running they find their own placements

Students were volunteering in a wide range of places (YMCA,
hospital, pharmacy company, KZN youth chamber of commerce)

Experiences varied; most students gained valuable skills; one
placement was more linked to professional than leadership
development

WCA staff were not able to visit all students on outreach

2/6 volunteered for less than 6 weeks (shortest = 3 weeks) due
to challenges finding placements/other issues

Students face financial & domestic pressures whilst on outreach,
which helps to develop their resilience

2/3 females encountered sexual harassment
Feedback (Phase 3, 1 week)

Students feed back/reflect on outreach and the course more
generally

Facilitators draw the course together & motivate the students

Students graduate

Feedback was 2 days not 1 week due to WCA Christmas closure

One staff member critiqued feedback:
“
You will find cases like this week, this week because we are closing
down, they only came for 2 days, and there are cases where we have
high schools, and they also come for two days also. So others will
come for the rest of the week, which also is an indication that its, it’s
a haphazard process, its not a well-structured process where first day
this, and second day this happens, but its just more, its just activities
that are not well structured” (R7).

M&E takes the form of:
◦ Course evaluation for the lecture phase(???)
◦ Student feedback forms(???)
◦ Feedback phase (1 week)

Students could not describe M&E
processes/tools other than the feedback
phase
Problem/Issue
Suggested improvement
Students have personal issues which affect
their participation (x2)
Before/during first week, facilitators to have
one2one’s with students, to discuss
issues/problems which may affect their
participation (x2)
Some students need professional counselling
Course duration is not sufficient
(x4)
Longer course (residential & outreach
components) (x4)
Restructuring: Life Skills II & Leadership
(advanced) (x1)
Need a course manual (x2)
Develop a course manual
Students need more support/guidance with
outreach (All)
Course should be assessed
More time needed between courses (x2)
Need for structured follow up (x2)
Recruitment & retention of speakers (x2)
Need more staff (x2)
Dedicated staff member to focus on this
Identified key areas were (see logic model):
 Hard work/diligence √
 Personal identity/taking a stand √
 Being a positive role model/exercising self control √
 Accepting responsibility √
 Leading others √
 Active in community/social issues √
 Communication skills √
 Empowering others √
Additional ‘areas’ which students/staff had developed
in were identified via the interviews:
 Being a servant leader √
 Forgiveness √
 Critical thinking √
 Computer skills √
Hard work/diligence
“Sometimes I don’t even go to my lunch, my tea break, just serving patients.
When its one o’clock they go to lunch, although they see patients’ sitting on
the bench… the manager, the supervisor find me only one, for the whole
hour, don’t go to the lunch just clicking those patients” (R3)
“Sometimes I have to deliberately, you know, do stuff, do hard work,
because I don’t want to be a hypocrite … I though my personality, I’m like ,
‘oh lets go have fun’ you know, ‘lets wash dishes, arghh’ . So you can
imagine the effort I had to put to washing those dishes. You know, so hard
work and diligence, I think I’ve grown tremendously in those areas” (R8)
Personal identity/taking a stand
“Even now I can speak to the people about it [the rape], and even the person
speak about it I didn’t care, because I know now who I am, I really know who
I am” (R4).
“I have a list of some, my printed list, values. So it helped me, cos before I
wasn’t having and I didn’t know what values is” (R5).
“Standing on, for what I believe in… it can be positive, it can be negative as
well… I believe that the people who left, who resigned within the
department, it was because of that, there are things that I couldn’t
compromise on, I said ‘this is it, if you don’t take this, you can see what you
can do’” (R10).
Being a positive role model/exercising self control
“It was Friday, I used to go to the tavern, chill with friends, drinking. But on this day I go to the tavern,
I sit down, I buy a cold drink, I put it on the table, I drank it, and they were so surprised, as ‘Hey…is
not drinking alcohol, he is drinking Coca Cola’, from alcohol to Coca Cola [laughs], so they were
surprised, by looking at me. I didn’t say ‘hey my friend, don’t do this, do that’, but by looking at me,
they see that I have changed” (R1)
“I was having five girlfriends and I slept with them… I decided that I have to go to the clinic, to know
my status, and I don’t care, even if I’m positive, I don’t care, but at least I will know my status and
know where I stand for the sake of other people. Then I go to the clinic and get tested, luckily I found
out that I am negative. Then I started that I have to be responsible. I don’t have to do like having
girlfriends. Then I decided that I have to have a right person at the right time “ (R1).
“I was smoking drugs before, so I had a lot of friends who were smoking, as I am talking, they are still
smoking. So as I was back at home, I realised that I have take responsibility for myself… I have
decided not to take drugs like I was doing before” (R6).
Accepting responsibility
“My grandmother start send me to go and make grocery. Cos [previously] they were no more longer
trusting me, they give me money for grocery, I would go drink alcohol and I would come back on
Sunday, they send me on Friday, you know. And now they start trusting me” (R3).
“It was a huge change, you know, from moving from the street, into a life where you now have
guidelines and boundaries and completely are accountable to different people… from a situation
where I was blaming and I was condemning and criticising and judging, where I actually saw my life
and the problems that I had as, you know, as maybe the issue with my parents, and with my
background. To a situation where I started to accept responsibility for my own actions, and said ‘my
life is like this because I have messed up’ and so I need to work and put things back in place” (R7)
Leading others
“They put me in charge [during outreach]… cos I was having a long time there
then I was knowing everything how is operating, where is the file, it goes here
and stuff, and how much money they pay per patients and those stuff. So yeah, I
becoming a leader, you know leading people in a good way, they say I was
leading people by doing it” (R2).
“I have to be the one that motivates them, that ok, lets just go and attend, lets
just make a meeting in this week, so that we can decide what we are going to
do. I have to be the one who push them to do things” (R5).
“Its only next year that I’m gonna take a formal role of leadership… I think I have
the ability and the capacity to do that, but I’ve never practised that formally”
(R8).
Active in community/social issues
““We didn’t have a [Life Skills] co-ordinator for our area. So I thought, this would
be a great opportunity for me to be involved in my community… I have been
there for a long time, just sitting not doing anything, seeing the things that are
going wrong, but still not doing anything about them… so that’s what motivated
me to come and volunteer” (R9).
Being a servant leader
“I believe in putting others before myself. Not neglecting myself, but I believe other people
come first... we were talking about the queues when we go to fetch food, like ‘which one do
you prefer to be in? The first one in the line or the last one in the line?’, and the others were
debating, ‘why would you be the first one, [corrects], the last one in the line? you won’t get
the best plate’, but I believe that being the last one in the line means there is a level of
sacrifice, even if I don’t get the best, but if the people are happy, that’s more so what I’m
about” (R9).
Communication skills
“Sis Precious said , ‘now it is an outreach report who will go first?’ I started, which means
that I am not afraid anymore to be in front of people. It wouldn’t be a problem at anytime
now to be in front of people talking” (R1).
“I was a quiet guy before, but now I have starting to talk with some others people, like
sharing ideas, like what we are going to do, like in future, what you like to do in future” (R6).
“I can say the way how I can communicate, like even at home, like if I have a problem with
my sister, we tend to, we like to fight, now I just tell him[her] ‘I don’t like that, just stop
that’. If it doesn’t understand, I’m just leaving him like that… the communication skills
helped me to understand others” (R6).
“When I first came here, I couldn’t even speak, you know, I struggled to put words together
and not to mention skills like computer skills and different things like facilitation, coordination, administration, I did all of that, within the organisation, slowly, you know” (R7).
Empowering others
“That’s my job. I, had to grow in that area… I’ve developed
tremendously as well, because I practise it every day. I have to break
down mentalities and uplift and motivate and encourage at the same
time, you know” (R8).
Forgiveness/Healing the past
“I learn even to forgive my mother for what she did to me. I learn even
to forgive my father for what he did to me, although he has already
passed away… I also learn even to forgive myself, for every bad things I
have made” (R3).
“I was having that guilty and I was having that grudge for them, I was
holding everything in my heart. But from Life Skills I see that I mustn’t
hold things for other people, because God create me.. ok that thing
happen in those days, it hurts me very well, because I think I am a
strong women now, I can stand for myself of what they did to me” (R4).
Leadership area
Student s citing
examples
Staff citing
examples
Total citing
examples
Identity/taking a stand
6
4
10 (All)
Communication
6
4
10 (All)
Being a positive role model
5
2
7
Active in community/social issues
2
4 (All)
6
Leading others
4
1
5
Servant leadership
4
1
5
Accepting responsibility
3
2
5
Hard work/diligence
2
2
4
Empowering others
2
1
3
Forgiveness
2
Critical thinking
NO CHANGE
Students citing
no change
Communication (report writing)
2
1
1
Staff citing
no change
Total citing
no change
2
2
Active in community/social issues
2
2
Leading others
1
1
Empowering others
1
1





Development of leadership skills began
during LS
Greatest improvement in Communication
skills, identity/taking a stand, being a
positive role model
Students found it difficult to provide
examples of improvements in a number of
areas
No change was cited in some areas
(community/social issues, leading &
empowering others)
Shallow vs deep examples
Students generally very positive about leadership facilitators & WCA staff
Approachable:
“maybe during his or her lunch time and you said ‘I have a problem here’ and then
they not just say ‘its my lunch time, I’ll see you later’ they just stay calm and discuss
(R5).
“they don’t look down to[on] you, like where you are coming from , or what kind of
people, person you are, they just came here to help. It seems like they are coming, we
are coming from the same background… they are great people” (R6).
Good role-models
“they preach what they practise” (R1).
“they didn’t tell me things if they don’t do… they are people that have made a change
to them, that’s why, those people have done a good job. And even the staff that are
volunteers here are a good person” (R2).
“I see anything wrong. I see the leadership, I see the leadership with them and I see
the way they do things and I just wish to do like them” (R3).
Minor complaints (contradictory)
◦
◦
Facilitators are too friendly with the students (x1)
Facilitator too formal (x1)
◦
Staff unfriendly (have not been through Life Skills)
Complaints about other WCA staff








WCA staff tended to have higher standards/judge
themselves/ colleagues more harshly
Change is a long process (behaviour change takes
time)
Emotions/misunderstandings/conflicts intolerance
arise in the workplace
Leadership facilitators highly regarded amongst WCA
staff
Facilitators face a steep learning curve (education &
skills)
Living & working together places WCA staff under
intense pressure
As does the high workload and
Staff shortages









Self-acceptance
Getting my life back on track
Attitude & behaviour change (alcohol, drugs,
sleeping around) & foregiveness
Direction & values
Accepting responsibility
Becoming accountable
Spiritual consciousness
Developing a (self) culture of learning
Searching for God
Holding/moving into leadership positions
 Starting a co-operative (student)
 Selected to participate in youth parliament in
Cape Town (student)
 Moving into WCA management (staff x2)
 WCA ambassador in USA (staff)
Future plans
 Family life
 Start a CBO (x2)
 Community/social initiative (x2)
 Studying (x3)
 Volunteering (x4)
 Career/business plans (x4)




Initiate a proper course evaluation
Pick a few key indicators from the Logic
Model & develop a simple M&E framework.
Monitor/track number of students enrolled as
well as number of students completing WCA
courses; calculate drop-out rate and identify
key reasons why students drop-out.
Periodic follow-up with 10% of course
graduates (selected randomly)





Apply cost/funding criteria fairly (students eligible
to apply for full/partial scholarship)
Develop a course framework & manual
Develop ‘guidelines’ for outreach; approve
placements; establish/nurture relationships with
suitable organisations; provide greater support
Consider assessment: Did students complete
outreach? Can they apply learning?
Consider introducing an advanced leadership
programme targeting community leaders; current
programme does not target them & is making less
impact in several key areas (leading, empowering,
community/social issues)



Consider dedicated position, focused on
initiating & nurturing relationships with
speakers
Leadership programme needs to be properly
staffed
Staff need time off inbetween courses to
prepare