Transcript Lessons from the NGO Sector to the Public Sector

U. Johnson and D. Manicom



Evaluators must contribute towards determining whether the promised social
improvements are actually delivered (Lipsey 2001: 326).
Quite recent in the South African public sector but has been happening in the
NGO sector for a long time. The paper presents some lessons from the NGO
sector for the South African public sector, as gathered from a case-study of
the Centre for Criminal Justice (CCJ) on the purposes and uses of Monitoring
and Evaluation within NGOs.
The chairperson of the Public Service Commission (PSC), Mr. Ben Mthembu
noted the value of M&E in the South African Public Service:
 The PSC is encouraged by the emergence of M&E role players in the Public Service
… the Department of Performance Monitoring and Evaluation in the Presidency.
Government has also adopted an outcomes-based approach which has a strong
emphasis on the value of M&E in achieving service delivery (Mthembu 2012: 2).
.



CCJ) ‘operates as an integrated development, outreach
and research entity within the Faculty of Law on the
Pietermaritzburg campus of the University of KwaZuluNatal’ (CCJ Document 4 2009: 2).
CCJ began by initially conducting research into the
limitations of South Africa’s criminal justice system; and
went on to implement an outreach programme that
provides access to justice to poor people who could not
access the justice system in rural communities within
KwaZulu-Natal.
CCJ is experienced in regularly conducting various types of
monitoring and evaluations to continually assess the
operation, progress and effectiveness of the outreach
programme that it implements (CCJ Document 3 2001).
Monitoring
 De Coning et. al (2008: 15) ‘monitoring focuses on ongoing surveillance or assessment of an initiative’. A
descriptive report of the process of evidence collection
through measurement and systematic observation, regular
record-keeping or planned qualitative study generally
integrated into everyday programmes to ensure regular
collection of both qualitative and quantitative data (Munce
2005: 6; Beck 2003: 12).
Evaluation
 A ‘systematic assessment of the operation and/or outcomes
of a program or policy, compared to a set of explicit or
implicit standards, as a means of contributing to the
improvement of the program or policy’ (Weiss 1998: 4).




Public and non-profit programmes cannot be measured based on demand or
customer satisfaction alone because they are often the only services available in
certain areas, or the only ones that are free (Weiss 1998: 6, 8).
Resource deficits in non-profit and public agencies, policy-makers and
programme managers are forced to cancel some programmes in order to provide
sufficient funds to launch and maintain others. M&E facilitates such decisions.
(Worthan, Sanders & Fitzpatrick 1997: 4; Estrella & Gaventa 2008: 3).
M&E is growing in popularity and practice and target audiences new questions
about M&E are emerging: Why is there confusion about what evaluations are in
theory and practice? (Estrella & Gaventa 2008: 4 & 12, Valadez & Bamberger
(1994: 4-5)? Why are there discrepancies in the purposes and uses of evaluations
among organizations that utilize them (Palumbo & Hallett 1998: 39-43)? Why the
high rates of failure recorded of M&E conducted by organizations, many
evaluations have been seen not to answer the questions that they were meant to
answer (Ambrose 2010: n.p.).
More people care about whether or not programmes are doing what they are
supposed to; and getting the expected results, how less money could be spent
on programmes or how better results could be achieved or, better still, how to
achieve both of these objectives in programme implementation (Weiss 1998: 6;
Patton 1997: 197). Weiss (1998: 6) these concerns can be addressed through
improved M&E information. This raises an important question for both scholars
and M&E practitioners: how do we make M&E better?



Why are organizations undertaking M&E?
How do Organizations use M&E Findings?
What Challenges do Organizations Face in
Using M&E for Programme Implementation?
Research Design
 Qualitative methodology was used to investigate and understand the phenomenon
within its real life context; not to generalize to some theoretical population
(Babbie & Mouton 2002: 270). A case study was used in responseto Brunner’s
(2004: 103) argument that M&E can be improved if it is made sensitive to each
specific situational context.
Sampling
 A purposive sample of seven respondents, including: the director of the
organization, two management staff, three of the 14 implementers of the
programme and the external evaluator.
Data Collection
 Primary data was gathered through in-depth, semi-structured interviews.
Secondary data was also gathered through the organization’s monitoring and
evaluation reports.
Data Analysis Methods

A thematic content analysis was used to analyse the data. Common themes were
identified from the respondents in relation to research aims and objectives. Data
was analyzed to describe the perspectives of the different respondents, which are
obviously impacted by their beliefs, histories, contexts, roles and level of
participation in the M&E process (Babbie & Mouton 2002: 271).



The main themes that emerged from the
analysis included:
The contextual issues of assumptions,
understandings of theory and practice; and how
these affect the actual practice of M&E within
the organization.
M&E practices for the purpose of learning
lessons for improvement of the the programme
and the M&E system, strengths and
Challenges experienced in the practice of M&E.



The study revealed that the purposes of M&E are determined by the
stakeholders who are interested in using M&E to answer questions about the
programme.
The conceptualization of M&E determines the different vested interest that
stakeholders have. These diverse interests often fit into one of the following
themes: generating knowledge, making judgments of merit or worth, to
improve performance and for accountability purposes (Babbie & Mouton
2002: 337, Rubbin 1995: 30).
Weiss (1998: 4) identifies that the people who have expectations about the
programme being evaluated determine its purpose. These stakeholders in
CCJ include the director, who wants to ensure that the programme is alive,
relevant and progressive, management staff, who want M&E to help them
learn lessons and make informed decisions/judgments about the
programme; programme implementation staff: who want M&E to help them
improve performance, funders, who want to know whether or not their
investment in the programme is yielding useful results, beneficiaries, who
are interested in how better the programmes can meet their needs, and
other institutional partners such as the police, the law courts and the South
African legislature, who are concerned with how effective their policies are.

M&E within CCJ is underpinned by the fact that the organization’s
outreach programme was deemed necessary after a research was
conducted in the community on access to justice as the quote below
illustrates:
◦ I think they started monitoring even before they opened our offices because our
offices were a result of monitoring [research] that they did in the communities… so
our offices was born through that (Johnson 2011: 55).

The above response from a community member, who has been trained
as a paralegal to facilitate access to justice for other community
members indicates how M&E research has been used to serve
programme development purposes, which responds to the voices of the
people for which it serves. A related response was as follows:
◦ CCJ was originally set up for research, and I think as part of the accumulation and
collection of information, it [M&E] had to be done in order to inform some of the
data as to how things go along… it was from the inception. It became a necessary
tool to meet the end (Johnson 2011: 55).

An implementation staff describes a case from
CCJ:
◦ Recently ,… the results of the cases that we reported,
mainly on customary marriages or civil marriages, the
corruption that is there,… CCJ, together with the… [a
professor in the] Law School compiled a... white paper…
(from) cases in the challenges that are faced by women in
their marriages… it’s got an impact in amending some of
the laws (Johnson 2011: 55).

M&E thus becomes valuable when it is “actionoriented” because, it can serve as a preprogramme planning or needs assessment (Rubin
1995: 33; Estrella & Gaventa 2008: 22); as well as
for policy adjustment when it can identify
problems timeously.


Stakeholders, for example management, the director and the funders, are using M&E to
answer their questions and to meet their needs concerning the programme. M&E is used to
improve the programme and provide relevant capacity for implementers to meet identified
needs in order to improve the programme.
Different stakeholders’ conception of how M&E is useful for learning lessons affects their
contribution towards it and the value that they attach to M&E:
◦
I think it must be useful for the head office to know what is going on and whether the community is
receiving the benefit that the programme had planned; whether it is necessary for the offices to
continue to be there (R_7) (Johnson 2011: 56).
◦
I think it is a way of helping them [the head office] to improve where they had to improve; and where
they are supposed to improve! (R_5) (Johnson 2011: 56).


The above responses do not attribute any direct benefit of the programme to the
implementation staffs, as it finds M&E to be only beneficial to the head office (management)
staffs. The next view from a management staff collaborates the views of the implementation
staff:
◦

Monitoring...[and] evaluation... results ... give the organization an understanding of what exactly is
happening, and what exactly needs to be changed, what exactly are the challenges, what is the way
forward? Or what are the new decisions to be taken? Do we have to shift the focus? Or we need to apply
more... or we need to change other things? Or what are the new trends in the community based on the
reports that are being provided... it does help, and then the organization ... grows (R_3) (Johnson 2011:
56).
It is quite clear that the above view is from someone who wants M&E for learning lessons for
proper management decision-making. The director, who makes decisions on how to correct
ineffective practices said:
◦
We don’t want to run a programme that people don’t want; we don’t want to run a programme that is
not practical - for the people that it is not meant to benefit - so it is very important that we have
monitoring and evaluation system to track progress! To track effectiveness! Relevance is very important
as well!! Efficiency is very important!!! Impact is very important!!! ...You are interested in the outcome
of your investment, isn’t it? To know what is working and what is not working… (R_1) (Johnson 2011:
56).

Funders tend to impose their needs on the M&E process, to the detriment of
other stakeholders, who also have interests to be met by M&E because of their
financial power in the process (Bamberger 1989: 388-391). In the CCJ, however,
findings reveal that M&E is primarily for the benefit of the programme and
funders are only a secondary consideration.
◦ When you do a proposal for the organization, you just say: this is what I do, and then the funder
will say: “ok, we are interested in this aspect and this aspect...” the funder ...would need reports
to ...know that the organization is implementing... and directed at its intended target group ...
they would also require you... to give monitoring report and evaluation report and say ... “now
has the organization achieved what it had intended to?” (R_3) (Johnson 2011: 59).

This approach allows CCJ the flexibility to attend to the needs and new trends
arising in the communities, while at the same time meeting the funding
requirements. They focus on funders who are still interested in whatever services
are being provided in the communities. Proper documentation is identified as the
key to this kind of success and one of the respondents noted:
◦ ... with documentation you can show the work you are doing. Obviously donors are very
important, not to please but to show; to demonstrate that the work you are doing is valuable.
Documentation wasn’t there, so our first documentation was monitoring and evaluation. ...So
we didn’t just prescribe, but described that in terms that would monitor and evaluate, which we
put together back at that time. We knew that we needed to do it, but ... donors really appreciate
that effort to evaluate! (R_2) (Johnson 2011: 59).
The sum of this section is that NGOs, some donors and international agencies,
such as the World Bank and the United Nations Children Fund (UNICEF), assert
that the main ‘objectives of social development programmes should be to help
the indigenous communities or underprivileged groups (such as women, landless
laborers, ...) develop the organizational capacity and knowledge needed to
identify and satisfy their own needs’ (Valadez & Bamberger 1994: 9). Failing to do
so could be termed paternalistic.


Arguably, the limitation on the M&E framework
in both public and NGO sectors is that with an
overly structured top-down means of service
delivery, followed by a top-down evaluation
framework provides services and gathers data
without sufficient attention to voices of the
beneficiaries.
This case study indicates that such an
approach perpetuates dependency among
beneficiaries, rather than an approach that
would empower them with responsibility and
retain greater chances of agency and
sustainability

Lipsey (2001: 326) suggests, a bottom bottom-up approach to service provision
and policy implementation through M&E activities. This should create some
balancing of powers; while the local beneficiaries are not only given a voice, but
also given a sense of ownership of the development process that serves them. The
attempt to achieve this in CCJ is expressed by the quote below:
◦


before anything takes place CCJ will have a meeting and plan to “say this is what we want to
do”, and “this is what we want to achieve...” after meeting ... with the head office, then the
next meeting with all the co-ordinators (implementation staff) and table everything and say:
“... this is what we have identified through what has been happening, and we want to now
shift focus based on the information you’ve given us, so what do you guys think of this
information?” And then they (the coordinator) would give you their views; they would tell us
exactly what has changed, whether the new trend is what is happening ... saying what
changes they themselves want to see in their own communities and how. And then ...
would start planning for monitoring and evaluation based on the information provided by
the outreach staff - most of the time (R_3) (Johnson 2011: 63).
This concern lends itself to a consideration and understanding of M&E as social
processes through participatory approaches within organizations (Estrella &
Gaventa 2008: 37). This challenge prevails despite assertions by the Framework
for GWM&E (2007: 3) that to improve governance through M&E, the participation
of and inclusion of the marginalized and traditionally excluded members of the
communities in M&E processes must be ensured.

The case study on CCJ revealed that a major contributor to success of the M&E
within the organization is its continuous shifting of the focus of services delivered,
based on lessons learnt from the implementation field through interaction with
beneficiaries and enquiries on what their pressing needs really were. An example
of such change of focus in service provision in CCJ is described thus:
◦


...it [M&E] has helped CCJ to see that ... the rights that were abused are not only for women
and children; the ... boys or males are also being abused, so there is a change that is
happening in the community, so if we deal with the clients, we are not [no longer] dealing
with the women and the children only (R_6) (Johnson 2011: 57).
There is no denial that changes from M&E affect public service provision, the
emphasis here is on when it does happen: to what extent do these changes
represent the voice and needs of the beneficiaries, other than by-products of
researches aimed at answering pre-determined questions. If these questions are
not properly investigated through M&E, they will have implications for the public;
taking the form of service delivery protests; often related to demand for wage
increase, solidarity, dissatisfaction with high crime rate, resistance to government
policy, mobilising of the masses in sympathy with the oppressed and service
charges (Alexander 2013). These enduring recurrences of protest demonstration
from beneficiaries of public programmes and service providers is an indication
among other things that the problems arising from programme implementation
are not being identified beforehand and averted (The Presidency 2010: 8).

Nevertheless, there is no easy way of achieving participation, whether that be by
NGOs or the public sector, because there is a challenge in gathering information
from beneficiaries and in fostering participation as the quote below illustrates the
experience of CCJ:
◦


During the M&E, when we have to call the clients to the office… the community at large, …
if you call them for this interview, some would say, “this people want to make money, using
us”. Some would say, “what are we going to be getting after the interview?” … The director
makes sure that transport money is given to them (R_6) (Johnson 2011: 69).
The above quote, in itself, does not indicate that beneficiaries are aware of the
value of the exercise and the potential that it has for the services that they receive.
The question that arises here is concerned with how government M&E system can
include as vital among its stakeholders, the programme implementers and the
beneficiaries who constitute the majority of those protesting for service delivery
into the planning and designing of M&E. This would not only be empowering to
those community members involved, it would also create a sense of ownership,
responsibility and accountability among them in the development process. It
would be a way of responding to Munce’s (2005: 7-9) caution that M&E may not
contribute sufficiently to development outcomes due to lack of stakeholder
participation and responsiveness. As such, careful attention must also be paid to
Estrella and Gaventa’s (2008: 37) note that issues of power and conflicts of
interests can constrain the translating of participation into practice among the
different stakeholders.

In CCJ for instance, the implementation staffs are
trained on M&E, they contribute to questions for
M&E as they verify questions already formulated
by the management. This seems to be lacking in
the public sector as the system is so structured
and stringent about what information should be
gathered. The contribution of M&E to the training
of staff in CCJ is articulated by one of its
implementation staff as follows:
◦ I think it always helps them because when... - after
evaluation, they see that we are not doing okay in a
particular issue, they call us and train us (R_5) (Johnson
2011: 56).

The immediate needs of the beneficiaries are the enjoyment of the service provided by the social programme, and the immediate concerns of service
providers are the provision of these services. Trying to educate beneficiaries and service providers has proven to be a challenging endeavour, but success
in doing so promises to escalate the successes of development initiatives as it empowers these stakeholders to own the services provided and to be
accountable for it. Some means of dealing with these challenges are noted by the management of the organization as follows:

Coordinators just don’t like the issue of gathering data (R_2) (Johnson 2011: 66).



This awareness by the management staff led to an action which is described by the quotes below as a deliberate attempt by management staff to
continuously encourage:
◦
... in the past, they [implementation staff] did not understand the importance [of M&E]... their immediate concern is helping the people they see on a regular basis,
so documentation is something that is not primary, ... For them to buy-in, you need to guide them, you need to encourage them, ... when they begin to realize
that in fact this M&E is actually helping us to manage our work better - then they start getting good information, then they become happy. So you’ve got to allow them
time to build this system into their day-to-day activities. ... then they are happy because their work is professionally managed. They can track their cases when
clients come to the offices, they have given them numbers ... when the clients come back, where to get that specific file, they can find it in time, and then they are
able to update their case; which is very helpful later on! ... if they did not put systems in place, they would not know when did they meet this client and for what?
(R_1) (Johnson 2011: 68).


The next quote collaborates the value of the intervention described above:
◦
Some of the things which I think contributes more to the success of M&E are that they [implementation staff] don’t see it [M&E] as an event; they do it along the way
when they do their work. ... compile the information, ... they provide the information as you go along ... at regular intervals during the service provision for the
evaluation, rather than saying “ok let us go and monitor, let’s go and evaluate”, it is built into... the daily routine so that it is something that is continuous, ... it is not
just ...started and stopped (R_4) (Johnson 2011: 66).


Nevertheless, some implementation staff expressed the opinion that M&E is not an important aspect of their jobs. It was just something that had to be
done to meet the needs of the head office and other interested parties. Implementation staff sometimes felt that the process was distracting them from
their services. They referred to M&E as the domain of activity for the head office, which they only assist in data collection. If their contribution to the M&E
process is not voluntary, stakeholders would struggle to take part in the M&E process.
◦
... the challenges, I think will be... we are dealing with our daily work,... it interferes with what we are doing at that moment in our offices. We have to leave everything
and focus
on M&E. It is just that it interferes with our routine work at the offices (R_6) (Johnson 2011: 68).


This is one of the greatest difficulties that M&E poses to the implementation staff of the CCJ and to the organization as a whole. The specific response
that management applies to ensure continuous cooperation is described in the response below:
◦
Every now and then... explanations have got to be done so that people understand that it is necessary to do it, rather than just a burden on them. ... they say ‘oh too
many forms, too many information’ and all that, so you have to explain, we need this information in order to stay in employment (R_4) (Johnson 2011: 68).


These lessons can be summed using Brunner’s (2004: 103) argument that a context-sensitive approach to M&E might help to address these concerns.
This is an approach that pays attention to differences and changes in context to accommodate uncertainties and ambiguities arising from the human
factors central to a programme. These human factors are issues related to people in poor communities trusting in themselves and in outsiders who bring
in development aid. Brunner (2004: 103) emphasizes that developing such a system must harvest from the experience of practitioners in the field.

A rationale for drawing M&E related lessons from the NGO sector to the public sector in South Africa is based on the
fact that M&E activities in South Africa was first established in the NGO sectors through donor-related requirements in
the early days of South African democracy (Mouton 2010: 73). This is vital, considering Tuckermann’s (2007: 21-22 &
27) argument that understanding the challenges and strengths of M&E should help organizations to better utilize M&E
in their programme implementation. This happens if an organization harnesses the identified strengths and uses them
to enhance their programme implementation. Weaknesses, challenges and bad practices, when identified, can be
reflected upon to make amendments and corrections, where necessary.


This paper has compared the main findings from the case study of CCJ to the Government-wide Monitoring and
Evaluation Framework to draw lessons from the NGO sector to the public sector. It follows the three main concluding
observations from the case study that are worth special consideration: firstly, that the purposes of M&E are determined
by those stakeholders who have interests in using it to answer questions about a programme; secondly, although M&E
within the CCJ is top-down, there are significant bottom-up contributions, both direct and indirect, to the process;
thirdly, there is a uniqueness to how funders’ contributions influence M&E in CCJ.


In as much as the purposes of M&E are the interests of those who need answers from M&E, the current study shows
that the South African public sector has its strengths and challenges. It has institutionalized M&E as a matter of
priority. It could be argued however that the available system is paying so much attention to the information needs of
stakeholders at the top of government hierarchies, to the detriments of the beneficiaries and the programme
implementers. This is against Rubin’s (1995: 30) notes that administrative staffs are the ones most in need of
evaluation information that would help to improve performance; they have practical day-to-day concerns about
techniques. They often rely on their own experience and intuitive judgment and thus often expect evaluations to come
up with something practical for them (Weiss 1998: 30). The government is always going to view service delivery from
funders and service providers’ perspectives. When focus of service delivery is not on the beneficiaries, M&E does not
consider their needs and perspectives it would continue to be viewed as paternalistic; viewing the problem with tinted
glasses. The voice of the beneficiaries would only be heard through service delivery protests which have been very
popular in South Africa in the recent past. Some approaches to evaluations discount the needs of beneficiaries for
whom the development initiative was meant, but the participatory approach to evaluations emerged to address this
(Brunner 2004: 103-104). This paper has identified the major challenges to using M&E in programme implementation
in both public and NGO sectors, and has drawn lessons from a case-study of CCJ on how challenges like a lack of
knowledge, capacity and buy-in by stakeholders, the need to include all relevant stakeholders in both design and
management of M&E and the challenges involved in these.