W

HERE

BER S

TARTED

Evaluation of Oxfam GB’s Global Climate Change Campaign Solution to evaluation challenge of considering value for money of a complex, multi-unit, international campaign

1

E

VALUATION CHALLENGES

Simplifying complex multi-unit programs Many evaluations are about ROI, but ROI is difficult to assess in social contexts Nothing is good or bad, except in comparison to something else 2

T

HE

BER S

OLUTION

Provide a simple framework for evaluating complex multi-component programs, campaigns, or activities Build on the basic concepts of SROI to evaluate unit's impact compared to their resources Offers a relative perspective on performance where units of analysis are judged in comparison to their peer units, operating under similar conditions 3

T HEORETICAL F OUNDATIONS : M ATRIX ANALYSIS

•

FRAMEWORKS

Boston Consulting Group • General Electric Grid • Customer satisfaction quadrant analysis by Andreasen • Bloc modeling techniques used by social network analysts • Multi dimensional scaling • SROI 4

BER V

ARIABLES

input constitutes a program’s resources which may

measure of capacity, concrete or abstract. be measured by its budget, number of staff, pool of talent, social capital, or any

Output measures a program’s impact, and will vary according

to a program's purpose – may include behaviour change; public awareness; policy change; reduced inequality; improved environmental health… or any other measure depending on a program's goals

Input High Low

Below Average Efficiency Average Efficiency Average Efficiency Above Average Efficiency

Low Output High

5

CONDUCTING A BER ANALYSIS

The example in following section is fictional and for illustrative purposes.

6

1. S

ELECTING UNITS OF ANALYSIS

For example: Coalition/partnership building; intra organisational coordination; lobbying and advocacy; media relations; online engagment; public mobilization; research...

  

Input

Program budgets (perceived and real) Number of staff Number and level of staff    

Output

How often a lobbying keyword appeared in policy Number of widgets produced Number of people engaged Perceptions of impact 7

2. D

ATA TYPES

• • Quantitative input data may include budgets, number of staff, or combined multi-dimensional resource measure Quantitative output data may include process evaluation measures such as the number of people engaged by a campaign or media hits • Qualitative measures can include perceived program investments and perceived output achieved 8

3. M

EASUREMENT TOOLS

1. Units of Analysis

Online engagement Research papers Lobbying and advocacy

3. Perceived Input (Budgets)

•Most •Average •Least •I don't know •Most •Average •Least •I don't know •Most •Average •Least •I don't know

2. Perceived Output (impact)

•Most •Average •Least •I don't know •Most •Average •Least •I don't know •Most •Average •Least •I don't know Organisation X has influenced policies or legislation because of the following activities:

Completely Disagree 1 2 3 4 5 Completely Agree 6

I can't say Online engagement Research papers Lobbying and advocacy 9

4. V

ISUALIZATION APPROACHES

High Low Low High 10

4. V

ISUALIZATION APPROACHES

High

  Media relations Public relations

Input



Low

 Intra organisational coordination Online engagement   Lobbying and advocacy Coalition/partnership building   Research papers Public mobilization Low High Output 11

4. V

ISUALIZATION APPROACHES

12

C ASE S TUDY : O XFAM C HANGE C AMPAIGN GB’ S G LOBAL C LIMATE

High 4.5

4.0

3.5

Low 3.0

2.0

Low Lobbying and advocacy Partnerships with others Global Oxfam affiliates Media engagement Policy analysis Climate hearings Staff in UNFCCC delegations Public campaigns/mobilization Rapid news dissemination Effective com at policy events Research papers Online campaigning Visual stunts & media relations Adopt a negotiator Using celebrities Oxfam internal program links 2.5

3.0

Perceived resourcing

Investment

3.5

High 13

5. I

NTERPRETATION

Use BER is as a starting point for deeper discussions into the performance of intervention units, their challenges, opportunities, and operating environment Understand the units of analysis and the informants who shared their perceptions Not all units within a program operate under the same conditions Some units contribute indirect effects, by empowering other units 14

L

IMITATIONS AND

R

ISKS

It is easy to draw conclusions from the simple visualizations that would never stand in the face of a deeper understanding of the reality behind charts Kotler et al. (2005) noted, reliance on matrix approaches prompted a number of companies to sell off strategic assets and plunge into businesses that they lacked the experience to manage 15