Performance Measurement in the Conservation Community Status, progress, barriers, and next steps Background material for the Measuring Conservation Effectiveness Summit May 5-6th, Palo Alto, CA.
Download ReportTranscript Performance Measurement in the Conservation Community Status, progress, barriers, and next steps Background material for the Measuring Conservation Effectiveness Summit May 5-6th, Palo Alto, CA.
Performance Measurement in the Conservation Community Status, progress, barriers, and next steps Background material for the Measuring Conservation Effectiveness Summit May 5-6th, Palo Alto, CA INTRODUCTION To prepare for the May 5-6th summit on Measuring Conservation Effectiveness, 29 conservaiton implementers and funders were surveyed to ask about their Systematic Performance Measurement (SPM) practice, which we defined as: The regular monitoring, evaluation, and adaptation of conservation actions based on clearly stated goals, objectives, and assumptions so as to assess effectiveness, promote learning, and report achievements. INTRODUCTION Respondents included: • 15 conservation implementers, representing more than 2 billion US$ in conservation spending and an effort equivalent to 16,000 staff person years. • 14 conservation funders, representing 1.1 billion US$ in annual support spread across about 1,000 grants. • Respondents varied in size, but most had budgets between 11 and 100 M USD. 12 Implementers 10 Funders 8 6 # of respondents 4 2 0 0-1 1-10 11-100 101-1000 >1000 $ Millions INTRODUCTION Respondents included: • Conservation Implementers • Conservation Funders Marisla Foundation Mulago Foundation INTRODUCTION In practice, implementers say SPM takes many forms… “annual plans and performance reviews” “OECD-DAC principles for evaluation” “project reviews against set criteria” “peer-review audits” “information flow up to organization level” “implementation of the Open Standards” “reporting of key deliverables against expected results” INTRODUCTION …as do funders. “communication tool for our board” “venture capital type investments with a clear theory of victory” “rigorous post-grant monitoring” “requirement for grantees” “tool to understand why a funded project fails” “constant and direct assessments of progress” “qualitative approach to evaluating grantee results and sustained change” INTRODUCTION We note that small organizations seem to do SPM to a greater extent than large organizations and this affected our analysis. We show results herein both by % of responding organizations and % of total combined conservation spending, and graphics are coded accordingly. For example, we asked: At the scale of projects & programs, is SPM practiced well at your organization? 8 2000 By Org 6 By $$$ 1500 # of respondent s 4 $ Million 1000 2 500 0 Almost never Occasionally (<10%) (10-49%) Often (5090%) Almost always (>90%) 0 Almost never Occasionally (<10%) (10-49%) Often (5090%) Almost always (>90%) INTRODUCTION Here‘s what we learned 1. SPM is widely acknowledged as important... 2. ...but is not done to a great extent across our community… 3. …however, we can improve by leveraging the progress we have made… 4. and overcoming several critical obstacles. SPM IS IMPORTANT IN CONSERVATION 1. SPM is widely acknowledged as important in conservation. SPM IS IMPORTANT IN CONSERVATION SPM is generally viewed favorably and as a priority. Survey Question: In general for each audience, how strongly do you agree/disagree with the following statements: Strongly disagree Has a positive/favorable attitude towards implementing SPM within the organization Upper Management Program Directors Program Directors Project Managers Project Managers 20% 40% 60% 80% Strongly agree 100% By Org Board Upper Management 0% Moderately agree Views the implementation of SPM in the near future as a high priority By Org Board Moderately disagree 0% 20% 40% 60% 80% 100% SPM IS IMPORTANT IN CONSERVATION When SPM is implemented, the primary driver is to improve effectiveness. Survey Question: If your organization does or attempts SPM, how important are the following reasons behind that effort? Not at all important Minimally important Somewhat important Very important To improve organizational effectiveness and/or efficiency By Org To enable project managers to evaluate effectiveness To satisfy requests from board and/or upper management To improve strategic fundraising To facilitate adaptive learning of what works in conservation To improve coordination and/or integration across programs To improve evidence-based communication To satisfy pressure from donors and/or supporters to show results To guide funding decisions by board and/or upper management To guide capacity building decisions by board and/or upper… To satisfy requests from project managers 0% 20% 40% 60% 80% 100% SPM IS IMPORTANT IN CONSERVATION To assess and improve our effectiveness, there are many important questions we want to answer.... Survey Question to IMPLEMENTERS. How important is answering the following questions within your organization? Not at all important Minimally important Somewhat important Very important Are our projects having their intended impacts? Are our programs having their intended impacts? By Org Can credible results be demonstrated to our board, donors, and supporters? Are our actions being adapted and improved? What can be learned to improve our organization's work? Are our actions cost-effective? Do we understand why a project/program fails when it does so? Can our actions be better coordinated across the org? 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% SPM IS IMPORTANT IN CONSERVATION Survey Question to FUNDERS: How important is answering the following questions within your foundation? Not at all important Minimally important Somewhat important Are our grantee's projects having their intended impacts? Very important By Funders Are our funding programs having their intended impacts? Are our grantee's actions being adapted and improved? What can be learned to improve our foundation's work? Do we understand why a grantee's project fails when it does so? Can our funding be better coordinated with other foundations? Are our foundation strategies regularly being adapted and improved? Can credible results be demonstrated to our board? Are our grantee's actions cost-effective? Can our funding be better coordinated across the foundation? 0% 20% 40% 60% 80% 100% SPM IS IMPORTANT IN CONSERVATION While funders are fairly confident they can answer key performance questions regarding their grantees… Survey Question: Using your foundation's current SPM system(s), how well can your foundation answer the following questions? Not at all Minimally well Somewhat well Can credible results be demonstrated to our board? Very well By Funders Are our grantee's projects having their intended impacts? What can be learned to improve our foundation's work? Are our foundation strategies regularly being adapted and improved? Can our funding be better coordinated with other foundations? Do we understand why a grantee's project fails when it does so? Can our funding be better coordinated across the foundation? Are our funding programs having their intended impacts? Are our grantee's actions being adapted and improved? Are our grantee's actions cost-effective? 0% 20% 40% 60% 80% 100% SPM IS IMPORTANT IN CONSERVATION …many of the implementing organizations feel less able to do so. Survey Question: Using your organization's current SPM system(s), how well can your organization answer the following questions? Not at all Minimally well Somewhat well Can credible results be demonstrated to our board, donors, and supporters? By Org Very well By $ Can our actions be better coordinated across the org? What can be learned to improve our organization's work? Are our actions being adapted and improved? Are our programs having their intended impacts? Are our projects having their intended impacts? Are our actions cost-effective? Do we understand why a project/program fails when it does so? 0% 50% 100% 0% 50% 100% SPM IS NOT DONE WIDELY 2. That we often cannot answer key performance questions with confidence is likely because... ...SPM is not done to a great extent across our community. SPM IS NOT DONE WIDELY For every conservation dollar spent, we say that only 10-30 cents worth is guided by SPM. 1800 1600 Survey Question: Of the total budget for your organization’s conservation efforts, what % is guided by SPM? 1400 conservation 1200 spend 1000 $ Millions 800 600 400 200 0 0-20 21-40 41-60 61-80 % of budget guided by SPM 81-100 SPM IS NOT DONE WIDELY Despite this estimate, we generally think we think we are doing SPM pretty well... 8 6 My foundation is effective at helping our grantees do SPM well. My organization does SPM well. 8 4 4 2 2 0 0 Strongly Moderately Moderately disagree disagree agree Strongly disagree Strongly agree By Funders Our grantees do SPM well. 8 2000 6 1500 4 $ M 1000 2 500 0 0 Strongly Moderately Moderately disagree disagree agree Strongly agree By Org 6 Moderately Moderately disagree agree Strongly agree My organization does SPM well. By $$$ Strongly Moderately Moderately disagree disagree agree Strongly agree SPM IS NOT DONE WIDELY …but when we are pushed, we admit that we rarely put our projects through a full SPM cycle. Of ~7000 projects currently undertaken by implementing organizations, only ~2500 have good conservation plans in place, but… only ~350-500 have completed the cycle of designimplement-monitor-evaluate-adapt. SPM IS NOT DONE WIDELY We generally do well at basic design and implementation of projects, but things break down when it comes to rigorous design, and monitoring, evaluation, and adaptation. SEE NEXT 2 SLIDES Survey Question: For your projects, indicate the extent to which your organization does each of the following SPM practices well. None Few Some Almost All A well-defined scope of work Identification & outreach to key stakeholders in planning process By Org Identification & outreach to key organizational partners in planning process Identification of conservation targets DESIGN Identification of threats Prioritization of threats to be addressed Situation analysis Identification of conservation actions Prioritization of conservation actions to be implemented Articulation of logic models for conservation actions IMPLEMENT Development of conservation action plan Development of operational plan Implementation of operational plan MONITOR Development of monitoring plan Implementation of monitoring plan EVALUATE & ADAPT Assessment of conservation status Assessment of performance of conservation action Use of data from monitoring and assessment to adapt future conservation action SHARE Sharing of lessons learned internally Sharing of lessons learned outside the organization 0% 20% 40% 60% 80% 100% These patterns seem even more apparent in None larger implementing organizations. Few Some Almost All A well-defined scope of work Identification & outreach to key stakeholders in planning process By $$$ Identification & outreach to key organizational partners in planning process Identification of conservation targets DESIGN Identification of threats Prioritization of threats to be addressed Situation analysis Identification of conservation actions Prioritization of conservation actions to be implemented Articulation of logic models for conservation actions Development of conservation action plan IMPLEMENT Development of operational plan Implementation of operational plan Development of monitoring plan MONITOR EVALUATE & ADAPT SHARE Implementation of monitoring plan Assessment of conservation status Assessment of performance of conservation action Use of data from monitoring and assessment to adapt future conservation action Sharing of lessons learned internally Sharing of lessons learned outside the organization 0% 20% 40% 60% 80% 100% KEY INGREDIENTS TO PROGRESS TO DATE 3. To improve our ability to measure effectiveness, we can leverage several ingredients key to our progress to date. KEY INGREDIENTS TO PROGRESS TO DATE Institutional mandate and SPM champions have been key ingredients to the progress we’ve made to date. Survey Question: Where you do see SPM happening in your organization, how important were the following 'key ingredients' or 'catalysts' to SPM adoption? Not necessary Moderately useful Very important Absolutely essential Institutional mandate Presence of a champion within organization By Org Evidence that SPM led to increased effectiveness and/or efficiency A vision for what could be accomplished with SPM Dedicated funding for SPM Dedicated SPM program with staff supporting implementation A comprehensive plan that integrated SPM Donor requirement to adopt SPM Donor reporting requirement Software tools that support SPM collection, management, &… Seeing it being successfully implemented by other conservation or… 0% 20% 40% 60% 80% 100% CRITICAL OBSTACLES TO PROGRESS Institutional mandate is a key driver of SPM adoption. Survey Question: Is SPM mandated at your organization? Is it practiced well at your organization? 100% By Org Almost always practiced well 80% Often practiced well 60% Occasionally practiced well 40% 20% Almost never practiced well 0% Informally Promoted Formally Promoted Mandated KEY INGREDIENTS TO PROGRESS TO DATE The 4 implementers that say that 80-100% of spending is guided by SPM reinforce the importance of institutional mandate, evidence that SPM helps, and champions. Survey Question: Where you do see SPM happening in your organization, how important were the following 'key ingredients' or 'catalysts' to SPM adoption? Not necessary Moderately useful Very important Institutional mandate Absolutely essential By Top Implementers Evidence that SPM led to increased effectiveness and/or efficiency Presence of a champion within organization A comprehensive plan that integrated SPM A vision for what could be accomplished with SPM Dedicated funding for SPM Donor requirement to adopt SPM Donor reporting requirement Dedicated SPM program with staff supporting implementation Software tools that support SPM collection, management, &… Seeing it being successfully implemented by other conservation or… 0% 20% 40% 60% 80% 100% KEY INGREDIENTS TO PROGRESS TO DATE Funders also emphasize the importance of a mandate, evidence that SPM helps, having a workplan that includes SPM, SPM being integral to an org’s mission and goals, and dedicated SPM funding. Not necessary Moderately useful Very important Institutional mandate Absolutely essential By Funders: Evidence that SPM led to increased effectiveness and/or efficiency Perception of Grantees A workplan of activities that integrated SPM SPM as an integral part of organization's mission & goals Donor reporting requirement Dedicated funding for SPM Donor requirement to adopt SPM Presence of a champion within organization Seeing it being successfully implemented by other conservation or development NGO Dedicated SPM program with staff supporting implementation Software tools that support SPM collection, management, & reporting 0% 20% 40% 60% 80% 100% CRITICAL OBSTACLES TO PROGRESS 4. To improve our ability to measure effectiveness, we also must overcome several critical obstacles. For all organizations, these primarily include: • Lack of time • Lack of money • Lack of staff dedicated to SPM • Perception that SPM is too complex In addition to these, obstacles particularly important in larger organziations include: • Lack of donor pressure • Lack of board pressure • Lack of demand from upper management CRITICAL OBSTACLES TO PROGRESS Funding dedicated to SPM represents 1-5% of total conservation spending by implementers. 2500 2000 $ Millions By $$$ 1500 1000 500 0 1-5% 5-20% >20% % conservation spend spent on SPM 10 8 6 # of respondents 4 2 0 By Org 1-5% 5-20% >20% CRITICAL OBSTACLES TO PROGRESS For every 100 projects, there is ~1 staff member dedicated to SPM at implementing organizations. Other staff data: • Collectively, there’s about 1 conservation coach for every 60 projects and 150 organizational staff. • There’s about 1 conservation trainer for every 140 projects. By Org 16,400 organizational staff 71 dedicated SPM staff • There’s about 1 auditor for every 500 projects. • At organizations, there is a ratio of 230 : 1 organizational staff to SPM staff • At foundations, the ratio is 30 : 1 • There’s about 1 foundation staff dedicated to SPM for every 90 grants • At foundations, about 20% of conservation program staff’s time is dedicated to SPM By Funders 578 foundation staff 19 dedicated SPM staff CRITICAL OBSTACLES TO PROGRESS Understanding of what SPM is and how it might benefit an organization seems weak. Survey Question: In general for each audience, how strongly do you agree/ disagree with the following statements? Moderately Strongly Moderately disagree By Org $$$ Has a thorough understanding of what SPM is and how it is implemented at the organization Board Upper Management Upper Management Program Directors Program Directors Project Managers Project Managers 20% 40% 60% 80% 100% Strongly agree Has a thorough understanding of what the possible benefits of SPM are to the organization Board 0% disagree disagree 0% 20% 40% 60% 80% 100% CRITICAL OBSTACLES TO PROGRESS The following four slides show responses to the survey question: In your experience, how important are the following factors in impediting adoption of SPM? The data is presented four ways: • Slide 33: By Implementing Organization responses • Slide 34: By Implementing Dollars spent • Slide 35: By Implementing Organizations that report less than 20% of their budget is guided by SPM • Slide 36: By Funder responses: perception of grantees Not an issue Minor hurdle Major barrier SPM: Obstacles to SPMLack of time SPM Extreme Survey obstaclePage <#> By Org Lack of money Lack of staff dedicated to SPM Perception that SPM is too complex Lack of training Money, time, and trainers the most important barriers. Lack of an overall culture of accountability to our bottomare line Lack of quality trainers Lack of good examples of SPM helping to achieve conservation… Lack of incentives to change the status quo Lack of support and/or interest from project managers Lack of donor pressure Lack of board pressure Lack of demand from upper management Lack of support from upper management Lack of database exchange to share practices and learning Perception that SPM is too simplistic Perception that SPM is unnecessary to doing effective conservation Lack of peer pressure Lack of good software tools to implement SPM Reporting guidelines of major donors discourage or inhibit SPM Perception that SPM has become a meaningless buzzword 0% 20% 40% 60% 80% 100% Not an issue Minor hurdle Major barrier SPM Extreme obstacle Survey Page <#> Lack of money By $$$ Perception that SPM is too complex Lack of donor pressure Lack of board pressure Lack of time Lack of demand from upper management Lack of staff dedicated to SPM Lack of an overall culture of accountability to our bottom line Lack of incentives to change the status quo Lack of support from upper management Lack of good examples of SPM helping to achieve conservation… Lack of quality trainers Lack of training Lack of support and/or interest from project managers Lack of peer pressure Perception that SPM is unnecessary to doing effective conservation Perception that SPM has become a meaningless buzzword Reporting guidelines of major donors discourage or inhibit SPM Lack of good software tools to implement SPM Lack of database exchange to share practices and learning Perception that SPM is too simplistic 0% 20% 40% 60% 80% 100% By the 5 implementers that say <20% of budget is guided by SPM Not an issue Minor hurdle Major barrier Extreme obstacle Lack of money Lack of time Lack of staff dedicated to SPM Lack of donor pressure Lack of quality trainers Perception that SPM is too complex Lack of board pressure Lack of support from upper management Lack of training Lack of an overall culture of accountability to our bottom line Lack of incentives to change the status quo Lack of demand from upper management Reporting guidelines of major donors discourage or inhibit SPM Lack of good examples of SPM helping to achieve conservation goals Lack of good software tools to implement SPM Lack of database exchange to share practices and learning Perception that SPM has become a meaningless buzzword Lack of support and/or interest from project managers Perception that SPM is unnecessary to doing effective conservation Lack of peer pressure Perception that SPM is too simplistic 0% 20% 40% 60% 80% 100% By Funders: Perception of Grantees Not an issue Minor hurdle Major barrier Extreme obstacle Lack of time Lack of money Perception that SPM is too complex Lack of staff dedicated to SPM Lack of an overall culture of accountability to our bottom line Lack of demand from upper management Lack of support from upper management Lack of support and/or interest from project managers Lack of good examples of SPM helping to achieve conservation goals Lack of quality trainers Lack of training Lack of incentives to change the status quo Perception that SPM is unnecessary to doing effective conservation Lack of board pressure Lack of donor pressure Lack of peer pressure Perception that SPM has become a meaningless buzzword Lack of good software tools to implement SPM Perception that SPM is too simplistic Reporting guidelines of major donors discourage or inhibit SPM Lack of database exchange to share practices and learning 0% 20% 40% 60% 80% 100% CONCLUSIONS Performance Measurement in the Conservation Community 1. SPM is important in conservation We say it is important and a top priority for the near future. We believe answering key questions regarding our effectiveness is very important. We generally do not believe we can answer these questions, however. 2. SPM is not done widely Only 10-30% of current conservation spending is guided by SPM. Very few projects do more than the initial steps of SPM Basic design and implementation of projects is generally done well, but things break down when it comes to rigorous design, monitoring, evaluation, and adaptation. CONCLUSIONS Performance Measurement in the Conservation Community 3. We can improve by leveraging key factors to progress to date... Institutional mandate Champions for SPM within organizations A vision for what can be accoplished with SPM Evidence of SPM leading to increased effectiveness of conservation action 4. ...and by overcoming several critical obstacles. For all organizations, these primarily include: • Lack of time • Lack of money • Lack of staff dedicated to SPM • Perception that SPM is too complex In addition to these, obstacles particularly important in larger organziations include: • Lack of donor pressure • Lack of board pressure • Lack of demand from upper management THANK YOU!! Thank you to all survey respondents! Marisla Foundation Mulago Foundation Summit Research Committee Sheila O'Connor (chair) Bernd Cordes William Crosse Brett Jenks Richard Margoluis Matthew Muir Elizabeth O'Neill Nick Salafsky Kristin Sherwood Frog illustrations © Ted Kahn 2010 Original artwork used with permission Neotropical Conservation Foundation www.neotropicalconservancy.org