Transcript “Learning from Existing Evaluation Practices on the Impacts and Effects of Intellectual Property on Development” Geneva 6th/7th October Evaluation Section Internal Audit and Oversight Division (IAOD)World.

“Learning from Existing Evaluation
Practices on the Impacts and Effects of
Intellectual Property on Development”
Geneva
6th/7th October
2011
Evaluation Section Internal Audit and Oversight Division
(IAOD)World Intellectual Property Organization (WIPO)
Using Facilitated Self-Evaluations for
Intellectual Property Projects
Daniel P. Keller, Director
Swiss Consulting Co. Ltd. Hanoi – Vietnam
Management & Development Consultants
Professional experience in technical
cooperation
Evaluation experience in the field of IP focuses mainly on national
projects that use a comprehensive approach of strengthening
different aspects of IPRs as a tool to promote social and economic
development.
One project evaluated used a regional approach: Strengthening a
regional IPR system in, combined with support at national level to
Least Developed Countries (LDCs) within ASEAN.
Designed 5 IPR projects (for different national contexts, including 2
pilots to respond to Art. 67 TRIPS agreement (TA to LDCs).
Beyond this, over 30 other evaluations, mainly in the area of trade
related technical assistance, many of which also include IPR
aspects within a broader approach to trade capacity building, SME
development, etc.
Extensive own experience in managing projects.
Introduction
My presentation will introduce the methodology applied as well as
the key lessons learned from the facilitated self-evaluation of the
Swiss-Vietnamese Intellectual Property Project (SVIP).
SVIP: 2007 – 2010 (phase II), executed by the Swiss Federal
Institute of Intellectual Property, budget of 1 million CHF, funded by
Swiss State Secretariat for Economic Affairs, SECO.
Purpose: share personal experience as a facilitator of this exercise.
Why to use facilitated self-evaluations?
What conditions need to be in place?
Explain the self-evaluation process and steps used
Highlight key lessons learned identified by the self-evaluation report
Briefly discuss future trends in evaluation
Draw conclusions and provide some recommendations.
Why to use self-evaluations instead of
external evaluations?
Self-evaluations are a capacity building process. Partners learn
to assess whether the project „did the right things“ and „did things
right“ rather than merely reporting on implementation of activities.
Stronger „ownership“ of evaluation results. This increases
likelihood that partners apply recommendations and lessons learned
in practice. Particularly important for countries that shift from the role
of an aid recipient to providing themselves technical assistance.
While external evaluations might provide a more independent, fresh
view on the project, self-evaluations benefit from detailed and
specific insight on the project and its context.
Self-evaluations are a cost effective alternative to external
evaluations: Funding for external in-depth evaluations of small
projects is often not available, so they are not evaluated at all.
What conditions need to be in place for
successfully conducting a self-evaluation?
Only in a relatively advanced development context: Partners
should have a certain experience in implementing technical
cooperation projects that enables them to make meaningful
contributions to the self-evaluation.
Prior capacity building in monitoring & evaluation essential.
The methodology is only suitable if project design & monitoring
tools have been properly applied (logical framework, result-based
financial and operational reporting). Reconstituting an intervention
logic ex post (method typically used in evaluations of projects that
did not properly apply planning tools) would not be feasible.
Good partnership among partners/stakeholders. Selfevaluations require mutual trust. They are not suitable for
problematic projects (e.g. with a conflict situation).
Self-evaluation process and steps used
Workshop with partners on monitoring & evaluation.
Establishing factual basis: For each of the performance indicators
in the logframe, partners were required to assess expected against
planned outputs and outcomes (using OVIs) in the form of a table.
Partners were provided with a simple evaluation framework for
rating relevance, effectiveness, efficiency and sustainability for each
of the expected outputs/outcomes (high/medium/low) on a scale
plus a brief explanation for their assessment.
Draft findings, conclusions and recommendations (report) were
discussed at a stakeholder workshop with all direct/indirect
beneficiaries. Purpose to “independently“ validate evaluation results.
Report was discussed by the Steering Committee, amended
based on feed-back and then formally approved.
Key lessons learned on IPR projects derived
from the self-evaluation of the SVIP (1)
Developing well-functioning IPR systems in developing
countries calls for a comprehensive, coordinated support,
addressing all subject matters in parallel and combining the
strengthening of demand/supply side of IPRs in addition to the legal
and regulatory framework. The three-pronged approach used under
the SVIP (strengthening legal framework, IP administration and
institutions that promote IPRs among key users) was successful.
Sustainability of capacity building requires institutionalizing
training functions rather than only “train-the-trainers”. Followup trainings organized by training divisions of local counterparts
contributed to strengthening staff training within institutions in a
sustainable way. Projects should decisively shift away to provide
direct training at the level of IPR users.
Key lessons learned on IPR projects derived
from the self-evaluation of the SVIP (2)
Decentralizing day-to-day management of the SVIP to the field
level, while strengthening financial and operational monitoring was
the right management structure (seen as a key success factor for
effectiveness and efficiency of implementation).
Joint-implementation of technically complex projects allows the
project partners to contribute what they are best at. This is a good
way to ensure ownership, capacity building and sustainability
without compromising on aid effectiveness.
Selecting the right key management staff for projects: Important
are leadership skills, result-orientation, ability to move the project
forward, network, and knowledge of the country. Added benefits, but
not crucial, were technical in-depth knowledge about IPR and
seniority.
Key lessons learned on IPR projects derived
from the self-evaluation of the SVIP (3)
Project management and technical cooperation capacities of
implementing agencies crucial: The SVIP also showed the
importance that implementation agencies strengthen their specific
capacities/know-how in technical cooperation and project
management capabilities in the context of a developing country. IPR
specific “know-how” is important, but not sufficient.
Physical presence of the implementing agency in Vietnam was
a key success factor (partners compared the SVIP with other
project managed from outside the country).
Future trends in evaluation (1)
Trend to strengthen the monitoring & evaluation framework for
projects, starting at the preparation stage. Without a proper
framework and a result-based operational and financial reporting
system, evaluations suffer from considerable limitations! Pay
attention to this during the appraisal stage for new projects. Still
weak in many IPR-projects, but increasingly looked at by donors and
beneficiary governments.
Trend towards a more participatory approach to monitoring &
evaluation: Increased stakeholder involvement (self-evaluation
combined or not with external evaluation) is one way to respond to
this. This is in line with international efforts to strengthen the role of
partner countries in project implementation (monitoring & evaluation
are a part of this).
Future trends in evaluation (2)
Trend towards a more regular, external or internal resultoriented monitoring (ROM) of projects: Evaluations come often at
a stage where it is too late to address fundamental problems. ROM
is more suitable as a basis for timely strategic decision making,
because the picture provided is more up-to-date. ROM could replace
mid-term evaluations and feed into final evaluations. EU applies
external ROM and Switzerland starts piloting a modified version of
this approach for more complex projects.
Shift towards assessing impact- rather than only
output/outcome level 2 – 3 years the after project ends.
Evaluations often come too early for assessing impact and partially
also sustainability. This requires impact indicators in the logical
framework and the collection of relating baseline data. External
impact assessment of SVIP by SECO is planned.
Conclusions and recommendations
Self-evaluations respond well to the Paris Declaration on Aid
Effectiveness, by increasing role (“ownership”) of partner countries,
enhance “mutual accountability”, contributing to “managing towards
results“ and reducing transaction cost (“aid effectiveness”).
We advocate for the use of facilitated self-evaluations as a tool of
organizational learning in a more advanced development context,
where project partners have a certain degree of experience and a
successful track-record in technical cooperation.
Conducting facilitated self-evaluations might be an alternative for
smaller WIPO-Projects, for which the cost of an external evaluation
would not be commensurate to the overall project budget. They
could be used instead of a final evaluation for projects that
subsequently undergo an impact assessment.
Thank you for your interest!
Questions and comments
Discussion
Daniel P. Keller, President
Swiss Consulting Co. Ltd. Hanoi – Vietnam
Management & Development Consultants
www.swissconsulting.com.vn