Transcript PROJECT PERFORMANCE APPRAISAL OBJECTIVES

INTERNATIONAL FINANCE CORPORATION
Workshop on IMPACT
ASSESSMNENT AND
RESULTS EVALUATION:
Developed by Alexander Kobzev
May 24, 2003
IFC KIEV
Introduction: anticipated results of this training
Upon completion of the training Ukraine Agribusiness Development
Project:
UADP team members would know how IFC monitoring and
evaluation system is working
Appropriate team members would fill in monitoring tool correctly
and use information
Input of UADP team members will be incorporated in monitoring
tool
Any other expectations?
TA MONITORING AND EVALUATION WAS DESIGNED TO MEET
TWO KEY OBJECTIVES:
To evaluate project progress and learn from best practices
To demonstrate projects transparency and to promote results to
PEP’s donors and IFC worldwide
REQUIRED STEPS AT VARIOUS STAGES OF PROJECT LIFE
Project Design
Critical steps
Clearly define
goals and
objectives
(Keep in mind
how you will
measure their
achievement )
Responsibility
SOM, IAM
Fundraising
Suggest our
approach or meet
specific donor
requirements
Project
Planning
Agree measurable
results of the project
(through M&E)
framework
Action
Monitoring &
Evaluation
Baseline data
collection at the
very early stage
Timely collect
progress data and
adjust project plan
(if appropriate)
Project team or
external source
Appointed team
member under
guidance of PM and
SOM
- Verify goals and
objectives
- Define actions, timing
- Set targets
SOM
SOM, IAM, PM and key
team members
(if appropriate)
FRAMEWORK FOR RESULTS MONITORING AND EVALUATION
Goal
Definition
Example
What we are
aiming
to change
To increase
financing to SME
through the
development of
leasing as a
financial instrument
Objectives
What should be
implemented
to achieve the
desired change
To improve/
increase leasing
service
To improve
knowledge and
attitude of potential
customers of
leasing service
To provide legal
assistance...
Activities
Our physical
actions
To deliver 24
trainings,
2 booklets
published, 5000
copies each
Output
-Direct results from
the Activities
- Have effect
immediately or in a
short term period
during the project
timeframe
600 trained
employees of financial
institutions
75% of participants
had significantly
improved
understanding of
leasing transactions
(measured by test)
Outcome
Impact
- Anticipated effect
- Can be observed
during the project life
and within 1-3 years
after completion
- Measures the
achievement of
Objectives
- Desired final change
- Has effect within 110 years after
completion, rarely
during the project
timeframe
- Measures the
achievement of Goal
-Increased number of
financial institutions
providing leasing service
-At least 4 as a direct
outcome of the project
Leasing companies
growing their business
More leasing finance is
available to SMEs
FIRST TOOL USED TO STRUCTURE PROJECTS AND TO SET TARGETS
E.g. Technical Assistance for Sugd Agro Serv
ACTIVITIES:
ACTIVITIES TARGETS:
What are w e actually doing?
EXPECTED
OUTPUTS:
OUTPUT TARGETS:
What are the im m ediate results of our action?
EXPECTED
OUTCOMES:
What should be im plem ented?
Objective 1: To introduce modern agronomic practices
Deliver trainings/
consultations to SAS
shareholders and
farmers in the broader
Sugd province
Set up and maintain
demonstration field
Provide field training
and advice to SAS
members
24 sessions of the basic
course (basic course =60
hours) and 8 sessions of
advanced course (advanced
course =100 hours)
a) crop production
b) sound environmental
practices, including irrigation
management
c) occupational health and
SAS shareholders
safety
and farmers in the
broader Sugd
province improve
their know ledge of
Establish 25 hectares demo modern crop
site, involving other supplier production practices
companies in the region and
governmental agencies
8 field days a year, (20 in
total) covering
a) laser levelling
b) irrigation
c) seeding
d) tillage
e) harvest management
OUTCOME TARGETS:
EXPECTED IMPACT:
IMPACT INDICATORS:
Where it should bring us at the very end?
Priority A
At least 400 participants
have received basic
training and at least 100 advanced training in:
a) crop production
b) sound environmental AG
practices
c) occupational health and
Improvement in:
safety
1.1 Salinity
management
80% of participants have
1.2 Pest
passed test w ith a pass
management
mark of 75%
1.3 Fertilizers use
1.4 Irrigation
1.5 Crop
diversification
At least 40 participants per 1.6 Safety of
field day (~800 in total)
farming
a) laser levelling
b) irrigation
c) seeding
d) tillage
e) harvest management
On all SAS shareholder farms
w ater table is low er than 2
meters at all locations by 2007
SAS area of salinized land does
not increase
Poverty is reduced
through:
Crop yield per hectare
Crop productivity gradual increases by 50% by the
increase
year 2010
Average net income
Increased SAS area under active Farms earn more income increases to 100USD per
hectare
use of biological control systems and become more
profitable
Loan defaults less tha
Full adoption of integrated pest
3%
management by SAS
Farmers debt burden is
Debt per hectare is
servicable
servisable
All SAS hectares receive
accurate timimg, rate and
placement of fertilizers by 2006
30% reduction in w ater use per
SAS hectare by 2006
Flexibility in crop production is
resulting in profitable operations
each year
Decreased accident and mortality
rate
Farmers earn more for
their families
Income per farm w orker
increased to 20 USD per
month by 2007
Model is replicated in
other Tajik and CIS areas # of replications
SECOND TABLE USED AS A PLANNING TOOL FOR MONITORING
E.g. Central Asia Leasing Project
Objective 2: To improve/increase the knowledge and understanding of equipment leasing
DATA SOURCE &
ACTIVITIES:
OUTPUT INDICATORs
FREQUENCY
COLLECTION
METHODs
Develop and deliver
# of customized training modules
Internal reports
Quarterly
customized training in
# of seminars for government officials
each country to
# of seminars for financial institutions
government officials and # of seminars for lessers accountants
businesses
# of seminars for lessees
Provide individual
# of hours of consultation provided
consultation to lessers, # of consultations provided
lessees, potential
lessees and lessers,
local commercial banks,
microfinance institutions
OUTPUTS:
Potential lessees and
lessers, and local
officials understand
concept and
practicalities of leasing
OUTCOME
OUTPUT INDICATORs
# of trained participants:
~300 governmental officials
~700 fin institutions rep's
~180 lessers' accountants
~1000 SME representatives
# of organizations served
OUTCOME INDICATORs
Internal reports
Quarterly
RESPONSIBILITY
TARGET
CURRENT
STATUS
Team assistasnt
9-12 (total)
40
50
20
40
4
2
0
1
1
Team assistasnt
1200
55
DATA
II 2002
III 2002
IV 2002 I 2003
II 2003
II 2002
III 2002
IV 2002 I 2003
II 2003
III 2003
1000
DATA SOURCE &
COLLECTION
METHODs
Internal reports
Internal reports
DATA SOURCE &
COLLECTION
METHODs
FREQUENCY
Quarterly
Quarterly
FREQUENCY
RESPONSIBILITY
TARGET
CURRENT
STATUS
DATA
III 2003
Team assistasnt
Team assistasnt
RESPONSIBILITY
300
700
180
1000
600
(200 per
country)
30
0
40
20
16
TARGET
CURRENT
STATUS
DATA
II 2002
Governmental officials
are more supportive to
leasing
Financial institutions
launch leasing or
improve leasing
operations
Lessers take all available
tax exemptions
Local leasing institutions report to
have support from local officials
Leasing institutions
survey
Annually
Deputy project
manager
N/a
N/a
# of participating companies launching
leasing operations
# of participating companies improving
leasing operations
% of participating companies improve
profitability
Leasing institutions
survey
Annually
Deputy project
manager
4
N/a
6
N/a
Leasing institutions
survey
Annually
Deputy project
manager
90%
N/a
Private sector is more
active in attaining
finance through leasing
% of participating SME consider
choice betw een credit and leasing
% of participating SME apply for
leasing
SME survey
(comparison of
participating and
other comppanies)
Annually
Deputy project
manager
40%
N/a
10-20%
III 2002
IV 2002 I 2003
II 2003
III 2003
PROJECTS BECOME MORE EFFECTIVE
Eg.: Pilot Training of Uzbek Consultants - Midpoint Evaluation



Project will train fourteen local Uzbek consultants over two separate rounds with
each round lasting some three weeks
The project team decided to allow consultants a period of six months to implement
their training
In March 2002 an evaluation of the first round was conducted through personal
interviews with consultants and their clients
Key findings:
– Consultants showed improved knowledge of key business specializations where they
previously had little or no exposure
– During the six given months they progressed toward service diversification,
project/client management, but less than expected
Result: redesign of program for 2nd round
Results for Activity 1
Activity:
Activity Specification:
Activities Outputs
Conduct and disseminate
policy-oriented research and
carry out consultations with
policymakers
Produce informational and
analytical materials;
Issues addressed in produced
policy papers(concepts,
proposals), the extent of papers
distribution;
Carry out consultations on
legislative issues;
Participate in working group
activities.
Submitted proposals and
comments to working groups.
Outcomes
Policymaking process is
facilitated;
Policymakers are better
informed about
agribusiness issues;
New strategies and
legislation are proposed
and developed by
policymakers.
Different groups/ministerial
departments consulted and issues
addressed;
Outcome Verifiable Indicators
Provided advice is used by
government;
New legislation specifically
incorporates our suggestions;
Requests from government for
information/advice or to
participate in working groups.
Impact
Policy environment is more
favorable for agribusiness
development (needed
legislation is adopted)
Results for Activity 2
Activity:
Promote credit and
insurance relations in
the agricultural sector
Activity Specification:
Activities Outputs:
Provide consultations to banks,
insurance companies and other
financial intermediaries;
Number of clients served and
issued addressed;
Promote interaction/be an
intermediary between
farms/agribusinesses and financial
institutions
Organize and conduct training
programs.
Outcomes:
Outcome Verifiable Indicators:
Better access to credit,
insurance secured;
Expanded the number of banks,
financial institutions providing
credit, insurance;
Financial institutions
more knowledgeable
about agribusiness (and
vice versa);
New financial products
are introduced, including
long-term credit.
Number of different companies
consulted;
Number of contacts between
farmers and financers created;
New concepts developed and
presented;
Number of business plans
developed and submitted to
banks.
Amount of credit, insurance provided
and extended amount of long-term
credit;
Impact:
Increase the rate of repayment;
Stable market for long-term
rural credit and insurance is
established.
Percentage of crops insured;
Number of new financial products
introduced.
Farms become more
profitable and efficient;
Results for Activity 3: Agronomic Activities
Activity:
Activity Specification:
Introduce new agronomic
practices.
Provide consultations, organize
seminars on new crop varieties
and use and management of
agricultural inputs;
Set up and maintain
demonstration fields introducing
new crop varieties;
Outcomes:
Be an intermediary between
input-supplier and producers.
New, more productive
crop varieties planted;
New techniques for
planting , input
management adopted;
Better utilization of
agricultural inputs;
Farmers are better
educated on agronomic
practices and better
decision makers.
Activities Outputs:
Farmers served through carried
out consultations and seminars;
Increased the number of farmers
introduced to new crops and
new technologies;
Training materials produced and
extent of their distribution to ag
producers;
Increased the access of input
suppliers to ag producers and
vice versa.
Outcome Verifiable Indicators:
Number of new crops farmers
cultivate;
Impact:
Number of new inputs used; more
economical/productive use of
existing inputs
Increase in yields, crop
productivity and in farm
income increases
Extent of new technologies
farmers use in their use.
Results for Activity 4
Activity:
Conduct an information
campaign through media
events, publications, and
public discussions.
Activity Specification:
Activities Outputs:
Publish bimonthly bulletin and
other publications;
Publication produced, issues
presented to the public, and
number of subscribers;
Organize and conduct press events
(press clubs; meetings with the
media) and prepare press releases;
Develop and maintain contacts with
government, the media, NGOs,
other research institutions, etc.
Outcomes:
Established project as a
reliable source of
information;
Provided regular and
free access to
information;
Facilitated contact
among government,
NGOs, media,
researchers.
Outcome Verifiable Indicators:
Number of times articles are
referenced from Project’s
publications;
Number of new subscribers, rate of
re-subscription;
Number of articles, broadcasts about
event.
Number of new requests for
interviews
Number of media representatives
and experts participated in
events;
Number of interviews conducted
and number of different media
outlets that cooperate with the
Project;
Number of contacts established
and joint events conducted.
Impact:
Increase awareness of
private agribusiness issues
and gain support for private
agribusiness.
OUTPUTS
- Direct results from the Activities
- Have effect immediately or in a short term period during the project
timeframe
Completed activities are not Outputs
Completed
Activities
E.g.
Outputs
Conduct training on
“Effective Presentation”
Nine UIP consultants have
improved their presentation skills
Provide 3 consultations to
Policy makers
Two draft amendments to Land Code
were proposed
Carry out study of Forests
Sector in NW Russia
At least 15 potential investors receive
the study report
OUTCOMES
- Anticipated effect
- Can be observed during the project scope and within 1-3 years after
completion
- Measure the achievement of Objectives
Outputs vs Outcomes – What’s the difference?
Outputs
E.g.
Outcomes
How many people have
improved their knowledge?
How well they have applied
the received knowledge
300 SME representatives
were trained on advantages
of leasing
50 SME participated in the training have
applied for lease financing
IMPACT
- Desired final change
- Has effect within 1-10 years after completion, rarely during the project
timeframe
- Measures the achievement of Goal
Outcomes vs Impact – What’s the difference?
Outcomes
E.g.
Impact
What was implemented?
How the implemented changes
have affected the business?
Accurate fertilizing
Crop management
Increase of yields per hectare,
Revenue and profits
WE SET TARGETS AND DEFINE BASELINE LEVEL
TO HAVE CLEAR EXPECTATIONS FOR THE PROGRESS
TARGET
- Planned level of the indicator
BASELINE
- Level of the indicator at the project starting point
HOW TO WORK WITH THE TABLE?
EXPECTED OUTPUTS:
…
OUTPUT
TARGETS:
Players on leasing market 100-150 managers from
of NW Russia improve
at least 50 companies
their know ledge and skills trained or consulted
of application and
advantages of lease
What should change?
Write short verbal description of the
expected result in the first column
…
How you will measure?
What’s your target for it?
Choose indicator and set target level
for it
COMPONENTS OF THE PROJECT
Training
Potential indicators
-
Objectives
-
Activities
-
Our means of
training
-
To introduce modern sustainable agricultural
technologies in XX oblast
To raise local management skills in financial
controlling
# of trainings delivered by subject
# of training hours
# of copies and key components of manuals
distributed
Name key components of online training tools
Outputs
-
# of people trained from # of companies
% of participants completed test with a XX pass
mark
Outcomes
-
Increase in # of hectares under biological control
systems
Improve in profitability of participating companies
% of participating SMEs apply for leasing
% of served companies improve internal documents
and CG practices
How many and
how well people
were trained
Results of
application of
received
knowledge
(NB measure
achievement of
Objective)
-
Comment
-
Better avoid: To raise local management skills – too
-
Better avoid: # of training modules and other
training materials developed – additional measure,
general objective, it’s better to be specific on the on
the subject of training
successfully conducted training assumes developed
training materials
-
Better avoid: Independent testing of the training
materials – additional costs & questionable value
COMPONENTS OF THE PROJECT
‘Train the Trainer’
Potential indicators
-
To raise local management skills in financial
controlling trough building local training capacity
To increase knowledge about CG of the next
generation of managers
Objectives
-
Activities
-
# of seminars/round tables other events for
partners
-
# of trained partners
# of case studies, manuals, training modules
transferred
-
# of partners delivering training over time
# of SME representatives trained by the partners
over time
% of participants completed test with a XX pass
mark
Our means of
knowledge
transfer
Outputs
Outcomes
How many people
and how well
are trained by
the partners
-
Comment
-
Be specific on the on the subject of training
-
Better avoid: Number of issued covered
-
NB to identify target audience: e.g. business people
vs students
For internal information can be worthy to revisit
partners and check quality of teaching
-
COMPONENTS OF THE PROJECT
Legislative Work
Potential indicators
Objectives
-
To improve legislative framework of XX sector
Activities
-
# of meetings/round tables/other events held
# of presentation given during the
# of hours of consultation to government
-
# of drafts/amendments proposed
Name of targeted laws to draft or amend
-
# and names of approved drafts/amendments
Our means of
influencing the
Policy makers
Outputs
What we
propose
Outcomes
What was
adopted
Comment
Concern for further usability:
# of issues identified or discussed
# of collaborative activities and joint activities
# of different departments consulted
# of other organizations requested our policy input
COMPONENTS OF THE PROJECT
Direct Assistance to Companies – Objectives and Scope Can Vary Greatly
Potential indicators
Objectives
Activities
Outputs
Outcomes
-
To support investment initiatives in particular sector
To built up supply chain links for XXX company
To increase access to XXX market of particular
companies
-
#
#
#
#
-
# of companies assisted with the tailored program
# of partnerships created
-
Value of investment in progress and completed in #
of companies
Additional sales, market share gain
Reduced costs, additional profits
-
of
of
of
of
feasibility studies/business plans
potential partners evaluated
hours of consultation
deals consulted
Comment
Concern for further usability:
# of obstacles identified
Level of satisfaction of investors/clients
Independent valuation of research quality
COMPONENTS OF THE PROJECT
Public Relations
Potential indicators
Objectives
Activities
PR means
-
To increase public awareness of equipment leasing
issues
-
# of round tables/conferences/ press conferences
held
# of press releases sent
Deadline for website launch and frequency of
update
-
Outputs
Expected
-
# of media appearances per month/quarter and in
total
-
# of people reached from target audience per
month/quarter and in total
# of project replications
publicity
Outcomes
Publicity reach
and effect
-
Comment
-
PR is useful to to get more ‘clients’/partners or
replicate the project. If there is no need in it then,
we should consider usefulness of nation-wide
campaigns
Concern for further usability:
# of new contacts established
#of requests from media to organize press events
# of times articles/press releases are referenced
# of new requests for interview