Transcript Document

ROI for Meetings & Events
Elling Hamso
Managing Partner
European Event ROI Institute
The Purpose of Events
To INFLUENCE* participants
to DO something
which adds VALUE to stakeholders
at the lowest possible COST
* or educate, empower, inspire, persuade
The Event ROI Pyramid
ROI
Business Impact
Application
Learning
Satisfaction & Planned
Actions
The Chain of Impact
1. A good learning environment
2. Participants learn something
3. Participants apply what they learned
4. A business impact is generated
© Copyright 2004 ROI Institute
The Phillips ROI Methodology
• Refined over a 20-year period
• Thousands of ROI studies conducted every year
• 100 case studies published
• 2,500 individuals certified
• 30 books written to support the model
• Adopted by hundreds of organizations in more
than 40 countries
Some Users of the Methodology
 AT&T
 Coca Cola
 General Motors
 Hewlett Packard
 Hilton Hotels
 Motorola
 PricewaterhouseCoopers
 Shell Oil
 Singapore Airlines
 US Department of Defense
 NASA
 Government of Italy
 Government Canada
Some Areas of Application
• Management Training
• Sales Training
• Team Building
• Organization Development
• Recruiting Strategies
• Safety & Health Programs
• E-Learning
• Coaching
• Executive Training
• Self-Directed Teams
• Compensation Strategies
• Technology Implementation
• Customer Binding
• Meetings and Events
• Quality Six Sigma
• Wellness/Fitness
Initiatives
The Phillips ROI Methodology
Evaluation
Planning
1
2
Develop
Meeting
Objectives
Develop
Evaluation
Plans
© Copyright 2004 ROI Institute
Data Collection
3
Collect Data
Before and
During Meeting
4
Collect Data
After Meeting
7
Calculate
Costs
of Meeting
Reporting
Data Analysis
5
Isolate the
Effects of
the Meeting
6
Convert Data
to Monetary
Value
8
Calculate
the Return On
Investment
9
© Copyright 2004 ROI Institute
10
Identify
Intangible
Benefits
Communicate
results
Evaluation Planning
Assigning objectives to levels of measurement
Objective:
90% of attendees will indicate an intention to
implement new sales strategies within two
months after the meeting
1. Satisfaction, reaction,
planned action
2. Learning
3. Application
Answer:
Level 1 - Satisfaction & perceived value
(intentions are always level 1)
4. Impact
5. ROI
Evaluation Planning
Assigning objectives to levels of measurement
Objective:
Participants rate the relevance of the meeting to
success in their jobs as 4.5 out of 5
1. Satisfaction &
planned action
2. Learning
3. Application
Answer:
Level 1 - Satisfaction & perceived value
4. Impact
5. ROI
Evaluation Planning
Assigning objectives to levels of measurement
Objective:
Participants score 75 out of 100 or better on the
new strategy quiz
1. Satisfaction &
planned action
2. Learning
3. Application
Answer:
Level 2 - Learning
4. Impact
5. ROI
Evaluation Planning
Assigning objectives to levels of measurement
Objective:
Participants will be using the new customer
interaction skills in 90% of situations where they
are needed.
1. Satisfaction &
planned action
2. Learning
3. Application
Answer:
Level 3 - Application
4. Impact
5. ROI
Evaluation Planning
Assigning objectives to levels of measurement
Objective:
Decrease the amount of time required to
complete projects by 5% within 6 months of the
meeting
1. Satisfaction &
planned action
2. Learning
3. Application
Answer:
Level 4 - Impact
(time may be converted to money)
4. Impact
5. ROI
Evaluation Planning
Assigning objectives to levels of measurement
Objective:
Rate the meeting as a good investment for the
company with an average of 4.3 out of 5.
1. Satisfaction &
planned action
2. Learning
3. Application
Answer:
Level 1 - Satisfaction & perceived value
4. Impact
5. ROI
Evaluation Planning
Assigning objectives to levels of measurement
Objective:
Increase sales from existing customers by 5%
within 9 months of the meeting
1. Satisfaction &
planned action
2. Learning
3. Application
Answer:
Level 4 - Impact
4. Impact
5. ROI
Evaluation Planning
Assigning objectives to levels of measurement
Objective:
Achieve a 25% return on investment within the
12 months of the meeting
1. Satisfaction &
planned action
2. Learning
3. Application
Answer:
Level 5 - ROI
4. Impact
5. ROI
Evaluation Planning
Assigning objectives to levels of measurement
Objective:
Successfully demonstrate new selling techniques
during classroom role play
1. Satisfaction &,
planned action
2. Learning
3. Application
Answer:
Level 2 - Learning
4. Impact
5. ROI
Data Analysis
Some examples of Level 1
and Level 2 questions after Risk
Management workshop
© Copyright 2004 ROI Institute
Educational content: Risk Management
Disagree
Neither
Agree nor
Disagree
Agree
Strongly
Agree
Average
Score
0%
0%
0%
37.5%
62.5%
4.6
0%
0%
0%
50%
50%
4.5
Strongly
Disagree
Disagree
Neither
Agree nor
Disagree
Agree
Strongly
Agree
Average
Score
Communicated knowledge
effectively
0%
0%
0%
25%
75%
4.8
Knew the subject matter and
could address questions
0%
0%
0%
0%
100%
5.0
Kept my attention
0%
0%
0%
25%
75%
4.8
Strongly
Disagree
The content is important to me
I can use what I learned in my job
Facilitator Elling Hamso
How will you use what you learned?
No
Probably
Not
Maybe
Probably
Yes
Definitely
Average
Score
Discuss with my colleagues
0%
0%
0%
37.5%
62.5%
4.6
Review risk management
procedures in my organisation
0%
0%
0%
50%
50%
4.5
Conduct formal risk assessments
before major events
0%
0%
50%
37.5%
12.5%
3.6
Review how we check on
suppliers’ safety procedures
0%
12.5%
12.5%
62.5
12.5
3.8
12.5%
12.5%
62.5%
0%
12.5%
2.9
0%
0%
0%
0%
0%
0.0
Meet with our insurance brokers
to review current policies
Other (specify below)
Do you know the basic principles of how to assess risk?
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100
%
Avg
BEFORE the seminar
0
2
0
4
2
2
4
0
0
0
0
40%
AFTER the seminar
0
0
0
0
0
0
0
4
2
2
6
87%
Do you feel comfortable with leading a risk assessment exercise
for a major event?
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100
%
Avg
BEFORE the seminar
2
2
2
2
0
4
2
0
0
0
0
31%
AFTER the seminar
0
0
0
0
2
4
0
0
4
2
2
70%
Could you draw a matrix for classification fo different types of risk?
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100
%
Avg
BEFORE the seminar
6
2
2
2
2
0
0
0
0
0
0
14%
AFTER the seminar
0
0
0
0
2
4
0
2
2
0
4
70%
Could you list the four different risk management alternatives?
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100
%
Avg
BEFORE the seminar
8
2
0
2
0
2
0
0
0
0
0
13%
AFTER the seminar
0
0
0
0
0
2
6
2
0
0
4
71%
Data Analysis
Isolating the effects of the meeting
Total measured Impact
The Problem:
Other
Influences
Meeting
Impact
The Solution:
• Control groups
• Forecasting methods
• Trend analysis
• Participants’ estimates
• Supervisors’ estimates
• Outside experts
© Copyright 2004 ROI Institute
Data Analysis
Using control groups to isolate
the effects of the meeting
Retail
Merchandise
Company
Interactive Selling Skills Programme
Two
groups
of 3 stores
matched–on:
•Store size
•Store location (median houshold income in the area)
•Customer traffic levels
•Previous store perfomance
Weekly sales 3 months after training ($)
Trained group
Control group
Increase due to training
12,075
10,449
1,626
(ROI was in this case calculated to be 118%)
© Copyright 2004 ROI Institute
Data Analysis
Participants’ estimates of factors influencing sales
increase following training seminar
Elements influencing
sales increase:
Sales training seminar
Advertizing campaign
Incentive programme
Product improvements
Other factors
Total
Influence
factor
9%
16%
7%
49%
19%
100%
Confidence of
estimate
Adjusted
influence factor
65% (i.e. 6 – 15%)
6%
Data Analysis
Converting data to monetary value
The Problem:
The Solution:
• Profit from sales
• Converting employee’s time
• Historical costs
• Internal and external experts
• Participants’ estimates
• Supervisors’ and managers’
estimates
• Staff estimates
© Copyright 2004 ROI Institute
Data Analysis
Converting data to monetary value
using participants’ estimates
Examples of business impact converted to money after leadership training seminar
Basis for Value
Estimated
Value ($)
Confidence (%)
Adjusted
Value ($)
Improvement in efficiency of group.
$1,500 month x 12 (group estimate)
18,000
0,85
15,300
Turnover reduction: Two employees
retained per year. Base salary x 1.5
36,000
0,9
32,400
15% improvement in customer response
time. Estimated value $1.100/month
13,200
0,6
7,900
Absenteeism reduction (50 absences per
year x $150 standard value)
7,500
0,85
6,300
With permission from Jack Phillips, ’Measuring Return on Investment’ Vol. 2. ASTD Press, 1997
Data Analysis
Assigning costs
•
•
•
•
•
•
•
•
•
Marketing Costs
Developments Costs
Program Materials
Facilitator/Speaker Costs
Facilities Costs
Travel/Hotel/Meals
Participants Salaries & Benefits
Administrative/Overhead Costs
Evaluation Costs
© Copyright 2004 ROI Institute
Data Analysis
Intangible Benefits
Benefits which can not be converted
into money with minimum effort and
high credibility
© Copyright 2004 ROI Institute
How to get a high response rate
1. Easy to respond, simple and unambiguous questions
2. Clearly relevant questions, no repetition or redundancy
3. Get commitment to respond (explain during meeting how and
why you will measure results, get participants to volunteer)
4. Cover letter signed by person in authority
5. Incentive, win a prize, donation to charity
6. Share results (link at end of survey or report)
7. Anonymous (or option to be anonymous)
8. Say how long it will take to complete (honestly)
9. Return by date
10. Reminders with option to opt out
How to get a high response rate
11. Telephone follow-up
12. Explain why it is important to get their response
13. How can I benefit from results
14. Who will use / see the results
15. Make it compulsory
16. Must complete to get certificate
17. You will learn something by completing it
18. Professional design
19. Elements of humour
Some level 1 questions
1. Were your expectations met
2. Relevance to your job (in general or per session)
3. Did you learn something new, were you challenged
4. Speaker qualities (knowledge, engagement, responsiveness, etc.)
5. Did the facilities provide a good learning environment
6. Was it worth the cost (including cost of time)
7. Was it motivating/inspiring
8. Were you satisfied with your own contribution
9. Will you recommend to others
Guiding Principles
1. When a higher level of evaluation is conducted, data must be
collected at lower levels
2. When an evaluation is planned for a higher level, the previous
level of evaluation does not have to be comprehensive
3. When collecting and analyzing data, use only the most credible
sources
4. When analyzing data, choose the most conservative alternative
for calculations
5. At least one method must be used to isolate the effects of the
meeting
6. If no improvement data are available for a population or from a
specific source, it is assumed that little or no improvement has
occurred
cont.
Guiding Principles
cont.
7. Estimates of improvements should be adjusted for the potential
error of the estimate
8. Extreme data items and unsupported claims should not be used
in ROI calculations
9. Only the first year of benefits (annual) should be used in the
ROI analysis of short-term meetings
10. Costs of the meeting should be fully loaded for ROI analysis
11. Intangible measures are defined as measures that are
purposely not converted to monetary values
12. The results from the ROI Methodology must be communicated
to all key stakeholders
Books and Resources
Some useful websites:
www.eventroi.com European Event ROI Institute
www.roiinstitute.net ROI Institute, Inc.
Books:
“Proving the Value of Meetings & Events” Jack J. Phillips, Monica Myhill and
James B. McDonough, available from www.mpiweb.org and
www.eventroi.com
“Return on Investment in Training and Performance Improvement Programs”
Jack J. Phillips, available from major booksellers and www.amazon.com
ROI Courses in Europe
5-Day Event ROI Certification Course with Jack Phillips
18 - 22 August 2008, Brussels, Belgium
1-Day Event ROI Courses
London, Amsterdam, Brussels, Frankfurt, Dusseldorf, Hamburg,
Barcelona, Madrid, Milan, Turin, Copehagen, Stockholm, Helsinki,
Warzawa, Tallin. Information and registration on www.eventroi.com
Contact Details
European Event ROI Institute
Dr. Elling Hamso
Managing Partner
E-mail: [email protected]
Telephone: +47 90 12 24 18