Transcript Document 7475284

Measuring Business Impact
in Workforce Development
Lisa Soricone
Commonwealth Corporation
[email protected]
www.commcorp.org
NAWDP Conference
May 19, 2011
Page 1
Presentation Overview
•
•
•
•
•
•
Audience
Definitions
Our Experience
Steps Involved in Measuring Business Impact
Examples from Programs in MA
Q&A
Page 2
What do we mean by Business Impact?
• How does the business or employer benefit from a workforce
development intervention?
• Rooted in Kirkpatrick’s model for evaluation of training
1: Reaction
2: Learning
3: Behavior or application of learning
4: Results or business impact
• Jack Phillips’ additions
– Return on Investment
– Intangibles
• Continuum: from good stories to ROI
Page 3
Where do we get our insights?
• 10 years experience with sector strategies --- workforce development
with dual benefit (workers and employers)
–
–
–
–
BEST
Bay State Works
Workforce Competitiveness Trust Fund (incl. case studies)
Extended Care Career Ladder Initiative
• Research literature and other efforts in measuring business impact or
employer impact
– Kirkpatrick, Philips
– Aspen Institute ‘s Business Value Assessment
– Skills for Work project in the U.K.
Page 4
Challenges to Measuring Business Impact
•
•
•
•
•
•
History of focus on individual outcomes
WD professionals’ discomfort engaging employers
Different needs/measures for different industries
Lack of time
Many employers don’t manage with this kind of data
Measuring the value of training is challenging work
Page 5
How can Business Impact be measured successfully?
5 key steps in the process:
1.
2.
3.
4.
5.
Defining the problem
Identifying business impact measures
Gathering data
Analyzing and interpreting data
Using and sharing data
Page 6
Step 1: Defining the Problem
What is the problem/goal to be addressed
through the training intervention?
• Ideally, part of the project design phase
• Best pursued jointly by employer and WD staff
• Requires some understanding of employer context/trends
- market
- how training fits their needs
• Often involves multiple individuals from the employer
organization(s)
Page 7
Sector
Business Goals to be Addressed Through a Workforce Development
Intervention – Examples from Projects in MA
Healthcare

fill staff vacancies

reduce recruitment, hiring and orientation costs

improve client satisfaction
Financial
Services

improve the preparedness and increase the tenure of new hires
Hospitality

reduce turnover costs

improve the productivity of workers in areas such as safety and
customer service
Manufacturing 
improve worker productivity and performance to maintain and
improve business competitiveness
Human
Services

improve the preparedness of entry level new hires

reduce staff turnover
Page 8
Defining the Problem:
 Things to Remember
•
•
•
•
Consider business impact as early on in a project as possible.
Set up a meeting with management/other staff at each employer.
Prepare a set of questions to learn about your employer(s).
You needn’t be an industry expert, just be curious!
Working with multiple employers:
• Look for common goals
• Assess employer interest in engaging in the process
• Consider the proportion of workers who’ve been trained at each
employer
Page 9
Step 2: Identifying Business Impact Measures
Once the problem to be solved is identified ,
how will you know you’ve solved it?
•
•
•
Define “success” in the employer context.
What would change?
Where can you see change that is related to training?
It depends…
Page 10
Organizational Benefit
Work Processes
Improved
Application of
Knowledge &
Skills
New
Knowledge
& Skills
Training
11
Finding suitable measures
Talk to employer staff at different levels:
Leadership, Training/HR, Operating staff,
Supervisors, Participants
• If possible, use data that the employer is already
collecting and using.
• Focus on meaningful and feasible measures.
•
- Do the measures relate to training?
- Might the measures be strongly impacted by other factors?
- Can you get data on measures with reasonable time and
effort?
Page 12
Sample Business Impact Measures
Workforce-related Measures
Performance/Productivity Measures
•
•
•
•
•
•
•
•
•
•
•
•
•
Vacancy rates
Recruitment costs
Retention of new hires
Staff diversity
Ability to perform new tasks
Communication
Scrap /error rate
Machine set-up/run times
Safety
Application of writing skills
Customer/patient satisfaction
Supervisor/peer satisfaction
Time to satisfactory productivity
(among new hires/promoted
within)
Page 13
Intangible Measures
Some things don’t convert to numbers but still matter:
•
•
•
•
•
•
•
Improved work climate
Increased organizational commitment
Improved teamwork
Improved cooperation/reduced conflict
Reduced employee stress
Improved communication
Improved labor/management relations
Page 14
Case Study: Arwood Manufacturing
Using an existing measure to show impact
• Business Problem: reduce production and inspection errors due to
misinterpretation of blueprint specifications and production
validation
• Training Solution: Geometric Dimensioning & Tolerancing and
Blueprint Reading
• Indicator: error rate in documentation and inspection
• Results: a 61% reduction in error rates through improved quality,
inspection and documentation processes
Page 15
Example: Food Manufacturing
Workplace Education Program
Focus on Application of Skills
Survey to determine employees’ improvement in 5 key areas:
1. Expressing ideas and asking questions in English
2. Correctly identifying ingredients, tools, equipment used regularly
3. Using safety gear (gloves, goggles) as instructed
4. Using correct procedures to request time-off /call-in sick
5. Identifying problem areas (safety hazards, wrong ingredients)
Scale: greatly improved to not at all
Space for additional comments
This approach may seek individual or collective feedback.
Page 16
Example: Early Childhood Education
Using a cross-agency survey to gather data on impact
• Context: State policy increasing educational requirements for staff
• Project goals include:
-- Reduce staff turnover
-- Increase # of teachers pursuing credentials / higher ed degrees
-- Increase agency ability to meet staff qualifications for funding and
other opportunities that increase quality and teacher compensation
• Cross-employer Survey (see handout)
- organizational impact – significant to minimal - on:
employee stability, attitude/motivation, understanding of child
development, family relationships, instruction and more
- captures baseline and interim/final results
Page 17
Identifying Business Impact Measures:
 Things to Remember
• Work with employers to understand the context and
determine meaningful measures.
• Choose measures that relate to training.
• Go as far toward organizational benefit as you can.
• Intangibles are ok – but require evidence.
• Keep it simple.
• The point is to demonstrate value.
Page 18
Step 3: Gathering Data
Once you have defined your measures, how do you get data to see
if desired change has happened?
• Comparisons: pre- / post- training or comparison group
• Timing: When do you need data? When are data
available? When should you collect your own?
• Data sources: What do you have? What can you get?
What tools/agreements will you need?
• Roles and responsibilities: Who does what? When?
Page 19
Examples from MA projects
• Surveys:
baseline
interim for project feedback
pre-post comparisons at project end
• Collection of employer data:
quarterly updates
annual updates
final reporting
* keep in mind employer data calendars
Page 20
Healthcare Technician Training 1– Dollarizing Example
ACTIVITY CATEGORY
ADDITIONAL INFORMATION
ASSOCIATED COSTS
TIME (hours)
DOLLARS
Advertisement
Monster, Newspapers, in house
printing
Flat fee $8-900/Year
cost for technician listing (1-3/yr)
TBD
TBD
Hiring
Interview with Human Resources
HR Director/ Time 2-4 hours
@ $ ____
Interview with Department Director
Director __ hours
@ $ ____
Column data $164.00
omitted for
confidentiality
$82.00
Orientation to facility
HR/Department/Other Staff:
__ hours @ $ ____
$82.00
Background Check and other checks
HR Admin __ hours @ $ ____
$415.00
4-6 weeks orientation
Dept. Director __ hours @ $ ____
$246.00
Preceptor __ hours @ $ ____
$7,920.00
HR __ hours @ $ ____
$41
Other Staff __ hours @ $ ____
$33
New Technician (@ $ ____/ hr)
$5,280.00
Director of Nursing / Dept Director
__ hours @ $ ____
$492.00
Orientation
Credentialing
Total
252
$14,681.00
Page 21
Gathering Data:
 Things to Remember
• Establish a plan and timelines for data collection and reporting.
For each measure:
 How will data be gathered? ( e.g., survey)
 When will data be collected?
 Who will be responsible?
• Develop your plan with employer data timeline in mind.
• Don’t be afraid to send reminders!
• Do a test run if new data is being collected or a survey is being
used.
• Set confidentiality parameters: what can be shared/aggregated at
organization/project/community levels?
Page 22
Step 4: Analyzing and Interpreting Data
For each measure, review the data: what do they tell you?
•
•
•
•
•
What kind of changes took place?
o Did participant skills increase?
o Are people working differently?
o Did the employer experience a cost savings?
To what extent did the changes that were expected to result
actually occur?
How significant is the change(s) observed?
Could this change be due to other factors/events?
To what extent do changes meet the project’s business impact
goals ?
Page 23
Healthcare Technician Training 2 Example
Healthcare Business Impact Summary
Objective and Relevant Measures
Reduction in Vacancy Rates
Baseline Rate FY08 (pre-training)
Rate FY11 (post-training)
Reduction in Overtime
Baseline Overtime FY08 (pre-training)
Rate FY11 (post-training)
Reduction in Orientation
Baseline Orientation Time FY08 (pretraining)
Rate FY11 (post-training)
Increase in Retention Rates
Baseline new grad-6 months
Baseline new grad-12 months
Participant new grad–6 months
Participant new grad-12 months
Facility 1
Facility 2
Facility 3
9%
5%
13%
0%
13%
2%
$588,555
$548,000
$21,319
$16,441
$283,334
$194,678
6-8 months
6-8 months
6 months
6 months
6 months
6 months
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
Page 24
Analysis of Data
• Reduction in
- vacancy rates
- overtime expenses
• No change in
- orientation time
- retention
• Other factors influenced outcomes but training was a contributor
• Need more time to see impact on overtime and retention
• What about the program content may have affected outcomes?
What could be changed going forward?
• Additional benefit: opportunity to experiment with staffing model
for a high demand position
Page 25
Analyzing and Interpreting Data:
 Things to Remember
 Focus on one measure at a time.
 Ask “What happened?” before you jump to determining
what results mean for business impact.
 Involve stakeholders in the process of analyzing data at
an appropriate time.
 If you didn’t find the changes you’d hoped for, discuss
findings within your project; there may be a lesson in
there somewhere.
Page 26
Step 5: Using and Sharing Data
What can you do with all this new information?
• Business Impact data can be used for:
 Accountability and reporting to stakeholders
 Continuous improvement
 Feedback to trainers and project staff
• When sharing your data, place them in the context of
the overall goals the project sought to achieve and also
what else is happening in the environment.
Page 27
Using and Sharing Data:
 Things to Consider
• Consider your audience:
- What do they really want to know?
- What context will they need to understand findings?
- What level of detail will be most appropriate?
• What’s the best mechanism for communicating?
- written report
- presentation
• Respect your confidentiality/permission agreements
at organization/project/community levels
Page 28
Value of MBI
• Demonstrates the value to employers of investing in their employees




Improved employee skills
Cost savings and/or quality improvements
Address skill shortages or deficiencies in the workforce pipeline
Improved relationships with education/training providers and workforce
development agencies
• Shows how public sector investment in workforce development
improves competitiveness of companies
• Builds both a culture of and capacity for examination of value of such
investments
Page 29
Resources
• Getting Started Document
• Commcorp website: www.commcorp.org
Resources Tools  Assessing Business Impact
- Sample surveys, tools and presentations
• Business Value Assessment:
-- Tips for using questionnaires (and more)
http://www.aspenwsi.org/wsiwork-bvatool.asp
• How to Measure Training Results by Jack Phillips and Ron
Drew Stone, McGraw-Hill, 2002.
Page 30
Questions?
Page 31