Transcript Tools & Techniques for Continuous Improvement

Common Measurements Tool
Client Feedback Workshop
Introductions
Welcome
 Introductions
 Objectives
 Agenda

Objectives
Learn why,when,how to use CMT
 Understand basics of survey research

Survey operationalisation, question design,
sampling, administration, analysis 101
Produce 2 draft surveys (Homecare,
Emergency Care)
 Produce action plans for implementation

Resources

To assist you
additional resource
material has been
provided in your
binder

CMT Managers Guide

Customer Feedback
and Customer
Satisfaction
Measurement - EPA

Detailed Handouts,
Articles, Case Studies
Why Customer
Feedback?

Performance
measurement is
basic element of
management.

Management
functions
 Plan
 Organize
 Lead
 Control
(Measurement)
Management’s
Contribution
Management performs
the functions of:
planning
organizing
leading
controlling
To coordinate the
behaviour of:
individuals
groups
organizations
Feedback
To attain:
individual effectiveness
organizational
effectiveness
Customer Service Model
Satisfied Customers
Leadership
Principles
Team-Oriented
Customer-Focused
Manage by Fact
Continuous Improvement
Enabling Environment
EXPECTATIONS OF SERVICE
QUALITY
80
56
60
40
22
22
20
0
Agree
Neutral
Agreement
Disagree
“What level of service should
you get from government,
compared to the private
sector?”
Percent of respondents
Percent of respondents
“Governments have a more
difficult task than the private
sector.”
80
60
45
52
40
20
3
0
Higher
About the
same
Level of service
Lower
Implement, Evaluate,
Repeat

Develop and implement action
strategies, test outcomes
Listen to Customers
Implement
Changes
Analyze
Results
Customer Feedback
Framework
Plan
Construct
Conduct
Analyze
Act
Step 1. Plan the Customer
Feedback Project

Define objectives of
feedback

Determine how
findings will be used
 What do I want to
 What will we do with
accomplish with this
feedback?
 Why am I conducting
this feedback activity
these findings
 Will they be used for
 performance indicator
 to improve a process
 to ID customer
needs/expectation
 planning, decision
making
What is most appropriate
feedback technique?

Survey is often not
the best technique
 Options include
 Focus Groups
 In Depth Interviews
 Customer Advisory
Panels
 Employee
Observations
 Performance Audits
 Customer Site Visits
 Mystery Shoppers
 Benchmarking
surveys
– see handbook for
description of strengths
and weaknesses
Strengths/Weaknesses
Employee Observations
Customer feedback, complaints
Service audits
 Internal customers
 “Walk-throughs”
Focus groups, interviews
Surveys
Two key questions

What information do you need?

Who (which segment of the population)
can provide you with this information?
Internal vs external customers
Demographic segmentation (age, income,
geography
Program function
Exercise # 1

In 2 groups answer the
following questions

What do you want to
know?
What technique would
you recommend?
Who would ask to
provide this
information?


Warning:

How ready is your organization for
customer feedback?
Do managers and staff sincerely intend to pay
attention and take action?
Are key managers committed to action?
Have staff members participated directly in
defining needs and approaches
Are there possible barriers to using feedback
effectively (concerns re: workload, negative findings,
change)
Top Reasons for Failure

Failure to involve the
right person in
developing customer
service measures
 customers
 sponsors
owners,users


Failure to capture data
that leads to action
Scorekeeping
vs.understanding


Failure to link
customer data to
operational, quality, or
performance data
Absence of closedloop process for
measuring and
managing satisfaction
Brandt,D.R. “Build Actionability
Into Your Customer Surveys”
1995
Customer Feedback
Framework
Plan
Construct
Conduct
Analyze
Act
Step 2.Constructing the
Survey






Types of Questions
Do’s and Don’ts
Scales
Ordering
Number of
Questions
Layout
Phase 2 Constructing your
survey

Operationalizing
your research
questions
 What kinds of
questions will get
you the information
you are looking for
 This crucial step is
often overlooked

Examples
 The Common
Measurements Tool
Common Measurements
Tool
Objectives:

Provide a ready-made tool for public
service organizations
 Enable like-organizations to compare
results
 Enable organizations to build internal
benchmarks
 Facilitate the sharing of information
gained and “lessons learned” between
organizations
Elements of the Service
Experience...

Client expectations

Client perceptions of service
experience

Level of satisfaction

Level of importance

Priorities for improvements
CMT - Service
Dimensions

Responsiveness
»
»
»
»

Timely delivery of service
# of contacts to receive service
Waiting time
Courtesy of service staff
Reliability
» What was promised was delivered
» Errors detected and corrected

Access & Facilities
» Hours of service
» Telephone access

Communications
Question Types

Open ended questions
allows respondents to provide his or her own
answer
– e.g.. What one thing could we do to improve service?

Closed ended questions
limit the response an individual can make
– e.g. (likert scale)

CMT includes both types
Closed Ended

Pro’s

Con’s
 Questions are
 no knowledge
relatively easy to
complete
 Easy to analyze
 Unlikely to receive
irrelevant or
unintelligible
responses
 may over simplify
 Risk “missing the
boat”:
For example if you asked “what is the
main reason for your visit?” and 30%
respond other.
This is a good reason to pre-test
questions to ensure you include all
the likeliest responses
Open ended

Pro’s

Con’s
 will receive more
 difficult to analyze
information, often
richer (intensity,
more specific)
 more likely to
receive surprise
insights
and code
 responses may be
irrelevant or
unintelligible
 demanding and
time consuming for
respondent
Guidelines
 Remember you want actionable info
what action could you take with this information
 Avoid questions which are
too vague, subject to misinterpretation
double barreled questions which imbed more
than one item
» on a scale of 1-6 please indicate how clear and
useful the materials are?
extraneous or irrelevant
sensitive
Avoid questions
that do not elicit responses which point to
specific remedies
– e.g. How satisfied are you with the quality of service provided
» Very satisfied
» Satisfied
» Neutral
» Dissatisfied
» Very Dissatisfied
which mix scales
– e.g. were you checked in within five minutes
» satisifed
» neutral
» dissatisfied
Avoid questions
which include terminology, jargon etc.
which includes words and terminology which is
too complex
» e.g. Are you aware of the impending
amalgamation of surrounding constituencies into
the “New Metro” catchement region?
with unclear frame of reference
»
»
»
»
e.g. what is your income?
“your” individual family, household?
Time reference bi-weekly, monthly, annual
Income - salary, tips or other sources
Impact of question
wording

See hand out for examples

Implied alternative
Considerations in wording





Are the words simple, direct, and familiar to
all respondents?
Are the questions as clear and specific as
possible?
Are the questions applicable to all
respondents
Are the categories mutually exclusive and
exhaustive
Are any questions leading or loaded?
Scale types

Likert scale
3,5,7,9 point

Thurston
Constructing
Questionnaire

Format
 Introduction - sets forth the purpose of survey
and guide respondent through questions
 Customer experience - establishes customers
level of knowledge regarding various parts of
questionnaire
 Measurement - CMT asks customer to
characterize experience, expectations, needs,
priorities
 Demographic information - classify
Ordering of Questions

Order of questions should
encourage respondents to complete
questionnaire
facilitate respondents recall
appear sensible
focus on the topic
follow a sequence that is logical to the
respondents
follow smoothly from one to next
Sensitive Questions

Sensitive Questions
 Introduce where respondent has
developed confidence and trust
 locate in section where they have most
meaning and context
 introduce slowly
Number of Questions

Be careful not to ask too many
questions
 CMT (100+)
 Do not recommend asking all
 11 core questions get 5 drivers,access,
communication and cost
– (see list in book)
Timing of Feedback

When you solicit feedback is very
important
too long after original event will lower response
rate and relevancy.
Be cautious re: decisions, administrative
process etc.
Exercise # 2

Group Exercise – Using CMT as a
guide each group will be given 1
hour to develop a draft survey based
on first exercise. Each group must
include a minimum of 5 CMT
questions and 5 Custom questions
Sampling
Census Vs. random
 What size sample do I need?

Depends on use of survey
Trade offs
Confidence Interval and sampling error
 How to calculate sample size
 Why 400 is the magic #

– (see handouts)
On-line tools
Pre-testing Questions

Where possible pretest your instrument.
 Ensure language is clear
Pre-testing is extremely
 Have you missed any
valuable, if you consider
how much time and
effort you have already
committed to customer
feedback
important items

Remember
Testing Methods
 Focus Groups/Interviews
 Ask test respondents to
“talk out loud”
Agenda Day 2
Review of Draft Surveys - Discussion
 Review of CMT
 Conducting Surveys (Who,When, Where,

How)
Analysis
 Action

Citizen-Centered Network
Common
Measurements Tool
Faye Schmidt
for the
IPAC
1999 Gold Award
ANAGEMENT DEVELOPMENT
A network of Federal/Provincial/Municipal
civil servants dedicated to improving
the quality of public service in Canada
Citizens First
Erin
Research Inc
THE CITIZEN-CENTRED
Citizens First
Five Drivers of Satisfaction
1. Timeliness
2.
3.
4.
5.
Competence
Courtesy
Fairness
Outcome
Customer’s Priorities for
Improving Access to
Government Services
Access a major weak spot
29% citizens can’t find service
89% experience at least 1 problem

Making government services easier to
locate in the phone book

Improving telephone service

Expanding electronic access

Expanding hours of service

Standards for routine services
Common Measurements
Tool
Objectives:




Provide a ready-made tool for public
service organizations
Enable like-organizations to compare
results
Enable organizations to build internal
benchmarks
Facilitate the sharing of information
gained and “lessons learned” between
organizations
Elements of the Service
Experience...

Client expectations

Client perceptions of service
experience

Level of satisfaction

Level of importance

Priorities for improvements
CMT - Service
Dimensions

Responsiveness
»
»
»
»

Timely delivery of service
# of contacts to receive service
Waiting time
Courtesy of service staff
Reliability
» What was promised was delivered
» Errors detected and corrected

Access & Facilities
» Hours of service
» Telephone access

Communications
The CMT

Go to tool
This graph
displays
satisfaction vs.
importance for
multiple service
issues
20 00-0 3-08
Cu sto mer Fe edb ack - C MT
22
Customer Feedback
Framework
Plan
Construct
Conduct
Analyze
Act
Step 3. Conducting
Surveys

Will Focus on Mail Surveys (80-90%)
 How to achieve a high response rate
 Data collection and coding
 Data entry options
–CMT Data Entry options
Mail Surveys

Mail surveys can achieve response rate
of 60%
 Begin with good data base
– unique id #, relevant customer characteristics,
mailing address, mail out dates and date responses
receive
 Professionalism in all Materials
– questionnaire, cover letter, return envelope with self
addressed stamp envelope; all professional look
Ensuring high response
rate
 Include cover letter; explaining purpose of
survey (print on official letter head)
 Booklet format recommended (see sample)
 Use stamp and hand written address
Objective is “coffee table adhesion factor”
 Follow up letter is a must
Generates an additional 50% response
Other tips
 Avoid mailing near
Survey Respnse Rate
50
40
1st Mail out
30
20
3rd Mail out
10
6
25
30
22
14
17
3
8
2nd Mail out
0
Dat
e
Responses
holidays
 use “hot-line” or
contact #
 use admin # and
explain this is to
avoid multiple
contacts
Telephone Surveys
Very common form of survey
administration
 Advantages are:

 faster
 offers more control
Call response ratio 10:1
 CMT provides Computer Assisted
Telephone Interview option

Internet Surveys

Advantages
 fast, cheap, automated data
collection/analysis

Disadvantages
 susceptible to sample biases
 too easy to administer
 security/privacy concerns
More info available on
request
Data entry options CMT

External data entry
contact Service 1st

Manual entry options
On-line survey
Data entry spreadsheet sheet
Bar coding
Customer Feedback
Framework
Plan
Construct
Conduct
Analyze
Act
Analysis

Quantitative
 Frequencies,
percentiles,
percentages
 Central tendencies
(mean, mode, median)
 Cross tabulations that
array independent
variables against
dependent variables
– e.g. customer type
who were satisfied
Cross tab example
Q: 5 Overall how satisfied were you with the service/product?
VD
D
N
S
VS
Total
Male
10
10
20
40
20 100%
Female
20
30
10
25
15 100%
7 Tools of Quality
MEASURING YOUR PROCESS: SIMPLE TOOLS

Cause and effect
diagram
 Check Sheets
 Control charts
 Flow charts
 Histogram
 Pareto charts
 Scatter Diagram
Tool
Use
Displays all possible causes of a particular problem in an
organized graphical fashion. Also called a fishbone diagram or
an Ishikawa diagram.
Cause and effect diagram
Delineates important items and characteristics, directs attention
to them, and verifies they were evaluated.
Check sheet
Determines process control status. Maintains statistical control.
Control chart
Organizes the steps in a process in a graphical manner. Shows
how inputs combine to create outputs.
Flowchart
Groups data and displays it graphically. Visually displays the
frequency distribution of data.
Histogram
Selects most promising opportunities for improvement by rank
ordering problems. Graphically displays the 80-20 rule.
Pareto chart
See Handout
Displays changes in one variable as another variable changes.
Illustrates correlation between two variables.
Scatter diagram
CMT Analysis

For each question
The CMT
 Bar/Pie charts
 Frequencies
 Percentages

For each section
 Satisfaction/Importance
 Actual Vs. Expected
 Priorities for
 Go to tool
This graph
displays
satisfaction vs.
importance for
multiple service
issues
Improvement

Service Standards
April, 2000
2000-03-08
Customer Feedback - CMT
22
Exercise 3

In groups of 4
review the Case
Study
 What are the main
findings?
 What are the clients
priorities for
improvement?
 What further analysis
do you recommend?
 How would you
present the results to
executive
management?
30 Minutes
Customer Feedback
Framework
Plan
Construct
Conduct
Analyze
Act
Acting on Results

What to do with the feedback?
 Step 1. Assess what the feedback is telling
you.
CMT for example allows you to evaluate 5
drivers of satisfaction, estimate service
standards, understand relationship b/w
satisfaction and importance and ID priorities for
improvement.
Acting on Results
 How good is
Type of Service
Re v e nue
Canada
Social
Se rv ice
Public
transit
100
80
60
40
20
0
Fire
 Satisfied vs..
Very satisfied
 4.0/5,0 vs.
4.5/5.0
 Nature of
organization
effects rating
SQ Ratings
Police
good enough?
Action Steps







Recover
Report
Brief
Prioritize
Communicate
Improve
Enhance

Reward
 Plan
 Feed into
Strategic/Business
Planning
Adapted from
Customer Service
Guidelines EPA
Defining Action Paths

Process involves
 identifying owners for
 linking issue to
each issue or area
targeted for
improvement
 Getting owners to
own readiness for
action
relevant processes,
systems
 using learning
achieved above to
define next steps
See Brandt
Satisfaction and Problem
Tracking
LOGGING
•Problem description
•Action required
INVESTIGATION
CLASSIFICATION
FORMULATION
•Bridge to Analysis
ANALYSIS
•Statistics
•Diagnostics
•Policy
PREVENTION
Training/Staffing
Process Improvement
Innovation
RESPONSE AND
FOLLOW-UP
REFERRAL OR SPECIAL
HANDLING
Technical Assistance Research
Programs (TARP)
Organizational Readiness
DO WE UNDERSTAND
WHAT CUSTOMER
IS TELLING
US?
DO WE HAVE A METHOD OF
ADRESSING THE
ISSUE?
GATHER ADDITIONAL
INFORMATION
NO
DEVELOP AND IMPLEMENT
SOLUTION
NO
BEGIN PROCESS
PERFORMANCE ANALYSIS
IS THE EXISITNG METHOD
WORKING?
NO
IS OUR TARGET ALIGNED
WITH CUSTOMER
EXPECTATIONS?
NO
ALIGN TARGET/SOLUTION
WITH CUSTOMER
EXPECTATIONS
Wrap - up
Review Objectives
 Tools and References
