Transcript Needs Assessment in CVME - UC Cooperative Extension

Needs Assessment:
Surveys
Dale A. Moore, DVM, PhD
School of Vet Med, UC Davis
Objectives
 Describe
the needs for needs
assessments
 Describe one way to conduct a needs
assessment
 Provide some examples
Needs For What?

Program topics
 Location
 Delivery Method
 Etc…
A need is a discrepancy
between what the desired
goal or standard is with what
actually exists.
Program Development

Identification of the Main Themes***
 Description of the Program Purpose
 Conduct Needs Assessment***
 ID others interested in development &
delivery of the program***
 Outline learning objectives***
 Select format and teaching methodologies***
 Select faculty / facilities***
 Evaluation tool***
 Marketing and promotion strategy*** / budget
Nancy Bennett, Dev CPE Programs, 1990
Traditional CE

Lecture-dominated format
 Little evidence of impact on behavior or
outcomes
 Minimal collaboration between learners and
providers
 Lack of timely response to learner needs
 Emphasis on credit
 Focus on course production driven by
enrollment economy
Don Moore, An approach to needs assessment in
Physician practices. Alliance for CME, 1998
New Paradigm in CE
 Increased
emphasis on learning
 Data-based educational activities
 Collaborative learning systems
 Blending of quality mgmt and CME
 Focus on improving patient outcomes
 CME as integral part of healthcare
system
Don Moore, An approach to needs assessment in
Physician practices. Alliance for CME, 1998
The Need For Needs
Assessment
 ID
learning objectives
 Data-driven education
 ID the standards of practice/knowledge
 ID the goals for outcomes &
stakeholder expectations
 ID the role of the educational program
in the practice of XXXXX
Needs Assessment for
Better Evaluations!
1
1
1
1
Needs Assessment: Which Way to Go?
Different Levels of Needs

Societal needs / client /
industry needs and
expectations
 Community needs
 Profession’s needs /
professional sector’s needs
 Farm’s needs
 Individual’s needs
Needs Assessment Methods

Environmental Scans



Survey




Review of literature
Discussion with experts
Mail survey
Telephone survey
Face-to-face interviews
Focus Groups
 Clinical audits / Practice audits
 Self-assessments
Surveys
 Advantage
of providing quantifiable
responses from randomly-selected
study participants
 Usually require that the response
choices are known by the investigator
in advance
 Issues of response-rates
List Frame for Surveys: WHO?
 Your
own/other database of previous
participants
 Company mailing lists
 Others?
Selection From the Listframe

How to randomly select?
 Random number generator in Excel
 Tools/Data Analysis/Random Number
Generator/OK/Variables……
 Sample size? EpiCalc 2000
 http://www.brixtonhealth.com/epicalc.html
Designing Your Survey and
Survey Questions
 Fowler,
FJ. (1995). Improving Survey
Questions: Design and Evaluation.
Thousand Oaks: Sage Publications.
 Dillman, DA. (2000). Mail and
Internet Surveys: The Tailored Design
Method. John Wiley and Sons: New
York.
Online Surveys
 ANR
Site Builder
 http://groups.ucanr.org/help/index.cfm
 The mechanics of putting a survey
together on-line
 http://www.surveymonkey.com/
Improving Response Rates
 Follow
Dillman’s Method
 Keep your promises to your
constituents
 Incentives – CASH
 Multiple contacts
 Respondent-friendly questionnaires
Improving Response Rates
 The
number of contacts Advance
letters, postcards, follow-up, additional
copies of questionnaires, telephone
calls (5)
 Salience
of the topic Relevance.
Common reason for non-response is
that survey doesn't mean anything to
the person.
Improving Response Rates
 Government
sponsorship: higher
response rates than private
organizations.
 Special third contact Follow-up of
advance letter / initial follow-up with
special mailing procedures: certified
mail or special delivery, personal or
telephone contact.
 Incentive on first contact: Incentives
included with first mailing increase
response rate. Incentives offered later
not effective. ($5 cash)
What does good questionnaire
design involve?
• IDENTIFY MEASUREMENT OBJECTIVES!
 • Draft questions based on objectives
 • Test questions
 – Expert review
 – Cognitive interviewing
 – Pilot run
 • Format of questionnaire

Respondent-Friendly Q’aires
 Not
too long
 Ask most important questions first;
demographic information last
 Unambiguous questions/responses
 Easy to complete
 Great directions -- subheadings
Critiquing/Drafting Questions
 Consistently
understood (avoid
ambiguous words, define key terms)
 Avoid “double-barreling” "(asking two
questions at once) “Would you like to be
rich and famous?”
 Avoid hidden assumptions
 Consistently administered and
communicated – interviewers read all
response options
Some Help With Questions
 http://www.statpac.com/surveys/surveys
.doc
 http://www.leeds.ac.uk/iss/documentatio
n/top/top2.pdf
 http://home.ubalt.edu/ntsbarsh/statdata/Surveys.htm#rsi
 Many others and books, like Dillman
9 rules for writing good
questions
 1.
Remember your survey's
purpose
 2. If in doubt, throw it out
3. Keep your questions simple
Example:
 "Imagine
a situation where the
production supervisor is away from the
line, a series of defective parts is being
manufactured, and you just heard that a
new client requires ten thousand of
these parts in order to make their
production schedule. How empowered
do you feel by your organization to stop
the line and make the repairs to the
manufacturing equipment?"
 4.
Stay focused - avoid vague issues
"When did you last see a movie?"
 5. If a question can be misinterpreted,
it will be
"What time do you normally eat dinner?"
 6. Include only one topic per question
 7.
Avoid leading questions "Most
doctors believe that exercise is good for
you. Do you agree?"
 8. Consider alternate ways to ask
sensitive questions Income, drug or
alcohol consumption and sexual habits.
 9. Make sure the respondent has
enough information
5 rules for obtaining usable
answers

1. Response options mutually exclusive and
exhaustive.
 2. Keep open-ended questions to a
minimum they pose problems in terms of
coding and analysis.
 3. People interpret things differently,
particularly when it comes to time Troublespots include responses such as "Always,"
"Sometimes" and "Never." Build in a temporal
frame of reference. "I am going to read a list of
publications. For each one, please tell me
whether you read it regularly. By regularly I
mean, at least three out of every four issues."
 4.
Consider a "Don't Know" response
Allow people to say they do not have an
opinion.
 5. Provide a meaningful scale End
points must be anchored with labels.
“Rate your satisfaction where 1 means
'Very Satisfied' and 5 means 'Very
Dissatisfied.‘ Odd number of points
provides middle alternative. Gain
nothing by having scale with > 7 points.
Survey Length
 30-40
questions
 Fewer the better
 Booklet form?
Survey Flow
Pilot-Testing
 With
peers
 With a small group of your audience
 Helps refine the tool
Surveys for Needs Assessment
 What
are your outcomes?
 What is your audience?
 Sampling frame, sample size, how to
select randomly
 Survey distribution
 Questionnaire design
 Data analysis