Transcript Inclusion Outcomes and Indicators of Success

Inclusion Outcomes and
Indicators of Success
Holly Matulewicz
Institute for Community Inclusion, ICI
(617) 287-7640
[email protected]
Lucy Bayard
National Service Inclusion Project, ICI
(617) 287-4355
[email protected]
www.serviceandinclusion.org
Toll-free hotline: 888-491-0326 (voice/TTY)
Training and Technical Assistance





















Values that Guide & Current Best Practices on Inclusion
Person-first Language & Inclusion Etiquette
Who are People with Disabilities?
Who is a "Qualified" Person with a Disability?
To Disclose or Not to Disclose – What is the Impact?
Creating an Environment that Encourages Disclosure
How to Conduct Effective Outreach & Recruitment to People with Disabilities
Interviewing: Welcoming & Appropriate Questions & Statements
Understanding the ADA & Section 504
Beyond the Laws: Creating a Welcoming, Inclusive Environment
Accommodation Strategies & Adaptive Products: No Tech, Low Tech & High Tech
Demonstrations & "hands-on" activities of how to use Accommodation Products
How to develop Inclusive Service Descriptions
How to Ensure Access for Everyone: architectural, programmatic, communication, &
alternative format access
Member & Volunteer Management - supervising & performance concerns
Recruitment & Networking Resources: Disability Organizations and Agencies
Preparing for Accessibility Site Visits
Reasonable Accommodations: definition & responsibilities
Disability Benefits: Supplemental Security Income & Social
Security Disability Income
Supporting Inclusion through Community Asset Mapping
Emergency Preparedness for People with Disabilities
Objectives:
To develop an understanding of:
• Indicators of successful inclusion
• Addressing program issues with datadriven management
• Measuring constructs
• Introduction to quantitative measure
• Introduction to qualitative measures
What is inclusion?
Inclusion means that all people, regardless of their
abilities, disabilities, or health care needs, have
the right to:
• Be respected and appreciated as valuable members of
their communities
• Serve as a member or volunteer in Senior Corps,
AmeriCorps or Learn and Serve America programs
• Work at jobs in the community that pay a competitive
wage and have careers that use their capacities to the
fullest
• Participate in service learning opportunities with peers
from elementary school through college and continuing
education
What are indicators of successful
inclusion?
• Individuals with disabilities are actively
recruited and welcomed at the
organization / service site
• Availability of accommodations is openly
publicized and reasonable
accommodations are provided upon
request
• Individuals with disabilities assist
in reviewing materials & policies
What are indicators of successful
inclusion?
• Buildings and programs are accessible
• Interviews, meetings, events and social
gatherings are held in accessible locations
• Individuals are asked about their experience and
satisfaction
• Individuals evaluate the effectiveness of
products and strategies
• Materials available in alternative
formats such as large print,
electronic, Braille etc.
What does successful inclusion look like?
Members in leadership roles:
• In addition to her service, Anna, an AmeriCorps
member, serves in a leadership role on Nebraska’s
InterCorps Council, which connects all AmeriCorps
programs in the state. She conducts peer-grant
reviews, and plans events and statewide service
days.
• Justen has been an AmeriCorps*VISTA for two
years and, in his third year, has become a VISTA
Leader in his Corps. He provides support to 32
members and fulfills the goals of the organization.
Presentation Overview
• Addressing program issues with data-driven
management
• Measuring constructs
• Introduction to quantitative measures
– administrative data
– surveys
• Introduction to qualitative measures
– 1:1 Interviews
– focus Groups
– field Observation
Part I.
Addressing program issues
with data-driven management
What is “data-driven” management
Integration of data into
your management
practices
Why is it useful?
•
• Setting measurable goals &
measure progress towards
them
Make decision based on evidence
not instinct, assumptions, or
perceptions
•
Have more information to use for
analyzing issues / developing
solutions
• Setting benchmarks or
standards
•
Helps managers and funders see
big picture, accountability for
outcomes
• Using data in presentations to
staff, funders
•
Helps identify trends over time
•
Provides benchmarks for staff
Using data-driven management
• How am I already using this today?
– Do your goals have measurable outcomes?
– What data do you use to measure progress?
– How often do you review data for:
• Member/ volunteer level outcomes
• Agency-level outcomes
• State-level outcomes
– How often do you share these data with
• Members / volunteers, staff, agency upper management,
funders
• How can I do more in the future?
How many programs can answer
these questions?
• Using the data you collect now – can you report on:
– # of members / volunteers in each program
– # of partnerships made between National Service and Disability
organizations
– # of applications for National Service of persons with disability
– # / Type of recruitment efforts to disability community
– Disability Organizations – type / volume of info shared about National
Service opportunities
– Change in these numbers over time
Strategies for becoming “outcomes
driven”
•
Focus on data that matter to you
•
Nurture the “inquisitive mind”
•
Help others see the benefits of using data
•
Build systems / procedures for enhancing data
quality
Part II.
Measuring constructs
Building Blocks of
Data-driven Management
•
Data is not as scary as people may think!
•
To gather data efficiently and effectively
– follow some basic steps:
1.
2.
3.
4.
Identify the “construct” to measure.
Conceptualize the construct.
Operationalize the definition.
Develop method for: collecting, entering, analyzing,
and reporting these data.
Step 1.
Identify the construct to measure
• Looking at your member / volunteer, staff, and agency
outcomes:
– What questions do you want to answer?
– What information is needed to answer the question?
• Classifying units of analysis (say individuals) into
categories (satisfied with course not satisfied with
service experience).
• Often times we want to “measure” things that in and of
themselves are intangible in the social world
(“satisfaction” with course, “quality of life,” etc).
Step 2.
Conceptualize the construct
• Process of specifying what we mean when we
use particular terms. Begin by clarifying what we
“mean” by a concept.
– Example: “Quality of experience” among members /
volunteers.
• What does that “mean” to those interested in measuring it?
• What does that look like? What are examples of it?
• Produces an agreed upon meaning for a concept
for the purposes of research.
Step 3.
Operationlize your definition
• Things like “satisfaction with life” or “fear of
crime” are hard to measure directly, so we have
to make inferences.
• Process of defining specific ways to infer the
occurrence of specific phenomena.
• Indicators are observations we think reflect the
presence or absence of the phenomena to which
the concept refers.
– How do we “know it when we see it” or “when
someone experiences it”
As you develop your measures
• Good research / evaluation strives for both reliability and
validity. Applies to qualitative and quantitative
measures.
– Reliable: Using your method – could others replicate your
research and get similar results?
– Valid: When operationalized – will your measure “truly show” the
concept you want to study?
Earl Babbie, Basics of Social Research, 2002
Step 4.
Select a data collection strategy
•
Once you have gone through these steps:
1.
2.
•
Today we will briefly cover 5 strategies to collect data:
–
–
•
Are you already gathering these data?
If not - select a data collection strategy
Quantitative: case record abstraction and surveys
Qualitative: 1:1 interviews, focus groups, field observation
Meant as introduction to key tasks – not exhaustive
summary!
Two kinds of data:
Qualitative and Quantitative
• Data come in many shapes, sizes, and formats.
Distinction is between numerical (quantitative) and nonnumerical (qualitative) data.
• Different data will be needed to:
– Answer different types of questions
– Measure different kinds of outcomes
• These two types of data
– Require different data collection and analysis techniques.
– Both are useful and valuable tools.
– Each have advantages and disadvantages.
Qualitative Vs. Quantitiave Data
• Most all data start out as initially a
qualitative measure - must be quantified
so that the researcher can perform
statistical analysis.
• Much controversy about whether using
qualitative or quantitative methods and
data will “prove the point” better.
Reporting on your data
•
Determine your audience and craft report accordingly.
– Who will hear or read this report - how much background knowledge do they
have?
– What questions will they bring to the table?
– What topics are of most interest to them?
– What do you want them to “take away” from the findings
•
Present your question or point of evaluation clearly for your audience.
– Use jargon / lingo appropriate to the audience: Same data may be packaged
differently.
– Identify themes, recurring ideas, or common experiences
– Ensure the report is true to the data – not just highlighting those supporting your
ideas
– Map out key findings visually (charts / graphs)
•
Clearly state your data collection processes.
– Methods transparent: audience should be able to use your process to replicate
results.
– Point out any qualifications or conditions (shortcomings).
Part III.
Introduction to quantitative measures
Quantitative Data
•
In numerical format naturally
•
Coded / assigned numerical values for analysis
•
Used for:
– monetary values, counts, dates
– placement= 1, non-placement=2
– coding into categories with numerical representations (e.g. 1 = yes, 2 = no).
– Describing information in aggregate, identifying trends overtime, quantifying
outcomes, conducting statistical analysis
•
Able to be analyzed using statistics
•
Examples include:
– # clients served at agency, # staff, % of staff with BA degrees, % of clients
placed in jobs, agency cost per client served
Advantages & disadvantages of
using quantitative data
Advantages
• Aggregate large volume of
data
• Numbers can be “persuasive”
• Track trends over time
• Measure relationship between
different variables
Disadvantages
• Training / expertise required to
collect, enter, and analyze
these data
• May not shed light on the
“whole story” or the “why” of a
situation
• Participants may feel limited to
preset response categories
Administrative Data or
Case Record Abstractions
• Collection of existing data from case records or
files:
– “Abstracting” key variables from the data for reporting
/ analytical purposes
– Because the data are not collected for research
purposes – files may have missing data or vary in
how items were recorded
• Unobtrusive research
– study social behavior without affecting it
Value of Record Abstractions /
Administrative Data
• Minimal costs, effort to gather these data – no burden on
participants
• Provides a snapshot of key indicators for your state or agency
• Gives you ability to aggregate these data for snapshot on progress
towards goals / outcomes:
– Recruitment: # or types of agencies tapped for recruitment
partnerships, # or type of sites where applications were distributed
– WEBBERS on members / volunteers: gender, age, race, education
– Agencies: # and type of service opportunities members / volunteers
placed in
What is a survey?
• Way to collect standardized information from large group
of individuals.
• Collection of data from a scientifically selected group of
people. Results can be representative of a larger
population.
• Data collected are used to address specific issues.
• A standard set of procedures are followed.
Advantages and Disadvantages
of Surveys
Advantages
• Collecting original data on
population too large to observe
directly
• Results can be generalizable to
whole population (when using
scientific sampling methods)
• Paper / web give respondents
the flexibility to return data at
their convenience
Disadvantages
• Can be extremely costly to
conduct
• Item non-response and unit
non-response
• Accounting for sampling bias
(based on mode), can leave
out some members of the
population (reading level, non
telephone household, nonEnglish speakers, persons with
disability)
4 Modes of Survey Administration
1. Self-administered: Mail
2. Self-administered: Web
3. Interviewer Administered: Telephone
4. Interviewer Administered: In Person
1. Mail Surveys
• Most common mode of survey data collection
• Low in cost
• Response Rates generally low – need multiple waves of
follow-up
• Used a great deal in business surveys when directed at
specific groups (such as members of professional
organizations)
• Who does it exclude?
2. Web Surveys
• New technology, seen most prevalently in convenience samples
• High costs associated with programming, yet once programmed:
– data available immediately
– structure quex. In such a way to eliminate item non-response (benefits
/ drawbacks of doing this)
– Respondents can answer at any time, like paper instrument
– Response options can be personalized based on previous responses
• Currently still LARGE bias in general population using web mode
alone, therefore not recommended (alone) for general population
study
3. Telephone Surveys
• Most large-scale surveys in the US are conducted by
telephone using CATI - improves the quality of the
data collection
– Must be tested for correct routing / branches of Qs
– Avoids an important error - omissions!
• Can increase cooperation rates.
• Faster, less expensive than in-person interviewing.
• Who doesn’t it reach?
– When might this be a problem?
4. In-Person Surveys
(Interviewer Administered)
• Presence of interviewer may have effects.
– Increase in cooperation
– Possible to get immediate clarification on issues in the
instrument
– Possible for bias because of interviewer presence
• High quality of data - training the interviewers in a
classroom like environment
– Good interviewing techniques stressed
• Professionalism
• Avoiding bias
• Why might this mode not get used as often?
Regardless of mode:
Use Advance / Cover letters
• Key Components:
– Explain the purpose of the survey
– Organization sponsoring the survey & any relevant endorsements
or supporters
– Lets person know you will be contacting them (or they may
contact you) with any relevant details about items needed for
survey
– Provide details on deadlines or submission requests
• Write from reader’s perspective: “Why should I
participate?
Schedule of survey data collection
• Set up your calendar with mail dates
• Identify total field period (start to finish)
–
–
–
–
–
Allow sufficient time to
Prep mailings
Recruit, hire, train interviewers
Design / test web survey
Process survey returns / enter data
• Goal: work backwards from end goal or deadline
Preparing to field your survey
• If interviewer-administered (on phone or in person) you must
hire interviewers and supervisors.
– Train them on your survey
– Ensure they have basic interviewer training
– Specify DC schedule, QC rates, production rates, and response rate
expectations.
• For mail surveys, training staff on schedule, receipt and followup procedures
• For all methods: developing QC process check on completion of
work, including collecting and editing documents.
Once the quantitative data are
collected …
•
Once the data have been: quality checked, edited, and entered you can
begin your analysis! Your analysis should focus on answering the questions
you posed when designing your data collection forms (abstractions or
surveys).
•
Spreadsheets may be useful to you for
–
–
–
–
•
Simple entry procedures
Few case records
Ease of reporting
Disadvantages of spreadsheets?
Databases may be useful to you for
– Simple entry, once entry form is designed, Minimizing entry error
– Ability to: link several datasets, program reports into database / create on-line
for field access
– Disadvantages of databases?
Part IV.
Introduction to qualitative measures
From survey to ethnography...
• This model of data collection allows for more
freedom … not making the respondents feel
“boxed in” to prescribed answer categories.
• Using a standardized format implicitly assumes
that all respondents will understand and
interpret the questions in the same way
• Structure can limit researchers ability to gain in
depth knowledge about an issue
Qualitative data
• Qualitative data are non-numerical
• Used for:
– Examining social world through stories, images, and experience
– Probing more deeply into constructs, examining the “how” or
“why” types of questions
• Examples include:
– Transcripts from 1:1 or group interviews
– Observations made in the field
– Pictures, texts
Why aren’t qualitative data used
more?
• Capturing and analyzing qualitative data
sets has been a tough business.
• Extremely costly process, quite time
consuming, often necessitating small
sample sizes.
• “Numbers” can be perceived as more
persuasive.
1:1 Qualitative interviews
• Interaction between participant and interviewer
where interviewer has “general plan” of inquiry –
but not set questions
• No specific order of questions
• Interviewer must be well trained, very
knowledgeable in subject matter (for probing)
• Essentially a “conversation” but participant does
95% of the talking
Advantages and disadvantages of
1:1 qualitative interviewing
Advantages
•
Participants share info in 1:1
format may not share in group
•
Allow participant to explore
concepts more freely / fully
•
Researchers not limited to script
or preset response categories
•
•
Great for exploratory work where
you may have limited info on topic
Focus on verbal and non-verbal
cues.
Disadvantages
• Relies heavily on skill and
knowledge of interviewer
• Costly to implement – per
interview costs may limit
sample size
• Due to small number, limits to
generalizability
• Large volume of data to
transcribe / analyze
Focus groups
• Group interviews - as they are like in-depth interviews
– Guided discussion on topics of interest
• Purpose is to explore rather than describe or explain in a definitive
sense
– Group of 7-12 people too atypical to generalize to whole population
• Very flexible form of DC, allow participants to frame answers and
construct meaning as they wish
• Examples:
–
–
–
–
member / volunteer service experience: successes, challenges
application experience: how heard of opp, why appealed, app process
retention issues: why left service, what can be changed
agency partnerships: quality of service provided by members /
volunteers
Advantages & Disadvantages
of Focus Groups
Advantages
•
Socially oriented research
method
•
Flexible – group may raise topics
researcher didn’t foresee or
anticipate
•
Speedy results
•
Low in cost
Disadvantages
•
Less control than individual
interviews. Tendency to produce
“group think” where people may
not readily express ideas that
deviate from group’s.
•
Data can be difficult to analyze.
– Difference between groups can be
troublesome.
•
Moderators must be skilled and
discussion must be conducted in a
conducive environment.
•
Groups are difficult to assemble.
Field Observation
• Methods of collecting data on people, likely in their
natural settings.
• People from somewhere going somewhere else &
sharing what they find
• “Informant” who gives you your data (like a narrator)
– Participant observation: performed by those who take part
in the activities they observe. Gain “verstehen” by immersing
themselves in the daily lives of those they study.
– Non-participant observation: made by an observer who
remains as aloof as possible from those being observed.
Using Field Observation
• Decide on a topic where field observations are
appropriate
• Identify your research questions and constructs to
measure
• Can include narrative and quantitative measures
• Examples can include:
– observation of worksite or agency
– observation of member / volunteers with those they serve
– Attending recruiting events – observing candidates and
recruiters
Advantages and disadvantages of
field observation
Advantages
• Direct observation, rather than
descriptions or interpretations
(via interviews) from
participants’ bias /
perspectives
• Data can richly supplement
other sources of info
Disadvantages
• Disruption of natural setting
• Time consuming / labor
intensive
• Relies heavily on skills of
observer
• Can rely on honesty – level of
disclosure of informants
• Can have ethical dilemmas
(participant vs. nonparticipant)
• Act of study can change
behavior of those observed
How can I assess if my program is inclusive
and accessible?
An accessibility checklist provides
guidelines for assessing your
program(s) to help ensure :
• compliance with the law
(Section 504 of the
Rehabilitation Act and Title II
of the Americans with
Disabilities Act)
• how to create an environment
that makes people with
disabilities feel welcome
• how to design programs and
services so that people with
disabilities can fully participate
Tips for conducting accessibility checklist at
your organization / program
• Involve Program Directors, service site supervisors, and
all relevant staff at site / organization
• Provide an opportunity for members, including members
with disabilities, to provide feedback and share their
experiences regarding accessibility and inclusion
• Be willing to collaborate with disability organizations in
your community to access resources and assistive
technology
• Assessing your program and improving areas in need
may take time, so it is important to keep it as a priority
and be patient!
Thank you!
Holly Matulewicz
Institute for Community Inclusion, ICI
(617) 287-7640
[email protected]
Lucy Bayard
National Service Inclusion Project, ICI
(617) 287-4355
[email protected]
www.serviceandinclusion.org