Transcript Data Use vs. Misuse: The challenging nature of publicly

Data Abuse:
Now That You’ve Found Data,
What Are You Going to Do With It?
Barbara Needell, MSW, PhD
Center for Social Services Research
University of California at Berkeley
Larry Brown, MSW
Casey Family Programs
Presented to
Children's Roundtable Summit
Philadelphia, PA
September 23, 2010
CSSR Slides created by Emily Putnam-Hornstein
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
There are three kinds of lies:
Lies, Damned Lies and Statistics
^
Abused Statistics
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
Six Ways to Abuse Data
1) Rank Data
2) Compare Apples and Oranges
3) Use ‘snapshots’ of Small Samples
4) Rely on Unrepresentative Findings
5) Logically ‘flip’ Statistics
6) Falsely Assume an Association to be Causal
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
1) Rank Data
Two streets in Anytown, PA….
“Jane Doe is the poorest person
living on Moneybags Avenue.”
“Joe Shmoe is the wealthiest
person living on Poverty Blvd.”
It’s all relative…
And SOMEONE will always
be ranked last (and first)
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
2) Compare Apples and Oranges
Two doctors in Anytown, PA…
Doctor #1
Doctor #2
Doctor of the Year?
2/1000
mortality rate
20/1000
mortality rate
What if the populations served by each doctor were
very different?
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
3) Data snapshots…
Crime in Anytown, PA…
Number of Crimes
Period 1: 76
Period 2: 51
Average
= 73.5
No
change.
Crime jumped by 49%!!
Period 3: 91
Crime dropped by 16%
Period 4: 76
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
4) Unrepresentative findings…
Survey of people in Anytown, PA…
90% of respondents stated that they
support using tax dollars to build a new
football stadium.
The implication of the above finding is that there is
overwhelming support for the stadium…
But what if you were then told that respondents had
been sampled from a list of season football ticket
holders?
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
5) Logical “Flipping”…
Headline in The Anytown Chronicle:
60% of violent crimes are committed by men
who did not graduate from high school.
“Flip”
60% of male high school drop-outs commit
violent crimes?
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
6) False Causality…
A study of Anytown residents makes the following claim:
Adults with short hair are, on average, more than 3
inches taller than those with long hair.
Hair Length
X
Height
Gender
Finding an association between two factors does
not mean that one causes the other…
CENTER FOR SOCIAL SERVICES RESEARCH
School of Social Welfare, UC Berkeley
Now What Are You Going To Do With It?
(Con’t)
Larry Brown, MSW
Consultant to: Casey Family Programs
PA Roundtable - Philadelphia
September 23, 2010
Larry G. Brown, MSW – Improving Outcomes for Children, Families and CommunitiesTM
How Do I Use My Existing Data More
Productively?
• A Quick Review - Stock and Flow
• Unpack the Data
• Connect the Dots / Tell the Story / Look from
the Balcony / Use Multiple Indicators
• Compare with A Purpose (and with Caution)
• Data Soup: When Is It Ready to Use?
• It Takes Time To Change (2-4 years!)
Larry G. Brown, MSW – Improving Outcomes for Children, Families and CommunitiesTM
“Stock and Flow” Model
Placement Rate
Case Mix
Placement Setting
Discharge Location
Educational Status
Vocational Status
Medical Insurance
LOS
Visits
Stability
Discharge Planning/Prep
Courtesy of Chapin Hall Center for Children, University of Chicago
Re-Entry Rate
Employment
Housing Stability
Permanent
Connections
Unpack the Data
Admissions
Age At Admission
Decrease of
Admissions of 20%
over the last five
years
Numbers going down,
but is who’s coming
through the front door
changing?
Admissio ns b y Ag e
3 5.0 %
3 0.0 %
2 5.0 %
2 0.0 %
1 5.0 %
1 0.0 %
5 .0%
0
a
r1
9
M
c0
D
e
e
p
-0
9
9
S
n
-0
9
Ju
a
r0
8
M
c0
D
e
e
p
-0
8
8
S
n
-0
8
Ju
a
r0
7
M
D
e
c0
7
7
e
p
-0
S
n
-0
7
Ju
a
r0
6
M
D
e
c0
6
6
e
p
-0
S
n
-0
6
Ju
a
r0
M
c0
D
e
S
e
p
-0
5
5
0 .0%
Unpacking Data: Truancy Example
•
•
•
Finding: High Truancy Rate
• Hard to think of this as Actionable.
Unpack the Data -- Where’s the variation? Where do I act?
Male vs. Female (52-48%)
Elementary vs. Middle vs. High School (25% - 25% - 50%)
Caucasian vs. Hispanic vs. Black in School: (74% - 7% - 15%)
Race and Ethnicity - Truants
(32% - 17% - 47%)
Where do these kids live? GIS
Match Your Plan to Your Evidence:
– Two pronged attack: Intervention AND Prevention/Early ID
– Focus where the numbers are: Communities, Neighborhoods and Parents
Larry G. Brown, MSW – Improving Outcomes for Children, Families and CommunitiesTM
Connect the Dots
• Entry Rate
• Length of Stay
Both powerful indicators. Together,
they tell even more:
Entry
Rate
LOS
High
Short
Front Door
Long
Front-end
Safety &
Discharge
Planning
Low
Larry G. Brown, MSW – Improving Outcomes for Children, Families and CommunitiesTM
Focus
Pushing At What It Means;
Is There a Story Behind the Numbers?
Entering Care:
% Chng
Age
Apr-06
Apr10
0-5
2,151
1,648
-23.4%
13-21
4,358
2,277
-47.8%
Total
8,056
4902
-39.2%
0-5
27%
34%
13-21
54%
46%
Compare with a Purpose
• Who Are My Peers?
• Why am I Comparing Myself to Them?
• This gives me context, but tells me little
about MY system.
• But if there IS a good idea…
…STEAL SHAMELESSLY
Larry G. Brown, MSW – Improving Outcomes for Children, Families and CommunitiesTM
Data Soup: CPCMS, PPI, HZA
• Lots of new data
• Lots of clean up to do
• So, what CAN I use this for?
• And where do I start?
Larry G. Brown, MSW – Improving Outcomes for Children, Families and CommunitiesTM
For Starters: Be Active, Informed Users
•
Examine what each item tells you:
• Is it a point-in-time measure? (Cases Pending on a Date Certain)
–
–
–
Only tells me about that day (e.g., Age of Pending Cases)
Biased toward those that take a long time
Most often cited data; not always best
• Is it an entry cohort measure? (Cases Filed During a Period/Term: All
starting together)
–
–
Tells us how the system works as things move through a system
Highly actionable data; highlights places where intervention may make a difference (how long
for new cases to reach disposition; milestones between filing and disposition)
• Is it a discharge cohort measure? (Cases Terminated During a
Period/Term: All ending together, but not starting together)
–
–
–
Tells the story from the back end.
Caution in that many influences cloud what this means
Convenient, but not always easy to untagle
Active, Informed Users (Con’t)
• Repeat process with Child Welfare (HZA) data
• Compare like data across systems
– Entry data to entry data
– Point-in-time to Point-in-time
– Discharge cohorts to discharge cohorts
• Reach agreement over which questions these data answer:
– How many cases are in my workload? (point-in-time)
– How many new filings are made in a quarter? (longitudinal)
– How long have these cases been with us? (point-in-time)
– How long does it take for cases to reach disposition?
(longitudinal)
Larry G. Brown, MSW – Improving Outcomes for Children, Families and CommunitiesTM
Urgency vs
When Is It Good Enough?
• Valid and Reliable Data Needed
• Won’t get better until it’s used
• 80-20 Rule
• Include all stakeholders in
using/seeing data
• Kids Can’t Wait
Larry G. Brown, MSW – Improving Outcomes for Children, Families and CommunitiesTM
Implementation: Lots of Moving Parts
.
Performance Assessment
Implementation
Drivers
Systems
Intervention
Coaching
Training
Adaptive
Selection
Technical
Facilitative
Administration
Decision Support
Data System
Leadership
Graphics by Steve Goodman,2009
© Fixsen & Blase, 2008
So, Now What?
• Examine the inputs, processes, outputs and outcomes.
What are the most important pieces of this for your
initiative?
• Match the data to what you need in order to manage
(Less IS More -- Mies van der Rohe)
• Have you looked at the data beneath your indicators?
Lots of things hide in a summary variable.
• Are you connecting the dots across data indicators?
What is the explanation that describes the performance
you are getting from your system?
• Do you understand why you look the same or different
from others? Different is not necessarily bad!
So, Now What? (Cont)
• Stay patient and focused; it takes
time.
• Leadership counts.
• Data champions: not the most
technical, but rather the most
curious. (Data phobics get to play!)
Barbara Needell
[email protected]
510 290 6334
CSSR.BERKELEY.EDU/UCB_CHILDWELFARE
______________________________________________________________
Larry Brown Associates
[email protected]
(518) 370-9999 (office)
(518) 421-7271 (mobile)