Transcript Document

Understanding Event History Calendars
(Update on Reengineering SIPP)
COPAFS
March 5, 2010
David Johnson
Jason Fields
US Census Bureau
1
The Evolution of a Phoenix
2
Congresswomen Maloney hails 25th
Anniversary of SIPP, Census Bureau’s
Survey of Income and Participation
"October, 2008 marks 25 years of SIPP data collection. The
vital data collected by career professionals at the Census
Bureau allows for the evaluation of the efficiency and
effectiveness of government programs and gives us a more
robust picture of how well we are doing as a nation in
helping families progress through tough economic
challenges"
“The SIPP allows Congress to allocate scarce government
resources and save tax dollars. It’s fitting that during this
national economic crisis we draw attention to this important
diagnostic tool which helps us understand how we can best
provide assistance to families in need.”
3
The Unique Value of SIPP
• To provide a nationally representative sample for
evaluating:
– annual and sub-annual dynamics of income
– movements into and out of government transfer
programs
– family and social context of individuals and households
– interactions between these items
4
National Academy of Sciences
National Research Council
Committee on National Statistics
Panel Report
Reengineering the Survey of Income and
Program Participation
Constance F. Citro and
John Karl Scholz, Editors
July 2009
5
CNSTAT Report – Importance of SIPP
Conclusion 2-1:
The Survey of Income and Program Participation is a
unique source of information for a representative
sample of household members on the intrayear
dynamics of income, employment, and program
eligibility and participation, together with related
demographic and socioeconomic characteristics.
This information remains as vital today for
evaluating and improving government programs
addressed to social and economic needs of the
U.S. population as it did when the survey began 25
years ago.
6
Our work on SIPP Improvements
• Improve Processing System and Collection
Instrument
• Develop Event History Calendar (EHC)
Instrument
• Examine use of administrative records data to
supplement and evaluate survey data
• Continue meetings with stakeholders,
development of survey content, and use of
reimbursable supplements
7
Statements from the CNSTAT Report:
On EHC methodology
• As discussed in Belli (1998), in an event history
calendar, “respondents are encouraged to consider
various events that constitute their personal pasts as
contained within broader thematic streams of events.
Not only can respondents not the interrelationship of
events within the same themes (top-down and
sequential retrieval) but, depending on which themes
are represented by the calendar, respondents can
also not the interrelationships among events that
exist with different themes (parallel retrieval).”
8
Timeline for SIPP Development
2007
2008
2009
2010
2011
2012
Sep --- Jan --- May --- Sep --- Jan --- May --- Sep --- Jan --- May --- Sep --- Jan --- May --- Sep --- Jan --- May --- Sep --- Jan 2013
SIPP 2008 Panel – Waves 1 – 10 collection
Waves 11 – 13
SIPP 2008 Panel – Waves 1 – 13 data release
SIPP 2004 Data Gap
SIPP 2004 Panel data release
2008 paper EHC
Systems Tests Preparation
Processing and
Evaluation
Systems Tests Preparation
2013 Reengineered
SIPP Reference Period
Field Activities
2nd automated
prototype
Reference Period
2012/13 SIPP Re-Engineering
Instrument Refinement
Field Act.
2009 Re-engineered
SIPP automated
Prototype
Reference Period
Field Activities
2009 SIPP Re-Engineering
Instrument Dev.
Eval. Analysis
9
Re-engineered SIPP
– Progress Update –
COPAFS
March 5, 2010
Current SIPP Basics
National panel survey – Since 1984 with sample size between
about 11,000 and 45,000 interviewed households
The duration of each panel varies from 2½ yrs to 4 yrs
The SIPP sample is a multistage-stratified sample of
the U.S. civilian noninstitutionalized population
The survey uses a 4-month recall period – 3 interviews / year
The sample is divided into 4 rotation groups for monthly
interviewing
Interviews are conducted by personal visit and by
decentralized telephone
EHC Interviewing
Human Memory
- structured/organized
- links and associations
EHC Exploits Memory Structure
- links between the occurrence and timing of events
EHC Encourages Active Assistance to Rs
- flexible approach to help elicit an
autobiographical “story”
Evaluations of EHC Methods
Many EHC vs. “Q-List” Comparisons
- various methods
- in general: positive data quality results
BUT, Important Research Gaps
- data quality for need-based programs?
- comparison to 4-month reference period?
SIPP Re-engineering Field Test Plans
- Proof of concept test
- - 2008 paper and pencil reinterview test
- EHC CAPI test
- - 2010 Integrated Blaise and C# instrument prototype
- CAPI Revised test
- - 2011 Test improvements to the wave 1 instrument, training,
and expand sample to all regional offices.
- - 2012 Test wave 2 concepts and instrument, examine
movers and attrition issues, and refine training procedures.
2008 Paper Field Test Goals & Design
Basic Goal:
Can an EHC interview collect data of comparable
quality to standard SIPP?
- month-level data
- one 12-month ref pd interview vs. three
4-month ref pd interviews
- especially for need-based programs
Basic Design:
EHC re-interview of SIPP sample HHs
Approximately 2000 HHs in IL and TX
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year
•
equivalent data quality
•
SSI -- % Participation for Each Month of CY2007 According
to the SIPP and EHC Reports
10.0%
Analysis Summary
- no “main effect” for method (SIPP = EHC)
- no significant method difference in any month
% Yes
7.5%
5.0%
SIPP % yes
EHC % yes
2.5%
0.0%
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
DEC
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year
•
SSI; WIC (IL)
•
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year
•
SSI; WIC (IL)
• 2. EHC < SIPP All Year
•
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year
•
SSI; WIC (IL)
• 2. EHC < SIPP All Year
•
Medicare; Social Security;
WIC (TX); Food Stamps (IL)
•
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year
•
SSI; WIC (IL)
• 2. EHC < SIPP All Year
•
reduced EHC data quality, but
•
not due to longer recall period
•
SOCIAL SECURITY -- % Covered in Each Month of CY2007
According to the SIPP and EHC Reports
25.0%
Analysis Summary
- significant “main effect” for method (SIPP > EHC)
- method difference is constant across months
22.5%
% Yes
SIPP %…
EHC %…
20.0%
17.5%
15.0%
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
DEC
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year
•
SSI; WIC (IL)
• 2. EHC < SIPP All Year
•
Medicare; Social Security;
WIC (TX); Food Stamps (IL)
•
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year
•
SSI; WIC (IL)
• 2. EHC < SIPP All Year
•
Medicare; Social Security;
WIC (TX); Food Stamps (IL)
• 3. EHC < SIPP, Early in the Year Only
•
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year
•
SSI; WIC (IL)
• 2. EHC < SIPP All Year
•
Medicare; Social Security;
WIC (TX); Food Stamps (IL)
• 3. EHC < SIPP, Early in the Year Only
•
Food Stamps (TX); TANF (TX);
•
employment; school enrollment
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year
•
SSI; WIC (IL)
• 2. EHC < SIPP All Year
•
Medicare; Social Security;
WIC (TX); Food Stamps (IL)
• 3. EHC < SIPP, Early in the Year Only
•
EHC data quality may suffer due
–
to longer recall period
FOOD STAMPS (Texas Only) -- % Participation for Each
Month of CY2007 According to the SIPP and EHC Reports
10.0%
Analysis Summary
- no significant “main effect” for method
- BUT significant variation by month -JAN-MAY: SIPP > EHC
7.5%
% Yes
later months: no difference (reversal?)
5.0%
SIPP % yes
EHC % yes
2.5%
0.0%
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
DEC
FOOD STAMPS (Texas Only) -- % Participation for Each
Month of CY2007 According to the SIPP and EHC Reports
and ADRECS
10.0%
SIPP %
yes
EHC %
yes
% Yes
7.5%
5.0%
2.5%
0.0%
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
DEC
Results Summary
•3 Patterns:
• 1. EHC = SIPP All Year
•
SSI; WIC (IL)
• 2. EHC < SIPP All Year
•
Medicare; Social Security;
WIC (TX); Food Stamps (IL)
• 3. EHC < SIPP, Early in the Year Only
•
Food Stamps (TX); TANF (TX);
•
employment; school enrollment
2008 Paper Field Test Overall Summary
Successful “Proof of Concept”
Overwhelming Finding: SIPP-EHC Agreement
Valuable Lessons to Inform Next Test
- larger, broader sample
- “correct” timing of field period
- automated questionnaire
Specific Data Comparisons are Instructive
2010 EHC CAPI Test Goals & Design
Basic Goal:
Can a CAPI Event History Calendar interview be:
- developed in-house (new/unknown requirements)
- integrated with survey management systems
- administered by regular field staff interviewers
Develop and test new processing system
Determine the comparability of data collected:
- month-level data
- (1) 12-month ref pd intvw vs. (3) 4-month ref pd intvws
- especially for need-based programs
Basic Design:
8000 HHs interviewed in 10 states limited to high poverty strata that
can be matched to areas in 2008 Panel SIPP data.
Recording 400-500 of the completed interviews
2010 EHC CAPI Instrument
Current Status
Interviewing in 6 of 12 regional office areas (10 States)
Interviews continue through March 13.
Recording 400-500 of the completed interviews –
transcription will begin shortly.
Reviewing and making changes to content and design.
Reviewing and revising training materials and methods
Planning for 2013 Production implementation
Mockup (example of possible changes) for
2011 EHC CAPI Instrument
DEPENDENT DATA FILL UP
TO LAST WAVES ‘INTV’
MONTH
REFERENCE YEAR 20XX
LANDMARKS
RESIDENCES
FILL’S ARE
MARITAL STATUS – (CURRENT MARITAL STATUS fill)
USED TO
PRESENCE OF PARENT - MOM
LABEL
INTERVIEW
PRESENCE OF PARENT - DAD
MONTH
SCHOOL ENROLLMENT – (CURRENTLY ENROLLED fill)
STATUS OR
JOBS/BUSINESSES – (fill name of emp/bus from ehc as entered)
+ Job/Bus 2 – (fill name of emp/bus from ehc as entered) NAMES FROM
JOBS/PROGR
+ Job/Bus 3 – (fill name of emp/bus from ehc as entered)
AM NAMES
+ Job/Bus 4 – (fill name of emp/bus from ehc as entered)
+ Job/Bus 5 – (fill name of emp/bus from ehc as entered)
ALL OTHER WORK FOR PAY
TIME NOT WORKING
SSI - SUPPLEMENTAL SECURITY INCOME
SNAP - FOOD STAMPS/SUPPLEMENTAL NUTRITION ASSISTANCE PRGM
TANF - TEMPORARY ASSISTANCE FOR NEEDY FAMILIES
GA – GENERAL ASSISTANCE PROGRAM
WIC – WOMEN, INFANTS AND CHILDREN PROGRAM
HEALTH INSURANCE – PRIVATE COVERAGE
+ ADDITIONAL PRIVATE COVERAGE
MEDICARE COVERAGE
MEDICAID COVERAGE – (fill interview state program name)
MILITARY COVERAGE – (VA, CHAMPUS, CHAMPVA)
OTHER HEALTH INSURANCE COVERAGE
TIME WITHOUT HEALTH INSURANCE COVERAGE
INTERVIEW YEAR 20XX+1
INTV
DATA ARE RECORDED UP TO THE MONTH OF
INTERVIEW, WE ALLOW UP TO 6 MONTHS FROM THE
END OF THE CALENDAR YEAR TO CONDUCT
INTERVIEWS – MONTHS BEYOND THE INTERVIEW ARE
GREY’ED OUT.
(FILL NAME OF PERSON FOR THIS EHC)
Topic – (FILL TOPIC LABEL)
* INTRO TO TOPIC QUESTION FILL – ASKS ABOUT STATUS NOW OR DIRECTS FR TO PROCEED
* FOLLOW-UP ASKS ABOUT EVER DURING REFERENCE YEAR OR DIRECTS FR TO PROCEED
[
[
] (1) YES/PROCEED (2) NO (3) REFUSED
] (1) YES/PROCEED (2) NO (3) REFUSED
*
IDENTIFY NEW PERIOD OF TIME
[
] 0-NEW
*
*
WHEN DID THIS PERIOD OF (TOPIC FILL) END?
WHEN DID THIS PERIOD OF (TOPIC FILL) BEGIN?
[
[
] CHOOSE MONTH
] CHOOSE MONTH
[
] (1) YES (2) NO
ANY MORE PERIODS OF (TOPIC FILL) DURING REFERENCE YEAR?
RECORDING ‘NO’ ADDS STRIKEOUT TO ANY UNASSIGNED
MONTHS IN THIS TOPIC – ANSWERING ‘YES’ CYCLES BACK
TO NEW PERIOD – UPON SPELL COMPLETION YOU EXIT
THE DETAILED QUESTIONS TO THE ‘ANY MORE’ ITEM.
[F4 – EDIT SPELL] [CTRL-X DELETE SPELL]
IF YES TO ‘NOW’ INITIALIZE
FIRST PERIOD AND FILL
INTERVIEW MONTH IN THE
‘TO’ BOX AND SET FOCUS
ON THE ‘FROM’ BOX
Assessing Users’ Needs
URL: http://www.census.gov/sipp
Comments:
Jason Fields – [email protected]