Transcript Common Sources of Error - Australian Council for

Common Sources of Error
Data Management Seminar, 9-12th July 2007, Hamburg
Codebook Adaptation
Instrument
Preparation
2
Data Sendout
S
U
R
V
E
Y
Receiving Material
Sampling
and Preparing
Survey Administration
Data Entry
and Verification
Marking
Data Management Seminar, 9-12th July 2007, Hamburg
Codebook Adaptation
Instrument
Preparation
3
Data Sendout
S
U
R
V
E
Y
Receiving Material
Sampling
and Preparing
Survey Administration
Data Entry
and Verification
Marking
Data Management Seminar, 9-12th July 2007, Hamburg
Errors During Instrument Preparation
Errors in translations (especially: careful if cooperate
with other countries)
The same terms in different questions translated
differently
Comments from translation verification not implemented
National adaptations not or incorrectly documented on
NAF
Deletion of international questions and categories
without strong reason and approval
International question stem changed
4
Data Management Seminar, 9-12th July 2007, Hamburg
Errors During Instrument Preparation
(ODC)
Conversion in SurveySystem started before paper
instruments are reviewed  start AFTER international
verification
Copy and paste to wrong text, numeral, or instruction
 check paper version against preview
Paper questionnaires and ODC questionnaires do not
match in terms of disabled questions, variables or
additional elements  verify within country before
submitting to IEA DPC
5
Data Management Seminar, 9-12th July 2007, Hamburg
Codebook Adaptation
Instrument
Preparation
6
Data Sendout
S
U
R
V
E
Y
Receiving Material
Sampling
and Preparing
Survey Administration
Data Entry
and Verification
Marking
Data Management Seminar, 9-12th July 2007, Hamburg
Errors During (Within-school)
Sampling
School sampling frame incorrect / major parts of the
population not included
Sample not approved by the IEA DPC
Replacement of ineligible schools (frame errors like
closed schools)
Missing selection of participation in Regional Module or
On-line data collection when creating new project in
WinW3S
Schools did not list all eligible students and teachers or
left out eligible but excluded them  follow-up
7
Data Management Seminar, 9-12th July 2007, Hamburg
Codebook Adaptation
Instrument
Preparation
8
Data Sendout
S
U
R
V
E
Y
Receiving Material
Sampling
and Preparing
Survey Administration
Data Entry
and Verification
Marking
Data Management Seminar, 9-12th July 2007, Hamburg
Errors During Codebook Adaptation
Length of a national variable specified to small
(one digit for a variable with 6 or more options)
Wrong labeling/numbering of categories or dimensions
when they were deleted or added
Deletion of variables for questions that are not
administered  set to „hide“ in WinDEM
Codebooks for different data entry computers are
different
If ODC is used, codebooks for WinDEM do not match
with ODC data (changes made after export from
SurveySystem)
9
Data Management Seminar, 9-12th July 2007, Hamburg
Codebook Adaptation
Instrument
Preparation
10
Data Sendout
S
U
R
V
E
Y
Receiving Material
Sampling
and Preparing
Survey Administration
Data Entry
and Verification
Marking
Data Management Seminar, 9-12th July 2007, Hamburg
Errors During Administration
Instruments distributed to wrong person (only to
designated individuals)
Sampled student/teacher replaced  do not substitute!
Participation status not tracked (Tracking Forms) or
monitored (On-line Monitor Report)
Spare Booklet assignment not documented (Tracking
Forms)
11
Data Management Seminar, 9-12th July 2007, Hamburg
Codebook Adaptation
Instrument
Preparation
12
Data Sendout
S
U
R
V
E
Y
Receiving Material
Sampling
and Preparing
Survey Administration
Data Entry
and Verification
Marking
Data Management Seminar, 9-12th July 2007, Hamburg
Errors During Material Receipt
Receipt of materials not tracked appropriately
Incomplete materials  contact school coordinator and
try to resolve
Incomplete materials because of low on-line participation
 consider following up to schools with paper
questionnaires
If ODC used, “Monitor” reports not taken into account
continuously
13
Data Management Seminar, 9-12th July 2007, Hamburg
Codebook Adaptation
Instrument
Preparation
14
Data Sendout
S
U
R
V
E
Y
Receiving Material
Sampling
and Preparing
Survey Administration
Data Entry
and Verification
Marking
Data Management Seminar, 9-12th July 2007, Hamburg
Errors during Marking
Single items are all marked by the same marker
Reliability booklets are marked by the same marker in
the instrument and in the reliability marking sheet
Main markers enter codes directly in reliability booklets
before reliability marking has been done
15
Data Management Seminar, 9-12th July 2007, Hamburg
Codebook Adaptation
Instrument
Preparation
16
Data Sendout
S
U
R
V
E
Y
Receiving Material
Sampling
and Preparing
Survey Administration
Data Entry
and Verification
Marking
Data Management Seminar, 9-12th July 2007, Hamburg
Errors During Data Entry in WinW3S
If ODC was used, questionnaire type not tracked
correctly in WinW3S
Participation status not entered in WinW3S
Spare booklet assignment not performed in WinW3S
ITLANG in NAF and WinW3S not set if more than one
language was used
17
Data Management Seminar, 9-12th July 2007, Hamburg
Errors During Data Entry in WinDEM
Not all materials entered
Wrong or duplicate IDs, column shifts, invalid values
(All avoided by using WinDEM)  train staff thoroughly
Wrong treatment of out-of-range values  enter data
“as is”!
Missing values used incorrectly (use ‘9’ for any omitted
response, ‘7’ for any invalid response and valid values for
all interpretable responses)
Data entered on different computers not combined or
combined multiple times (leading to duplicate IDs)
Values for Check-All-That-Apply questions used
incorrectly (use 1 for CHECKED, 2 for NOT CHECKED)
Filters interpreted  enter data “as is”!
18
Data Management Seminar, 9-12th July 2007, Hamburg
Errors during Verification
WinDEM checks not performed
Checks performed but data not corrected
Manipulating the data files with MS Excel (data file will
be cut of after 255 variables)
19
Data Management Seminar, 9-12th July 2007, Hamburg
Codebook Adaptation
Instrument
Preparation
20
Data Sendout
S
U
R
V
E
Y
Receiving Material
Sampling
and Preparing
Survey Administration
Data Entry
and Verification
Marking
Data Management Seminar, 9-12th July 2007, Hamburg
Errors during Data Send-out
Data submitted, but without documentation (Tracking
Forms, NAF)
NAF or instruments not sent for all languages used
Completed Tracking Forms are printed from WinW3S 
send paper copies or scans of original forms
WinW3S file not exported
Double-punching data missing
Documentation incomplete or does not match data
After data is submitted, national centers are not
prepared to answer questions from the IEA DPC
 Although data processing is exhausting, it’s not time
for vacations, yet ☺
21
Data Management Seminar, 9-12th July 2007, Hamburg
Thank you very much for your
attention.
22
Data Management Seminar, 9-12th July 2007, Hamburg