Transcript Document

Educational Action
Research
Todd Twyman
Summer 2011
Week 2
Gathering Quantitative Data
Numbers!
 Attendance records, test scores, grades,
specific counts of behavior, responses to
selected response survey questions, etc.
 Consider the validity of the source of your
numbers for the specific use you intend.
 Has this use been empirically validated?
 How do you know the numbers are
‘good’?
Designing Your Dependent
Measure
1.
Identify the construct.
2.
Identify the purpose of the measure.
3.
Define the content to address the
purpose and construct.
4.
Decide on the structure of the test.
5.
Formalize administration protocols.
6.
Write down the scoring methods.
Structure of the Measure
Consider the construct under investigation
and the intended use of the observed
scores. What level of knowledge are you
hoping to measure?
 Format
of the Measure
 Selection/Production
Response
 Written/Oral/Demonstration/etc.
Structure of the Measure

Context of the Problem
 Situated/Not

Situated
Context of Administration
 Timed/Not
Timed
 Group/Individual
 Standardized/Not
Standardized
Scoring the Behavior
Consider the construct under investigation
and the intended use of the observed
scores
 Type
of Scoring System
 Objective/Subjective
(analytic/holistic
rubrics)
 Dichotomous/Partial-Credit
Scoring the Behavior

Type of Reporting
 Letter
grade/number or percentage
correct/2 category/ checklist/narrative
report

Basis for Interpreting Scores
 Norm-/Criterion-/Individual-reference
Selected Response Items




Make sure each answer choice is a viable
option (syntactically possible).
Make sure each answer choice is
approximately the same length.
Randomly order the correct answer choice
location.
Avoid confusing question stems that might
detract from people’s understanding.
Example
Poor Example
During the past month, have you felt depressed?
0 = No
1 = Yes, once in a while
2 = Yes, some of the time
3 = Yes, most of the time
4 = Yes, all of the time
Harder to develop questions and response categories
May force invalid responses􀂾
Less depth and substance
Poor Example, cont.
􀂾Do you think it should be possible for a
pregnant woman to obtain a legal
abortion? Circle your response.
1 = Yes
2 = No
Respondents unable to explain, qualify,
or clarify answer
Dos and Don’ts
Be Concise
Poor:How do you feel about building an ice arena
in downtown Coquitlam where the railroad
property has been sitting unused for a number of
years?
Better:An ice arena should be built on the railroad
property in downtown Coquitlam.
1 = Strongly agree
2 = Agree
3 = Disagree
4 = Strongly disagree
Dos and Don’ts, cont.
Use simple, clear language
Poor:How often do you punish your toddler?
Better:How often do you put your toddler
into timeout? Check only one.
___ Once a day
___ Several times a day
___ Once a week
___ Several times a week
Dos and Don’ts, cont.
Make questions concrete
Poor:Did you enjoy the book?
Better:Have you recommended the book to anyone else?
Use mutually exclusive and exhaustive categories
Poor:What is your marital status?
___ Married ___ Single
Better:What is your marital status?
___ Married
___ Divorced
___ Separated
___ Widowed
___ Never Married
Dos and Don’ts, cont.
Limit “skip”patterns
Do you participate in sports?
1 = No (GO TO QUESTION 3)
2 = Yes (Check all sports that apply)
___ Football
___ Volleyball
___ Basketball
___ Soccer
___ Swimming
___ Other
(Specify_________)
Dos and Don’ts, cont.
Use caution when asking personal questions
Poor:How much do you earn each year?
$______________
Better:In which category does your annual income
last year best fit?
___ Below $10,000 ___ $10,001-$20,000
___ $20,001-$30,000 ___ $30,001-$40,000
___ $40,001-$50,000 ___ $50,001-$60,000
___ $60,001-$70,000 ___ over $70,001
Formatting a Survey




Focus on clarity of questions.
Consider carefully the # of answer
choices.
Use appropriate language for your target
audience.
Have your survey reviewed for
appropriateness prior to use.
Formatting a Survey


Include a short explanation of what you are
planning to do with the results/ why you are
seeking their input. Make a plug for why their
input is really important to you (how it will
help improve something if possible).
Try to fit all the survey items on 1 piece of
paper – double sided is OK.
Formatting a Survey

If possible, phrase the question stems as “I
statements” rather than as questions.

Include demographic information that will
allow you to group respondents for later
analysis – be sure to consider how you will
analyze the data prior to administering the
survey so you have this covered!
Example
Your Turn!

Write questions for a survey or a selected
response test appropriate for your Action
Research study. (10 minutes)

We’ll hold ‘content review’ focus groups as
a means of providing you with feedback on
your instrument.
Content Review Focus
Groups




Groups of 5 people.
Decide on appropriate questions for the
group to consider. (2 minutes)
Take turns facilitating [for discussion of
your own measurement instrument] and
acting as note-taker. (5 minutes)
Review each person’s survey or test and
provide feedback on ways to improve.