One-way ANOVA
Download
Report
Transcript One-way ANOVA
CUE Forum 2008
Research design:
The backbone of academic
inquiry
Peter Neff – Doshisha University
Matthew Apple – Nara National College of Technology
David Beglar – Temple University Japan
Olympic Memorial Youth Center
November 1, 2008, 1:15 - 2:50 p.m.
Overview
Introduction: The importance of good
research design
Approaching the study
Developing the study design
Break for Q&A
Designing the right instrument
Implementing your design
Conclusion
Further Q&A
Introduction: The importance
of good research design
Poor design vs. Good design
Poorly-designed study
Hit on an idea, dive right in
No background research
Throw together a survey, give to a group of
unwary participants
Collect data, then ponder how to analyze
Run to a colleague for help
Fish around for most “interesting” findings
Pray to get published
Poor design vs. Good design
Well-designed study
Hit on an idea, do background research
Formulate relevant, specific, practical RQs
Consider participants, context, data analysis in
advance
Decide/develop instrument; pilot and revise it
Decide on appropriate pre/post-test instruments
Plan stages and structure of data collection
Prepare participants adequately
…Then carry out the study
The importance of good design
A well-designed study provides many
benefits:
– Demonstrates researcher knowledge
– Ties the study to an underlying
philosophy
– Provides a clear path for the
researcher(s)
– Helps avoid mishaps of previous
studies
The importance of good design
Other benefits
– Leads to more concrete results,
more definitive conclusions
– Improves chances of publication
– Raises the status of SLA as a field of
inquiry
A word about mixed methods
designs
The great quan-qual debate
Mixed methods – the “best” of both
worlds
Add a qualitative component to a
quantitatively-oriented study:
–
–
–
Participant interviews
Observational, audiovisual data
Open-ended survey questions
Plan for qualitative analyses (text
analysis, response coding)
Part 1
Approaching the Study
Approaching the study
Hitting upon research ideas
Review of the literature
Formulating research questions
Hitting upon research ideas
Identify the topic in a few words
Reflect on “doability” of research
–
–
–
Can I research this?
Should I research this?
Am I interested in researching this?
Review of the literature can help
redefine and revise ideas
Identifying the topic:
Hints for starting to narrow
Pose a short question using “what” or
“how”
Write a short title that consists of one
sentence under 12 words
Ask a friend or colleague to read your
topic and gauge their reactions
Draft research questions to see if the
topic can be adequately explored
A “researchable” topic
“Can I do this in my current situation?”
“Does this concern people at other
institutions?”
“Does this add to the current body of
research related to this topic?”
“Does this study contribute something
from a unique perspective?”
Filtering “probably not so good”
ideas:
To boldly go where no research has gone
before… (The “Star Trek” idea)
My theory is clearly better than X
(The “Steven Krashen is so wrong” idea)
My classroom is totally unique
(The “I don’t need theory” idea)
This is a really cool technology /
methodology / text book
(The “I am primarily a teacher” idea)
Filtering ideas: A few hints
Review research designs and
statistical techniques
Review teaching methods and overall
SLA research results
Evaluate access to potential study
participants
Plan time for material creation, study
design, and implementation
Review of the literature
Relate the study to continuing
“dialogue” in current research
Finding a “gap” in the literature
Provide a framework for the
importance of the study
Review of the Literature:
Finding a “gap” in knowledge
“We do not enough about X…”
“This way of looking at X has never
been done…”
“This way of learning about X has not
been duplicated in my context”
“Previous research has inadequately
explored X…”
Finding literature: Some hints
Google Scholar using key words or
researcher names
Scour recent literature review articles
Check for “cited” numbers online
Get access to university databases
Refer to recently published articles
–
–
After the year 2000
During the previous 2 to 3 years
Examine “outside the field” articles
Finding literature:
Separating the wheat…
“Top tier” journal articles
Most-often-cited articles
Recent articles
Research articles (not reviews)
Books / Edited book-articles
Major international conference papers
Dissertations / dissertation abstracts
…from the chaff
“In-house” journal articles
Articles from “proceedings” books
Online journal articles with only .html
versions
Unedited books from small publishers
Newspaper and magazine articles
Web pages
Anecdotal evidence
Your own previous papers for an MA
course
Research questions:
A few useful guidelines
Naturally flow from the literature review
Strongly connected to the topic
At least two or three (not one)…
…but not five or six or more
As specific as possible
Directly concern variables in the study
Do not contain yes/no question words
RQs: What not to ask
“Is X true/false?”
“Will X happen if…?”
“Does X cause Y?”
“What do participants think of X?”
“Why does X happen?”
RQs: What to ask
“What differences exist between…”
“Compared to X, how does Y…?”
“To what degree do X and Y differ…?”
“When X is controlled for Y…, how
does Z…?”
“What are underlying patterns
among…?”
“To what degree does X predict Y?”
Part 2
Developing the Study Design
Research design
Cross-sectional design: A design in
which data are collected from a sample
at only one point in time.
Longitudinal design: A design in which
data are collected at more than one
point in time.
Randomized Control-Group
Pretest-Posttest Design
Experimental Group 1
T1
Xa (Method a)
T2
Experimental Group 2
Control Group
T1
T1
Xb (Method b)
T2
T2
Randomized Control-Group
Pretest-Posttest Design
Reasonably strong conclusions can be reached
about the effects of the treatments.
Problem 1: Within session variation (e.g.,
different teachers or room conditions) may
intervene.
The solution? Randomly assigning participants,
times, and places to the experimental and
control conditions.
Problem 2: The pretest may interact with the
treatment. This potential problem is dealt with in
the next design.
Randomized Solomon Six-Group
Design
Pretested (Random assignment)
Pretested (Random assignment)
Pretested (Random assignment)
Unpretested (Random assignment)
Unpretested (Random assignment)
Unpretested (Random assignment)
T1
T1
T1
Xa (Method a)
Xb (Method b)
Xa (Method a)
Xb (Method b)
T2
T2
T2
T2
T2
T2
Randomized Solomon SixGroup Design
This design amounts to doing the
experiment twice –once with and once
without pretesting.
It is possible to know what effects, if any,
are associated with pretesting.
If the results of the “two experiments” are
consistent, greater confidence can be
placed in the findings.
Counterbalanced Design
This design is useful when randomization is not
possible and intact groups must be used.
Replication
Xa
Xb
Xc
Xd
1
A
B
C
D
2
B
D
A
A
3
C
A
D
B
4
D
C
B
C
Counterbalanced design
The counterbalanced design rotates out
the participants’ differences (e.g., one
group has more aptitude or motivation than
the other groups) by exposing each group
to all variations of the treatment.
Order-of-presentation effects are controlled.
Primary weakness: The possibility of
carryover effects from one treatment to the
next exists. Allowing time between
treatments can alleviate this problem.
Control-Group Time-Series Design
Experimental Group 1 T1 T2 T3 T4 Xa (Method a)
Experimental Group 2 T1 T2 T3 T4 Xb (Method b)
Control Group
T1 T2 T3 T4
T5 T6 T7 T8
T5 T6 T7 T8
T5 T6 T7 T8
Control-Group Time-Series
Design
This design allows the researcher to
determine growth over time, and the effect
of an intervention.
The presence of a control group increases
the trustworthiness of the results because
the possibility of a contemporary event
causing any gains can be determined.
Control-Group Time-Series
Design
This design can be extended by exposing the
participants to the intervention on multiple occasions.
This approach is more sensitive to partial gains in
knowledge and tests the strength of the intervention
more than once, thus giving the researcher a more
accurate understanding of the effectiveness of the
intervention.
Experimental Group 1
T1 T2
Xa
T3 T4
Xa
T5 T6
Experimental Group 2
T1 T2
Xb
T3 T4
Xb
T5 T6
Control Group
T1 T2
T3 T4
T5 T6
Q&A Break
Part 3
Designing the Right Instrument
Instrument Design
Commonly used instruments in SLA
research
–
Scored tests
– Rater scores
– Surveys
– Interviews
Consider your eventual data analysis
when developing instruments
Instruments - Scored tests
Pluses
Quantitative items
(M/C, Cloze/C-tests)
–
–
–
–
Simple to score large
# of participants
Easier to analyze
Qualitative items
(short answer, timed
essays)
–
Minuses
Quantitative items
Good complement to
quantitative scores
Can provide more indepth assessment of
participants’ abilities
Limited to one type of
data
Qualitative items
–
–
Take more time/effort
to score
Rater bias
Instruments – Performance
ratings
An assessment of participants’
performance in an assigned task
Tasks may include presentations,
interviews, written essays
Performances can be scored using a
Likert-scale, a rubric, or holistically
Usually scored by at least two “expert”
raters; sometimes also by peers
Performance ratings
Rating criteria should be concretely
established with little ambiguity
Avoid including too many (or too few)
criteria for one performance task
All raters should undergo a “normative”
training session prior to assessment
Use models to train raters
Avoid single-score holistic ratings
Instruments - Surveys
Often used for:
–
Collecting learner history data (L2 study
experience, other background info)
– Assessing participants’ attitudes towards
a predetermined construct (language
learning motivation, anxiety using the L2)
– Determining reactions to an experimental
treatment (teaching methods, innovative
learning tasks)
Survey making
For non-advanced learners – surveys
should be in their L1
Build in redundancy - Include multiple
questions for each concept area
Questions should be simply worded –
avoid negative or confusing wording
Depending on the purpose of the
survey, 20-40 items/session is a good
range to shoot for
Survey making
Any survey used in a serious study
should be piloted in advance
It is acceptable to make adjustments to
an existing instrument
Likert-scale items should usually have
between 4 and 6 choices
A few qualitative questions can provide
a nice complement to quantitative
instruments
Instruments - Interviews
Interviews can provide an excellent
qualitative component to a larger study
It is not necessary to interview all
participants
–
a subsample as small as 10-20% can be
acceptable
Use your best judgment on
participants’ language ability
–
For intermediate-and-above learners, L2
interviews are often fine
Conducting interviews
Inform students they are being
interviewed, obtain consent
Record unobtrusively
“Warm up” the participants before
getting into the heart of the interview
Collect more data than you need
Validating Instruments
Instrument Validity
The construct = The heart of the matter
What construct do you wish to
measure?
How do you define the construct?
What are its component parts? Do they
form a unified whole?
Operationalizing the
construct: The items
Conceptualize the construct as a continuum:
easy—difficult items and less able—more
able persons.
How have other researchers measured the
construct?
Write original or adapted items.
Cover the estimated range of your
participants.
Write 50% more items than you intend to
use. This will allow you to “cherry pick” the
best items as well as items at various levels
of difficulty.
Operationalizing the
Construct: The Items
More able | More difficult
persons
| items
|
x | item 1
xx | item 2 item 3
xxx | item 4 item 5
xxxx | item 6 item 7 item 8
xxxx | item 9 item 10 item 11
xxx | item 12 item 13
xx | item 14
x | item 15
|
Less able | Less difficult
Persons
| items
Operationalizing the
Construct: The Items
After piloting the items, statistically
analyze the results.
Examine dimensionality, item difficulty,
and item content.
Select the best items to make an
efficient, highly reliable instrument.
Part 4
Implementing Your Design
Implementing the design
Including other researchers
Practical issues
Handling ethical concerns
Including other researchers in
the study
The nature of the researchers involved
–
–
Main researcher plus “helpers”
One researcher plus “other participants”
The nature of the instructors involved
–
–
–
–
Teaching methods
Students taught
Course goals
University program goals
Working with other researchers
Work with people you know and trust
Establish a schedule early
Define clear roles for each researcher
Decide definite research goals prior to
data collection
Keep in regular contact
The “band practice time” principle
Example research roles
Head researcher / contact person
Data entry specialist
Statistician
Interviewer
Literature analyst
Editor / proofreader
Heading off potential problems
Explain study commitments prior to starting
the study
Agree on “ownership” prior to data collection
and data entry
–
Who will keep the data?
–
Whose name comes first, second, etc.?
Keep everyone aware of deadlines
Include everyone in decision-making
processed and data analysis
Things to avoid in group
research
People you don’t know
Research groups larger than four or
five members
Using someone for language skills
Involving others just to get more
participants
Forgetting to thank others for their
assistance
Improving relations with
research helpers
Write clear instructions
Thank profusely for their time and effort
Offer to send copies of final research
papers and/or results
Offer to assist in future research
Keep in touch after research ends
Practical Issues
Timing of implementation
Learning and research context
Participant consent
Financial considerations
Timing of the implementation
Beginning, middle, or end of semester
Day of the week
Time of day
Exams and exam preparation periods
“Culture Festivals” or other club-related
events
“Open classes” or “parents’ day”
Learning and research context
Differing course goals (I.e., listening
class vs. reading class)
Different major field of study
Gender, age, year in school
Number of class meetings
Perception of the value of research by
institution heads
Participant consent
Always allow for “non-participation”
choice from potential participants
Write clear instructions for participants
asking for their cooperation
Ask co-researchers or helpers to briefly
inform participants about their choice
Financial considerations
Copies for questionnaires, exams, etc.
Computer analysis software
Mailing costs
Interview travel costs
Recording equipment
Reference books
Journal article costs
Heading off lack of
cooperation problems
Review requirements of the study
–
How many items in the questionnaire?
– How many treatments?
Recognize that students are busy, tired,
etc.
Plan to get between 30-50% more data
than you need
Try to get more data at a later date
Ethical Considerations
Ethical considerations
Students should not be exploited just
because they are there
In theory, they should derive benefit
(directly or indirectly) from the research
Explain the basic purpose of your
research before collecting data
–
1-2 sentences should be fine
It is also good to briefly explain this at
the beginning of any survey instrument
Ethics - Consent
Provide students with the chance to opt
out of participation
Verbal consent is usually acceptable
for surveys, ratings, and test scores
A written consent form may be
necessary for more involved forms of
participation (interviews, essay
passages)
When in doubt, check recent articles in
well-known journals for guidance
Ethics – Other considerations
The role of this study within your
institution
Potential gender, proficiency or other
issues that may affect your data or
conclusions
Have a plan for anonymizing the data
(and consider making this conspicuous
on your instruments)
Conclusion
In conclusion…
There are many factors to consider when
embarking on a serious study
Some points to take away…
Most studies should fill a place (a “gap”)
within the current academic dialogue
Research questions should reflect
continuity with the literature and should be
specific
Carefully consider the design setup which
will work best with the participant groups
and the research and analytical goals
In conclusion
Further points…
Different instruments work better in
different circumstances. Choose those
which best reflect your aims.
Plan your analyses at the same time you
are developing your instruments
Develop and pilot instruments which can
cover responses from a range of
participants
In conclusion
Even more points…
Work with others who will be serious and
committed, and then be an organized
and conscientious leader
Consider how practical, pedagogical, and
institutional concerns may affect your
study
Do not forget current ethical guidelines
for carrying out participant research
Good luck with your
research!
Thank you for listening
Q&A
For a copy of this
presentation:
http://jaltcue-sig.org