Transcript Measures of Central Tendency & Experimental Research

Experimental Research I
Day 4
For Tomorrow
• Email first draft of introduction by class
tomorrow.
– Follow framework from day 1 and in syllabus
– Use published articles as a guide
– Feel free to cite authors that support your
statements – Non-research articles are OK for
this.
– Before the purpose statement, explain the
problem or the issues involved. This material
will lead to your purpose statement.
Review
• Standard Error of the Mean
• Confidence
– Level
– Interval
– Limits
• Upper bound
• Lower bound
Experimental Research
• Only type of research with an intervention
• A direct attempt to influence a particular variable
• Only method that can truly begin to untangle
cause and effect hypotheses
• Directional Hypothesis = Theory statement
predicting the outcome [directional] (There will
be a significant difference…). Reflects
researcher’s expectations.
• Bilingual 3rd graders taught with the Kodaly
method will demonstrate significantly higher
musical achievement than bilingual 3rd graders
taught with a traditional eclectic method.
Null Hypothesis
• Null Hypothesis = Theory statement predicting
the outcome stated in the negative [nondirectional] (There will be no significant
difference…) The statistical hypothesis that
states that there are no differences between
observed and expected data. Does not reflect
researcher’s expectations (value free)
– There will be no significant difference in musical
achievement of bilingual 3rd graders taught with the
Kodaly methods and bilingual 3rd graders taught with
a traditional eclectic method.
– The goal is to REJECT the Null Hypothesis based (on
95% Confidence level or above)
– Cannot prove the null hypothesis (a negative)
Type I and Type II Error
• Type I Error is erroneously claiming statistical
significance or rejecting the null hypothesis
when in fact, it’s true (claiming success when
experiment failed to produce results)
– Possible w. incorrect statistical test
• Type II Error is when a researcher fails to reject
the null hypothesis when it is in fact false
(claiming failure when successful)
– The smaller the sample size, the more difficult it is to
detect statistical significance
– In this case, a researcher could be missing an
important finding because of study design
Group Comparisons
• Experimental Group
– Receives a particular treatment specified by the
researcher
• Control/Comparison Group
– Does not receive that particular treatment
• Sometimes difficult in educational research to
have a strict no-treatment, control
– Example: Any instruction is likely to be more effective
than no instruction
Randomization
• Random assignment to groups
– Every individual has an equal chance of being in the
experimental or control/comparison group
• Supposed to help eliminate extraneous sources of
variance
– For example… if the groups are sufficiently large, any
differences between groups on extraneous variables are likely to
be due to chance or randomly distributed among the groups
• Quasi-Experimental=non-randomized groups
– Most ed. research
– Intact classes & convenience samples
– Impacts ability to generalize to whole population
Variables
• Independent variable (IV)
– The experimental or treatment variable
– This variable is manipulated by the researcher
– Examples: instructional approach, environmental condition, the
introduction of a particular musical element
– Participant attribute
• Dependent variable (DV) – Compared b/w groups
– The criterion or outcome variable
– Examples: student attitudes, student achievement, teacher
effectiveness as measure by ?
• Experiments can be expressed as “The effect of the
‘IV’ on the ‘DV’”
• Extraneous Variables
– Those that are not specifically included in the study but never the
less may effect the outcome
– Object is to control for extraneous variables
– The researcher may not know them all
Manipulating the IV
• Presence of the variable vs. absence of the
variable
– Kodaly instruction (treatment group) vs. Kodaly
instruction (control group)
• One form of the variable vs. another
– modeling vs. verbal music instruction (vs. control
group)
• Varying degrees of the same variable
– 100% positive feedback, no negative feedback vs.
50% positive feedback, no negative feedback
Controlling for Extraneous Variables
• Best case scenario – all individuals are as
similar as possible on all variables other than
the independent variable
• Methods to control:
– Randomization & large sample
– Holding variables constant (freeze private lessons)
– Build variable into the design (compare private
lessons w/ no private lessons)
– Matching pairs – one to control, other to exper.
– Statistical control – analysis of covariance
(ANCOVA)
Design and Experiment
[Effect of Colored note heads on Music Reading]
• State Hypothesis and Null Hypothesis
• Select sample and assign to group (control
and treatment). How many in each?
• Identify independent and dependent
variables. Any possible extraneous
variables?
• Describe experiment. What will you do w/
each group and for how long? How will
you know what they already know?
Discussion of Projects
• On task
• Practice explaining your project
– Background; State the problem
– Purpose statement
– Research questions
– Methodology (research design)
Experimental Research
Designs
Nomenclature/Abbreviations
• When looking at the symbols used to
describe various experimental design
approaches:
– R = random assignment
– O = testing (pre- or post-)
– X = treatment
– C = control/comparison
– M = matched
Pre-Experimental Designs
[Pilot Studies – Generally Weak]
• One Shot Case Study (X O)
– No random assignment, No control/comparison, no pretest
• One-Group Pre-test, Post-test (O X O)
– No random assignment, No control/comparison group
• Static/Intact-group Comparison X O
– No random assignment
O
• Static/Intact-group Pre-test, Post-test
– No random assignment, possible pre-test effects O X O
O
True Experimental Designs
Stronger – not always possible in educ.
• Randomized Post-test Only, R X O
Control Group
R O
– Still not sure about pre-test levels
• Randomized Pre-Test,
Post-test, Control Group
R OXO
R O O
– Good checking whether smaller groups are
actual similar at the start of the study and
possible effects of pretest
Randomized Solomon Four-Group
& Posttest Only, Control Group
• Solomon 4–Group controls for possible
sensitization effects due to testing or maturation.
1. R O X O
2. R O O (maturation or pretesting?)
3. R
X O (effect of pretest?)
4. R
O (control group)
In a successful experiment, what would we expect for each group?
What if the Post Test scores for group 2 were as high as the Post Test for group 1?
What if the Post Test scores for group 3 were lower than group 1?
What if the Post Test scores for group 4 were the same as groups 1 & 2?
Counterbalanced Design
(Latin Square) Order effect
• Ex: Introducing note values: 1=tatiti; 2=num/no subdiv; 3=num/sub
div; 4 dotted notes
Quasi-Experimental Design
• So called b/c there is no randomization…
• Matching Only
– Participants matched in pairs to control for an
extraneous variable rather than randomly assigned
• Counterbalanced Design (next slide)
– Multiple groups receive all treatment types in different
order
– Average post-test scores across groups are
compared to determine effectiveness/effect of the
treatment order
– Vulnerable to multiple-treatment interference
• Time-series Design
– Outcome measured several times before and after
introduction of the treatment O O O O X O O O O
Quasi-Experimental Design
• Factorial Design (2 or more factors (IVs) at
different levels)
– Examining the effect of more than one
independent variable
– Allows for examination of attribute variables (i.e.
gender, age) and interaction effects b/w
combinations of IVs
– Example: Kodaly vs. trad. instruction for bilingual
and non-bilingual students
• Possible outcome showing interaction of two IV’s
• Non-bilingual students may do equally well w/ Kod. and traditional
methods, while bilingual students may do better w/ Kod. vs. traditional
methods.
• What if you had not separated these groups out?
Factorial Example
(Two Way - 2x2)
• IVs = Language classification (biling. vs.
non-biling.) & method (Kodaly vs. trad.)
• DV = musical achievement test?
• Groups (Six 3rd gr. Sections-3 Kodaly; 3
Trad; Bilingual & Non-Bilingual in all
groups.)
–
–
–
–
Bilingual & Kodaly
Non-Bilingual & Kodaly
Bilingual & Traditional
Non-Bilingual & Traditional
2 Way Factorial Designs (2 independent variables
[often one manipulated, one attribute)
2X2 (2 levels of both variables)
METHOD
Language
Classification
Kodaly
Traditional
Bilingual 1
Bilingual 2
Non-Bilingual 1
Non-Bilingual 2
Internal Validity (Usefulness/Meaningfulness) Control of Extraneous Variables: Time Bound Factors
• What happens within the experiment
– History – What happens b/w pretest and
posttest (private lessons, change in
practice routine)
– Maturation – is change result of
treatment natural result of repetition and
improvement over time?)
– Mortality – Loss of participants may
cause imbalance b/w groups
Internal Validity – Sampling &
Measurement Factors
• Testing – pretest affect posttest. Ceiling and floor
effects (eliminate outliers?)
• Instrumentation – changes in measurement or
observers (judges at contest from one site to the
next)
• Statistical regression – students who score
extremely high (ceiling) or low (floor) on pretest may
regress to the mean on posttest
• Selection – participants do not represent normal
population (also affects external validity)
• Interactions – influence of a combination of the
above factors
Internal Validity
• John Henry Effect
–Control group performs beyond
usual level because they perceive
they are in competition with the
experimental group
External Validity – Generalizability
• Population Validity
– Extent sample is representative of the population to
which the researcher wishes to generalize the results.
• Ecological
– Study conditions and setting are representative of the
setting in which the researcher would like to apply the
findings
• Replication
– Results can be reproduced (problem w/ Mozart effect)
• Detailed description of the sample needed in study
– Important regardless of sampling method
– ‘Next best thing’ if not a large, random sample – often
the case in music ed. research
– Consider demographic questions in descriptive
research
Other Threats to External Validity
• Effect or interaction of testing (testing will not occur in
natural setting)
• Sample does not reflect population
– Discuss in research report
• Reactive effects of sample
– Hawthorne Effect
• Effects due simply to subjects’ knowledge of being in a study
– Teacher or Researcher interactions different than in
population
• Subconsciously encouraging or discouraging a group
• Research setting does not reflect typical settings
(ecological validity)
– A university lab school