Sessions 5, 6, 7, 8 Quantitatve research

Just checking

• Write down 3 things you learned yesterday • Share with a colleague how these might (or might not) be of use for your research proposal

Session outline

Quantitative research • 9 Designs • Data collection • Data analysis • Literature review

Session 5 Quantitative Research Design

Aim of quantitative research

• Quantitative research seeks to explain aspects of the world • It aims to generate causal statements • It seeks to show that A causes B, so we can say that if A happens then B is likely to follow • Hence researchers advise not only how to explain but also how to order and control aspects of the world • Quantitative research is favoured by policy makers

Designs

1. Pre-experimental 2. Quasi-experimental 3. True experimental 4. 3 group experimental 5. Single case ABAB 6. Meta analysis 7. Causal Comparative 8. Correlational 9. Survey

Terminology

• O - observation (data collection) • X - treatment (intervention) • R - randomly assigned to groups • E - experimental group (gets treatment) • C - control group (does not get treatment)

1. Pre-experimental design

• One group pretest – posttest design • Example: numeracy test on 5 years at start of year then try out a new way of teaching number concepts then retest at the end of the year. Question: did the new method of teaching work? • If posttest scores are higher than pretest scores, what can you claim? Why?

O 1 X O 2

2. Quasi-experimental design

• Give numeracy test at start of year to two classes of 5 year olds. Randomly assign treatment to one (experimental group). Then give posttest to both classes at end of year.

• What can you claim if the experimental group has higher posttest scores than the control group? What if they have the same?

Experimental

Control

O 1

O 3

X O 2

O 4

3. True experimental design

• Randomly assign students to one of two classes. Give pretest at start of year. Give intervention to one class (experimental class) but NOT the other (control group). Give posttest at end of year.

• What claims can you make about the relationship between numeracy performance and the intervention?

Experimental

Control

RO 1

RO 3

X O 2

O 4

A caution about randomisation

• In pairs: one person tosses a coin 10 times and writes down results (repeat 10 times). The other person imagines tossing the coin 10 times and writes down the imagined result (repeat 10 times). Then compare • Discuss: Does random mean equal numbers??

4. 3-group experimental design

• To find out if the pretest itself has an influence add a third class which does not get the pretest but does get the treatment.

• If second control does better than the first control group on post test, then we can rule out the pretest as influencing the students.

•

Experimental RO 1 X O 2

First control Second control RO 3 X O 4 O 5

Session 6 Quality of research

Threats to internal validity

Is the treatment causing the effect? Or something else? For example: 1. History 2. Maturation 3. Statistical regression 4. Testing 5. Instrumentation 6. Selection 7. Experimental mortality

Quality of research

• Research must have internal validity to be externally valid (generalisable) • However, if it has internal validity, it may not necessarily be externally valid

Threats to external validity

Factors that may limit generalisability: 1. Poor description of independent variable 2. Sample may not represent the population 3. Hawthorne effect 4. Weak operationalising of dependent variable (eg pen and paper test vs career decisions)

In pairs • Share your thoughts about the possibility of conducting a true experiment, for your research proposal • What are the possible barriers? What about ethical issues?

5. Single case experimental design

• ABAB design: Alternating no treatment and treatment for single case (constant throughout) • Example: student calls out incessantly to interrupt teacher. Teacher wants to reduce this. Treatment is if student doesn’t call out, the student is allowed to spend time working one-to-one with the teacher (which is valued by the student). Monitor student calling out behaviour for 1 day, first without the treatment, then with, then without then with.

• Plot the frequency of calling out by student. If systematically lower with treatment, then consider success.

• But what about generalisability?

6. Meta analysis

• Statisticians’ heaven • Aim is to find how much difference a variable makes, rather than whether or not it is statistically significant at a particular level of confidence.

• How much difference is given as effect size • Benefit is that meta anlaysis combines all known research • Example: class size and student performance research: 77 studies involving 725 comparisons of smaller and larger classes and 900,000 students; found an inverse relationship!!

Two types of design

Experimental

• Deals with difference between groups • Looks from causes forward to effects

Correlational

• Deals with differences between variables • Looks from effects back to causes • Called ex post facto (after the fact) designs

7. Causal Comparative

• Studies of characteristics of individuals (eg high/low performing students; ethnicity; age) • Deals with causal relationships but not through experimental manipulation of variables and so cannot yield certainty • Example: What are the sex differences in likelihood of careers in engineering, among people with strong positive feelings about Mathematics? Data on attitude to Mathematics and on career. Found males likely to choose engineering; females no relationship between strength of attitude to Mathematics and likelihood of selecting a career in engineering

8. Correlational

• Aims to find size of relationship between variables (eg age and reading ability) • Often called prediction studies • Caution: correlation does not mean causation (number of churches and number of bars!) • Literacy and Numeracy at age 5 and then at at age 8: serves to confirm predictive validity of PIPS instrument • Analyses using multiple regression, path analysis, factor analysis etc

9. Survey

• Very widely used in educational research • Hard to get high response rate (incentives) • Often simply descriptive: to find out characteristics of a sample and to generalise to a population (eg market research or political research) • But can be used for correlational studies • Eg Coleman 1966 study of relationship between school characteristics and student achievement. Found family background was more important than school characteristics in explaining student achievement. Later the School Effectiveness research overturned this, showing that schools matter. More recently, studies show that the teacher matters most of all

In threes • Discuss possibility of using a correlational design for your research proposal • Any barriers? Ethical issues?

Debate preparation

Prepare your arguments for the debate for the topic: ‘That qualitative research is better than quantitative research’

Data collection

Sources of data – direct observation – student assessments – surveys – scales – performances Types of data – Frequency tallies – Test results – Likert scale responses

Data analysis

Descriptive statistics – Mean, mode, median, standard deviation, variance, range Measures of relationship – Correlation Indices of relative standing – Percentiles, standard scores Inferential statistics – Probability or likelihood of occurrence – Allows you to generalise from your sample to the population from which it is drawn

Summary of quantitative research designs

• Most rigorous are the true randomised control experiments. Without these we can not have a great deal of confidence in our findings (and even then there are doubts!) • Educational research does not have the luxury of random assignment of students so we adopt the less rigorous correlational designs. • We make these as strong as we can and interpret the findings cautiously

Session 7 Literature Review

Definitions

Two type of definitions • Conceptual: meaning stated in terms of theories of the discipline • Operational: process for determining its presence or absence is carefully described

Literature review

• Provide a theoretical/empirical background to the research and place in a meaningful context.

• Convince audience your proposed research will make an original and significant contribution to its field (resolving an important theoretical issue; or filling a major gap in the literature; resolving a ‘real world’ problem through effective application of existing knowledge)

Sources

• Journal articles – peer reviewed • Technical and other reports • Books and book chapters

Acknowledgement of sources

• Acknowledge those who have laid the groundwork for your research • Plagiarism – ideas not just words • Referencing considerations • Using quotes

Demonstrate

• Broad in-depth mastery of the theoretical and research issues related to your research question • Ability to evaluate critically the literature relevant to the problem • Ability to synthesize a large body of literature

Shaping your lit review

• Begin broadly – define the overarching aim or problem to be addressed • Establish the context for the problem/aim • Indicate what has been done previously, if anything, to address the problem/aim • Build a thesis/argument which leads logically to your specific research questions

Quality

• Organization, structure, focus, coherence • Comprehensiveness – considering all influential papers in the field and keeping up with recent developments • Summarizing previous work effectively • Critically evaluating cited papers • Relying on primary sources

Activity

• Critique two examples of literature reviews on the basis of any of the quality characteristics discussed in this session • What have you learned about writing a Literature Review?

• Would you now find other people who have been reading the Literature Review with the same letter as you (eg, A, B, C, D, E) • Please sit in a group • Share your reactions to the Literature Review you have been reading • Please critique the review

Session 8 Conceptualising your research

Major paper 12 point unit

You can choose • to undertake a small empirical study (and your research proposal might be the plan for this study) OR • To undertake a full Literature Review (of the topic you have already written about in your proposal)

• Further drafting of your research area and question • Discuss in pairs

For tomorrow

• Find out what is meant by ‘thick description’ • Write notes and bring to share