Transcript Document

The Effect of Computers
on Student Writing:
A Meta-Analysis of
Studies from 1992 to 2002
Amie Goldberg, Michael Russell, & Abigail Cook
Technology and Assessment Study
Collaborative, Boston College
Computers in the schools
• Increase in presence:
– 1983: one computer for every 125 students
– 1995: one for every nine students in 1995
– 2001: one for every 4.2 students
– (Glennan & Melmed, 1996; Market Data Retrieval, 2001)
– Most common educational use of computers by students
is for word processing (Becker, 1999; inTASC, 2003)
Our study:
• Builds on earlier research (Cochran-Smith, 1991;
Bangert-Drowns, 1993)
• Covers the new generation of research spanning from
1992 through 2002
• Combines quantitative and qualitative measures
• Focuses on a generation of research that differs in two
ways from earlier research:
– vast improvements in technology
– increased student usage and comfort levels with the
technology
Research Questions:
• Does word processing impact K-12 writing? If
so, in what ways? (quantity and/or quality)
• Does the impact of word processing on
student writing vary according to other
factors, such as student-level characteristics
(grade level, keyboarding skills,
urban/suburban/rural school setting, etc.)
Meta-Analysis
• Gene Glass - the first to propose such
methods and coined the term in 1976.
• “Meta analysis refers to the analysis of analyses …
it … refer[s] to the statistical analysis of a large
collection of results from individual studies for the
purpose of integrating the findings. It connotes a
rigorous alternative to the casual, narrative
discussions of research studies which typify our
attempts to make sense of the rapidly expanding
research literature”.
Methodology
• Employed procedures detailed by Lipsey and
Wilson (2001) as well as those set forth by
Hedges and Olkin (1985).
• Five main phases:
– identification of relevant studies
– determination for inclusion
– coding
– effect size extraction and calculation, and
– data analyses.
Determination for inclusion
• Criteria:
• Quantitative study conducted between 1992-2002
• Results reported in a way that would allow effect size calculation
• Research design that employed a measure of word-processing’s
impact on writing over time OR a direct comparison between
paper-and-pencil and computerized writing
• Quality and/or quantity of student writing and/or revision of
student writing as its outcome measure(s).
• Not specifically focus on the effects of grammar- and spellcheckers or heavily multimedia enhanced software
• Not examine the differences in writing within the context of a test
administration (i.e., mode of administration rather than mode of
learning)
• focus on students in Grades K-12.
Articles collected (N=99)
Coding of outcome measures
•Quality
•holistic (n=10) vs. individual dimension scores (n=5)
•Quantity
•number of words (n=14)
•Revision
•diverse operational definitions- insertions, deletions,
corrections, surface/format changes, content/meaning
changes (n=6)
Extracting and calculating
effect sizes
•Calculation: mean performance difference between
computerized and paper-and pencil groups divided by
the pooled standard deviation
.2 thru .5 - small
.5 thru .8 - medium
.8 or higher - large
•Unit of analysis: “independent study finding” -controls for Type I errors
Adjusting for bias and applying
inverse variance weights
•Each effect size multiplied by the inverse of
its sampling variance in order to give more
weight to findings based on larger sample
sizes
•Outlier analysis (+ or - 2 SD), publication
bias (Forest plots, funnel plots, and fail-safe N
analysis)
Significance and homogeneity
analysis
•Set of independent effect sizes were aggregated and tested
for homogeneity: is the group of effect sizes part of the same
population, and thus, are not influenced by any other
variable?
Summary of findings- Quantity
Summary of findings- Quantity(2)
•Regression analyses: following groups of variables were
not significant factors affecting the quantity:
•student supports (keyboard training in study, technical assistance,
teacher feedback, and peer editing)
•student characteristics (keyboard training prior to study, student
achievement level, school setting and grade level)
•study characteristics (i.e., publication type, presence of control group,
pre-post design, length of study)
•Regression analyses on subset of studies with
interventions longer than six weeks revealed:
•effect sizes were larger for studies situated in middle and high school
students as compared with elementary school students.
Summary of findings- Quality
Summary of findings- Quality(2)
•Regression analyses: following groups of variables were
not significant factors affecting the quantity:
•student supports (keyboard training in study, technical assistance, teacher
feedback, and peer editing, etc.)
•study characteristics (i.e., publication type, presence of control group, prepost design, length of study, etc.)
•However in analyzing student characteristics, a significant, positive
relationship was found between grade level and effect size.
•Regression analyses on subset of studies with
interventions longer than six weeks revealed no
significant relationships, indicating relationship between
school level and quality occurred regardless of length of
study.
Summary of findings- Revisions
•Meta-analytic techniques could not be applied
•Nonetheless, all six studies reported more changes to their
writing between drafts with word processors as compared
with paper-and-pencil.
Summary of findings
Excluded studies
•Sixty-five articles were determined to be focused on the effects
of computers on student writing
•Some not on variables of our interest
•Some did not report statistics necessary to calculate effect sizes
•Others employed qualitative methodologies and covered such
topics as:
•writing as a social process
•writing as an iterative process
•computers and motivation
•keyboarding and computers
•generally positive effects on student writing
Conclusions
•Computers have a positive impact on the quantity and quality of
student writing
•Similar to previous era of research: fairly large positive effect
on quantity of writing
•Relationship between computers and quality of writing
appears to have strengthened over time:
On average students who develop their writing skills while
using a computer produce written work that is .4 standard
deviations higher in quality than those students who learn to
write on paper.
•Qualitative study analysis is consistent with earlier research
syntheses