Transcript Slide 1

Can Third Graders Type?
An examination of the
fairness by grade of
requiring constructedresponse items on largescale assessments
administered via the
computer
By Steve Martin
Connecticut State Department of Education
Connecticut Assessment Forum
8/13/12
Good Afternoon
Today we will
 Practice using responders.
 Use responders to get a sense of where the group is.
 Review where SBAC is on this question and cover standards







and other background on this issue.
Discuss what other states are doing.
Examine typed-response data from a recent CMT pilot test.
Discuss possible contributing factors to trend data.
Review where Connecticut is going in the short-term.
Discuss reporting and further research possibilities.
Q&A
We will finish by exploring some free sample online resources
designed to help students learn to type.
Practice Using Responders

Start->programs->2know toolbar
① To start your responder, first hit the on/off
button on the bottom.
Practice Using Responders

Start->programs->2know toolbar
① To start your responder, first hit the on/off
button on the bottom.
② You will know it is on when words appear in
this screen.
Practice Using Responders

Start->programs->2know toolbar
① To start your responder, first hit the on/off
button on the bottom.
② You will know it is on when words appear in
this screen.
③ Wait for the 2Know Toolbar to appear on this
screen before you enter the session.
Practice Using Responders

Start->programs->2know toolbar
① To start your responder, first hit the on/off
button on the bottom.
② You will know it is on when words appear in
this screen.
③ Wait for the 2Know Toolbar to appear on this
screen before you enter the session.
④ Once you see the toolbar press ENTER.
⑤ When you see the network “Steve” on your
responder, press ENTER again.
⑥ Once everyone is on we will practice this
ABC question. When instructed, press A if
you are MALE, and press B if you are
FEMALE; and then press ENTER.
Question #1
Q:
What age were you when you first learned to
type?
A. Age nine or younger
B. Age 10 -12
C. Age 13 -15
D. Age 16 -17
E. Age 18 or older
Question #2
Please answer this question only if you have children of your
own.
Q:
At what age was your child(ren) when he/she/they
first learned to type?
A. Age nine or younger
B. Age 10 -12
C. Age 13 -15
D. Age 16 -17
E. Age 18 or older
Question #3
If you have a child or children who are at least 13 years
of age, (or at least grade 8) please answer this
question about typing speed.
Q: I think that my child (or my children) can type…
A. Faster than I can.
B. About the same speed as I can.
C. Not as fast as I can.
D. I don’t know how fast my child(ren) can type.
Question #4
Please answer this question if you have recent experience
with third graders using the computer.
Q:
Which of the following is the fairest assessment of
typical third-grade computer skill?
A. They cannot use the keyboard or mouse with any reliability.
B. They can use the mouse to point and click reliably
C. They can use the mouse and keyboard reliably, but their
typing is very slow.
D. They can use the mouse and keyboard reliably and they
can type words using the computer keyboard at least as
fast as they can write with a pencil.
Question #5
Q:
Given that the statewide test will be administered
to third graders via computer beginning in the
spring of 2015, which of the following test
environments for them would you currently
support?
A. Third graders should only need to point and click.
B. Third graders should point and click AND type responses to
short-answer items.
C. Third graders should point and click, type responses to
short answer items, AND type responses to extended
response items (like the CMT writing sample).
Where SBAC Stands
Steve: What research is planned on this issue?
SBAC
 Cognitive lab research is planned to learn how students
across a range of grades approach and interact with
different types of computer-administered items.
 This is part of a larger effort to eliminate or minimize
factors that could distract students from the ability to
demonstrate their knowledge and skills.
 For more information, please see the SBAC accessibility
guidelines
Where SBAC Stands
Steve: What about keyboarding skills specifically?
SBAC: The Common Core State Standards emphasize
proficiency with technology. For example, the 4th grade
writing standard calls for students to “demonstrate
sufficient command of keyboarding skills to type a
minimum of one page in a single sitting.” Administering
assessments by computer is a part of ensuring that
schools and teachers can measure mastery of the full
range of the standards.
Technology Standards for
Grade 3 Students
What exactly are the CCSS Standards on the keyboarding
issue for grades 3 and 4?
CCSS:
 Grade 3 Standard—W.3.6—With guidance and support from
adults, use technology to produce and publish writing (using
keyboarding skills) as well as to interact and collaborate with
others.
 Grade 4 Standard—W.4.6.—With some guidance and support
from adults, use technology, including the Internet, to produce and
publish writing as well as to interact and collaborate with others;
demonstrate sufficient command of keyboarding skills to type a
minimum of one page in a single sitting.
What is Age Appropriate?
Differing perspectives
“There is no right or wrong time to
introduce your child to the family
computer. Much like deciding
when to buy the first bike, or
when your child can go out alone
with friends, the choice will be
based on your parenting style,
your values and, most
importantly, your child’s
readiness.”
Source:
http://familyinternet.about.com/od/introtofamilycomputing/a/growthchart.htm
What is Age Appropriate?
Differing perspectives
What about computers and Waldorf education?
“Waldorf teachers feel the appropriate age for computer use in the
classroom and by students is in high school. We feel it is more
important for students to have the opportunity to interact with one
another and with teachers in exploring the world of ideas,
participating in the creative process, and developing their
knowledge, skills, abilities, and inner qualities. Waldorf students
have a love of learning, an ongoing curiosity, and interest in life. As
older students, they quickly master computer technology, and
graduates have successful careers in the computer industry.”
Source:
http://www.whywaldorfworks.org/02_W_Education/faq_about.asp
General Typing Speed Data
Source: Lynda S. Hartman rev 1/20/05 Permission to use if credit is retained. [email protected]
General Typing Speed Research
 Researchers suggest that 10-20
wpm is sufficient in later elementary
school (grades 3 – 6).
 Some teachers, though, have
made lower targets:
2002-2003 Study
In the Illinois
School District
U - 146
Grade
Target in
WPM
3
3
4
7
5
7
 Others made their goal to be 2-3
times the students handwriting
speed.
Source: Lynda S. Hartman rev 1/20/05 Permission to use if credit is retained. [email protected]
General Typing Speed Research
Task Force Report:
 No recommendations for
keyboarding speed until grade
8.
Buhler, Kansas
USD – 313
Conducted Research of
student typing speed.
 At this point, though,
students are expected to type
at a rate of 30 WPM with high
accuracy.
Source: Lynda S. Hartman rev 1/20/05 Permission to use if credit is retained. [email protected]
General Typing Speed Research
Utah State
Office of
Education
Utah provides a detailed Web site in
which keyboarding speed requirements
are laid out very explicitly:
Grade
WPM
3
15
4
20
5
25
6
27
7 - 12
45 by end of
keyboarding
course
http://www.schools.utah.gov/cte/keyboarding_stand
ards.html
Source: Lynda S. Hartman rev 1/20/05 Permission to use if credit is retained. [email protected]
What Are Other States
Doing?
I asked states that were listed in a State Educational
Technology Directors Association 2011 report as having
computer delivered assessmentsalready the following six
survey questions:
1. Is your state’s summative test administered mostly via
computer at any grade?
2. Is your state’s summative test administered mostly via
computer to grade 3 students?
3. For how many years has your state allowed for grade 3
students to be administered a large-scale exam on
computer?
4. Are there any computerized, grade 3, open-ended items
requiring a keyboarded short-answer response on the
statewide exam?
What Are Other States
Doing?
Survey Questions 5 and 6:
5. Are there any grade 3, open-ended items requiring a
keyboarded extended-answer response on the statewide
exam? (For purposes of this question, consider an “extended”
response as typically requiring more than two
sentences to
receive the highest score.)
5. Has the issue of keyboard competence for younger
students in any way shaped your state’s decisions in the
selection of computerized item-types?
What Are Other States Doing?
1. Is your state’s summative test administered mostly via
computer at any grade?
State
Response
Delaware
Georgia
Hawaii
Idaho
Indiana
Kansas
Maryland
Minnesota
Nebraska
Oklahoma
Texas
West Virginia
Yes
No
Yes
Yes
Yes
Yes
Gr 5 &8science
Yes
Yes
No grade 3 test
No
Yes
9/12 state’s surveyed administer
some portion of their summative
test via computer.
 For Georgia, the computer
version is an option for the grade 3
reading retest only.
 Oklahoma reports that there is no
grade 3 computer assessment in the
state at all.
 Texas administers an assessment
in grades 2-12 online. TELPAS
(Texas English Language
Proficiency Assessment System) is
for LEP students only. That is the
only online assessment at grade 3 in
Texas.
What Are Other States Doing?
2. Is your state’s summative test administered mostly via
computer to grade 3 students?
State
Response
Delaware
Georgia
Hawaii
Idaho
Indiana
Kansas
Maryland
Minnesota
Nebraska
Oklahoma
Texas
West Virginia
Yes
No*
Yes
Yes
No
Yes
No
Yes
Yes
No
No*
Yes*
7/12 state’s surveyed administer
their summative test via computer
to grade 3.
 For Georgia, the computer
version is an option for the grade 3
reading retest only.
 Texas LEP students only.
 Limited to the Direct Assessment
of Writing. The rest of the Westest 2
is administered via paper and pencil
at this grade.
First draft can be done in pencil, but
then it must be copied into the
computer for a score.
What Are Other States Doing?
3. For how many years has your state allowed for grade 3
students to be administered a large-scale exam on
computer?
State
Response
Delaware
Georgia
Hawaii
Idaho
Indiana
Kansas
Maryland
Minnesota
Nebraska
Oklahoma
Texas
West Virginia
2years
4 years*
2years
3years
2years
6years
Was modified/ no longer
3years
3years
NA
NA
3years
9/12 state’s surveyed have
administered some test to grade 3
students for 2 years or more.
 Maryland gave a small number of
students a modified assessment
online, but they no longer administer a
modified exam.
 Kansas has been at this the
longest having six years of computerdelivered testing.
 Georgia makes this an option for
grade 3 retesters.
Indiana—not the summative test.
What Are Other States Doing?
4. Are there any computerized, grade 3, open-ended items
requiring a keyboarded short-answer response on the
statewide exam?
State
Response
Delaware
Georgia
Hawaii
Idaho
Indiana
Kansas
Maryland
Minnesota
Nebraska
Oklahoma
Texas
West Virginia
No all m-c and grid
No
Yes*
No
No
No
No
Yes*
No
No
No
No*
Only 2/12 state’s surveyed require
keyboarded short-answer responses
from students at grade 3.
 Hawaii gave a tentative “Yes” to this
question. This still needs confirmation.
 Minnesota administers the MCA in
Mathematics to 80% of the students.
The short-answer items are only
numerical. MN ran cognitive labs and
decided to limit students to just the
numbers.
 West Virginia only gives the Writing
Sample online. No short-answer openended.
What Are Other States Doing?
5. Are there any grade 3, open-ended items requiring a
keyboarded extended-answer response on the statewide
exam?
State
Response
Delaware
Georgia
Hawaii
Idaho
Indiana
Kansas
Maryland
Minnesota
Nebraska
Oklahoma
Texas
West Virginia
No
No
Yes*
No
No
No
No
No
No
No
No
Yes*
Only 2/12 state’s surveyed require
keyboarded extended-answer
responses from students at grade 3.
 Hawaii gave a tentative “Yes” to
this question. This still needs
confirmation.
West Virginia direct assessment of
writing is keyboarded at grades 3-11,
the rough draft can be by paper/pencil
but final must be keyboarded by grade
3 students. When surveyed, 85% of
3rd graders said that they preferred the
online version of the test over the
traditional paper and pencil test.
What Are Other States Doing?
5. Has the issue of keyboard competence for younger
students in any way shaped your state’s decisions in the
selection of computerized item-types?
State
Response
Delaware
Georgia
Hawaii
Idaho
Indiana
Kansas
Maryland
Minnesota
Nebraska
Oklahoma
Texas
West
Virginia
No
No
Yes*
No
No
Possibly*
Yes*
Yes*
Yes*
No
NA
Yes*
6/12 state’s surveyed have made decisions
based, at least in part on, keyboard competence.
 Hawaii gives students three opportunities to test.
 Kansas has statewide standards for keyboarding
that begin in kindergarten.
 MD moved away from administering online.
 MN’s Cognitive Labs indicated that they should
limit keyboarding to just the numbers.
 Nebraska does paper/pencil until grade 4.
West Virginia did a survey, 85% of 3rd graders said
that they preferred the online version of the test over
the traditional paper and pencil test.
A Question for Your
Responder
Given what you just learned about the experience in other
states…
Q: which of the following test environments for thirdgrade students would you currently support?
A. Third graders should only need to point and click.
B. Third graders should point and click AND type responses to
short-answer items.
C. Third graders should point and click, type responses to
short answer items, AND type responses to extended
response items (like the CMT writing sample).
March 2012 CMT DAW Pilot Test
 Over a thousand CT students per grade (range 1,381 –
1,581) participated in an online Direct Assessment of Writing
pilot which required them to keyboard their responses.
 These responses are not reported back to schools or
students. They don’t “count.”
 The purpose of the pilot test is to determine the difference in
difficulty between the two modes. If the two modes can be
“post-equated” then a choice of modes can be offered
before the 2015 online test.
 The tests were recently scored. Both the pilot scores, and
the observations of the scoring leads at the various
Measurement Incorporated scoring centers were reported to
the state.
Anecdotal Evidence From the
Scoring Center
Mathematics:
(Marianne Nasiatka,
Grade 3 Scoring
Director)
We saw evidence that
some third graders
don’t know how to use
the shift key. An equal
sign would appear in
their responses instead
of an addition symbol.
Anecdotal Evidence From the
Scoring Center
Reading and Science (Steve Zrimec,
project monitor)
The quality of the typewritten
responses appeared to be equal to
the handwritten responses from
past projects. Keyboard
unfamiliarity presented no obvious
problems. There did not appear to
be any obvious advantages for the
students (i.e. spelling, grammar,
and even ideas weren’t improved
because of the format). And
actually, this seemed true across all
grades, not just Grade 3.
Anecdotal Evidence From the
Scoring Center
Direct Assessment of Writing (Sharon Williams, Grade 3 scoring director)
We were pleasantly surprised at the typing skills. There were few responses that
indicated that the students did not know how to type. In other words, we saw very
few "fingers on the wrong keys" gibberish responses but rather a fairly competent
sample.
Readers noted (as did I) that an unusually high
number of responses ended abruptly, as if the
writers had run of time before finishing their
narratives. We wondered if this was because the
third graders are unable to type as fast as they can
write. We also noted that many of the responses
seemed brief and undeveloped compared to the
responses we saw in the spring assessment. Very
few typed responses went on to the second page,
and many were only a half page long. This could be
an issue with time and typing speed too.
A Question for Your
Responder
Now that you have heard of the observations from the Scoring
Centers…
Q: which of the following test environments for thirdgrade students would you currently support?
A. Third graders should only need to point and click.
B. Third graders should point and click AND type responses to
short-answer items.
C. Third graders should point and click, type responses to
short answer items, AND type responses to extended
response items (like the CMT writing sample).
9
CMT Handwritten vs. Typed
Score Difference by Grade
CMT Typed Response Data
8.6
8.6
8.5
8.5
8.4
8.3
Grade
Difference
8
0.3
8.3
8.2
8
S
c
o
r
e
P
o
i
n
t
7.8
2012 CMT_DAW
(HAND-WRITTEN
CENSUS)
7.7
7.6
7.5
7.5
2012 MIST_DAW
(COMPUTER PILOT)
7
6.8
6.5
6
Grade 8
Grade 7
Grade 6
Grade 5
Grade
Grade 4
Grade 3
9
CMT Handwritten vs. Typed
Score Difference by Grade
CMT Typed Response Data
8.6
8.6
8.5
8.5
8.4
8.3
8.3
8.2
Grade
Difference
8
0.3
7
0.5
8
S
c
o
r
e
P
o
i
n
t
7.8
2012 CMT_DAW
(HAND-WRITTEN
CENSUS)
7.7
7.6
7.5
7.5
2012 MIST_DAW
(COMPUTER PILOT)
7
6.8
6.5
6
Grade 8
Grade 7
Grade 6
Grade 5
Grade
Grade 4
Grade 3
9
CMT Handwritten vs. Typed
Score Difference by Grade
CMT Typed Response Data
8.6
8.6
8.5
8.5
8.4
8.3
8.3
8.2
Grade
Difference
8
0.3
7
0.5
6
0.8
8
S
c
o
r
e
P
o
i
n
t
7.8
2012 CMT_DAW
(HAND-WRITTEN
CENSUS)
7.7
7.6
7.5
7.5
2012 MIST_DAW
(COMPUTER PILOT)
7
6.8
6.5
6
Grade 8
Grade 7
Grade 6
Grade 5
Grade
Grade 4
Grade 3
9
CMT Handwritten vs. Typed
Score Difference by Grade
CMT Typed Response Data
8.6
8.6
8.5
8.5
8.4
8.3
8.3
8.2
8
S
c
o
r
e
P
o
i
n
t
7.8
Grade
Difference
8
0.3
7
0.5
6
0.8
5
0.6
7.7
7.6
7.5
7.5
7
6.8
6.5
2012 CMT_DAW
(HAND-WRITTEN
CENSUS)
2012 MIST_DAW
(COMPUTER PILOT)
6
Grade 8
Grade 7
Grade 6
Grade 5
Grade
Grade 4
Grade 3
9
CMT Handwritten vs. Typed
Score Difference by Grade
CMT Typed Response Data
8.6
8.6
8.5
8.5
8.4
8.3
8.3
8.2
8
S
c
o
r
e
P
o
i
n
t
7.8
7.7
7.6
Grade
Difference
8
0.3
7
0.5
6
0.8
5
0.6
4
1.0
7.5
7.5
7
6.8
6.5
2012 CMT_DAW
(HAND-WRITTEN
CENSUS)
2012 MIST_DAW
(COMPUTER PILOT)
6
Grade 8
Grade 7
Grade 6
Grade 5
Grade
Grade 4
Grade 3
9
CMT Handwritten vs. Typed
Score Difference by Grade
CMT Typed Response Data
8.6
8.6
8.5
8.5
8.4
8.3
8.3
8.2
8
S
c
o
r
e
P
o
i
n
t
7.8
7.7
7.6
7.5
7.5
7
6.8
6.5
Grade
Difference
8
0.3
7
0.5
6
0.8
5
0.6
4
1.0
3
1.7
2012 CMT_DAW
(HAND-WRITTEN
CENSUS)
2012 MIST_DAW
(COMPUTER PILOT)
6
Grade 8
Grade 7
Grade 6
Grade 5
Grade
Grade 4
Grade 3
CMT Typed Response Data
CMT Handwritten vs. Typed
Score Difference by Grade
12
11
S
c
o
r
e
P
o
i
n
t
10
9
8.6
8.3
7.7
8
8.6
8.4
8.3
8.2
7.8
7.5
8.5
2012 CMT_DAW
(HAND-WRITTEN
CENSUS)
7.6
6.8
7
2012 MIST_DAW
(COMPUTER PILOT)
6
5
4
3
2
Grade 8
Grade 7
Grade 6
Grade 5
Grade
Grade 4
Grade 3
Possible Contributing Factors
to This Trend
 Were the piloted (keyboarded) prompts more difficult?
 Was the novelty of computer testing a factor?
 Are the scorer’s showing a bias in favor of handwritten text?
 Because of a lack of clear standards on the pilot, did the
time of administration of the pilot generally rob students of
the full 45-minutes intended?
 If time on task is a factor, will lengthening the standard time
limit for a handwritten response be sufficient to overcome
the differences for students who are typing?
 Is keyboard familiarity (typing skill) a contributing factor?
A Question for Your
Responder
Now that you have seen the data from the CMT DAW Pilot
Test…
Q: which of the following test environments for thirdgrade students would you currently support?
A. Third graders should only need to point and click.
B. Third graders should point and click AND type responses to
short-answer items.
C. Third graders should point and click, type responses to
short answer items, AND type responses to extended
response items (like the CMT writing sample).
Connecticut’s Short-Term
Plan
1) The Department would like to encourage equipped
districts to administer more statewide tests on the
computer before 2015.
2) Pilot data will show which of the test sessions can be
equated to take into account the different difficulties of
taking tests on the computer.
3) The Department plans to give districts some level of
choice about which mode (paper-and-pencil vs.
computer submitted) for students taking the census
test.
Connecticut’s Short-Term
Plan
4) Because selected forms will be equated, this means
that a resulting scale score for the same test may have
a different raw score as a result of mode. (written vs.
computer submitted)
=
Further Research
 RESPONDER TIME: Should these CT findings be
reported to SBAC? Please use “TRUE” for YES and
use “FALSE” for NO.
 A repeated study could be done to see if the patterns
are persistent over time. (Students may be improving
on typing skill each year.)
 It is possible to design a factor analysis to try to find out
how much of the difference between the two modes is
attributable to typing skill alone.
Q&A
Online Resources
 http://teachersites.schoolworld.com/webpages/yollis/typ
ing.cfm Mrs. Yollis’ blog (above) sends you to these
fine resources:
Steve’s Review:
Learn to type AND Dance Mat
Typing Game (same site)— By the
BBC the best one here.
I learn Tech…Website—hit the drop
down “typing games.” the best ones
are:
 typing chef
 keyboard climber game
 super hyper
I also got addicted to “Typing
Defense”
Typer Shark game???—could not
find it nor “Typing Lessons”
Dance Mat Typing
 http://www.bbc.co.uk/schools/typing/
Sequential lessons
Animal guides
Reward Screens
10-minute lessons
20 small typing
tasks.
 About a dozen
lessons in the
sequence.
Typing Chef
 http://www.sense-lang.org/typing/games/typingchef.php
 Type the word in the
bubble before it pops.
 In later, more
challenging screens other
food falls into the pan.
 Game works on speed
of typing target words that
get increasingly complex.
Keyboard Climber Game
 http://www.tvokids.com/games/keyboardclimber
 Help the monkey to
climb ever higher by
correctly typing the letter
above him.
 Clue to which hand to
use based on left or right of
screen.
 Monkey falls to bottom
of current level with each
mistake.
Super Hyper Spider Typer

http://funschool.kaboose.com/fun-blaster/games/game_super_hyper_spider_typer.html
 Three levels of
difficulty.
 Each progressively
difficult within level.
 Type the letters or
words on the
chameleons’ backs to
protect the spider.
 Different colored
chameleons have
different numbers of
lives.
Typing Defense
 http://games.wordgames.com/media/typingdefense.swf
Type the word
associated with each
asteroid before it
collides with the earth.
 Planetary defense
ray destroys correctly
spelled asteroids.
 Additional key words
are earned like “slow,
ice, and strike” as you
increase in levels.
A Question for Your
Responder
Now that you attended this session…
Q: Will you fill out an evaluation sheet?
A. Yes!
B. No!
C. There is no evaluation sheet.