Transcript Closing the Communication Gap Between Undergraduates and

Closing the Communication
Gap Between Undergraduates
and Mathematics Professors
Daniel Villarreal
November 24, 2009
Presentation Format






Basics about the project
Background Linguistics info
Mini literature review
Methodology
Results/Discussion
Homework!
Project Basics
Project Basics




Conducted in support of an Honors thesis in
Linguistics for the 2009-2010 academic year
(Committee: Charity Hudley, Taylor, Li)
Supported by a Dintersmith Fellowship
Social-science-grounded project
The inspiration…
James Villarreal
Background Linguistics Info
Terminology





Syntax
Semantics
Phonetics/Phonology
Prosody
Pronunciation
–
Includes phonetics, phonology, and prosody
Terminology

Accent
–

Dialect
–

The way a person pronounces the words in a
language
A variety of the language spoken by some predefined group (regional, social, ethnic)
Ideolect
–
The variety of the language particular to a single
person
Languages ~ Species

A good way to conceptualize linguistic
variation is to consider biodiversity
–


Example: Dogs
Just as there’s no one canonical “dog”,
there’s no canonical “English”
It is possible, however, to define a “Standard”
Linguistic Principles


Universality of Accentedness
Rule-based Accents
–



William Labov 1969: “The Logic of Nonstandard
English”
In other words, no accent is inherently
inferior to any other
But some accents may be harder for a
speaker of another dialect to understand
When that “different dialect” is Standard
American, that accent is “incomprehensible”
Mini Literature Review
Backstory


Beginning in the 1980s, the majority of
American-born STEM (Science, Technology,
Engineering, and Mathematics) graduates
began to choose jobs in industry rather than
academia (Mooney, 1989)
By 1989, more than half of STEM degree
recipients were foreign-born (ibid.)
The “Foreign TA Problem”


In 1984, the linguist Kathleen Bailey identified what
she called the “Foreign TA Problem”: “the
communicative difficulties engendered by [the
interaction between non-native speaking teaching
assistants and their students]” (Bailey, 1984, p. 3)
In a 1980 study of University of Minnesota
undergraduates, almost half reported that having a
NNS TA had hurt the quality of a course they had
taken, whereas only 9% believed that an NNS TA
had helped (cited in Bailey, 1984)
The Communication Gap

A 1989 study examined the effects of
instructor gender, student SAT score, class
term, age, international TA, and textbook on
undergraduates’ test scores in a
macroeconomics survey course.
–

Of these, no variable was responsible for a
greater drop in scores than was the presence of
an international TA (Watts & Lynch, 1989)
Anecdotal problems abound
The Communication Gap, Cont’d

So it’s just a matter of the professor speaking
in a way that the students can’t understand,
right?

WRONG

Rubin, 1999
Prof. Li’s California story
Thus, in my formulation, the communication
gap consists not only of actual
misunderstanding, but also of bias


English Language Learners


The older a second language learner is at
the time of learning a language, the more
difficult it becomes to make one’s accent
resemble a native accent (Gass & Selinker,
2001)
English language learners can have great
difficulties pinpointing the source of accentrelated communication breakdowns
(Derwing, 2003)
Professor Training

Lots of universities, especially state
universities, mandate that new hires
(especially TAs) whose first language is not
English pass TSE, TOEFL, etc. (Cassell,
2007; Plakans, 1997; Davies, Tyler, and
Koran, 1989)
–
Several researchers have doubts as to the
efficacy of these tests in evaluating classroom
readiness (Tyler, 1992; Young, 1989; Halleck and
Moder, 1995)
Student-Centered Research

Donald Rubin: “Support for ITAs (and also
continuing support for non-native English
speaking faculty members) is key, and much
progress has been made in many fine
programs on that score. But also key is
attention to undergraduates' listening
abilities. Very few--if any--programs exist to
support undergraduates as listeners of
World Englishes.” (qtd. in Gravois, 2005)
Math is Different



Byrd and Constantinides, 1992: “so many of
our early assumptions about teaching (based
on teaching styles preferred in ESL) do not
hold for the teaching of mathematics.” (p.
166)
Topic familiarity influences comprehension of
non-native speakers to a greater degree than
even accent familiarity (Gass and Varonis,
1984)
Math anxiety
The Need for This Research

In summary, there are several factors that
make this research unique and necessary:
–
–
–
–
Focuses on interactions with professors, not TAs
Shifts some of the burden of communication to
students
Focuses on the mathematics classroom
Attempts to address both components of the
communication gap: bias and genuine
misunderstanding
Methodology
Methodology




Null hypothesis: Training program not
effective in aiding students’ comprehension
Alternative hypothesis: Training program IS
effective
Population of interest: William and Mary
undergraduates
Sampling method: Random…ish
–
In social science research, recruitment is often
the hardest part!
Recruitment

Hoped to get students to participate out of
the goodness of their hearts
–
–


Help advance research that improves classroom
interactions between professor and student
Help a fellow student out with research
But in the end…
Fortunately, the Charles Center agreed to
reimburse Cheese Shop-related expenses
Testing Sessions


Six testing sessions in Dulin Learning Center
(in Swem) from October 3-6
Five parts:
–
–
–
–
–
Consent Form/Assignment
Lesson 1 + Assessment 1
Inter-lesson Module
Lesson 2 + Assessment 2
Linguistic Profile Questionnaire
Testing Sessions




Six testing sessions in Dulin Learning Center
(in Swem) from October 3-6
Five parts:
Three
different
inter-lesson modules: the
– Consent
Form/Assignment
accent
training
program,1 a Control group,
– Lesson
1 + Assessment
and
another variable
– Inter-lesson
Module group
– Lesson
+ Assessment
2
Thus,
the2variable
of interest
was the
– Linguisticin
Profile
Questionnaire
difference
scores:
A2 – A1
Experimental Design Check




But what if Assessment 2 were significantly
harder (or easier) than Assessment 1?
We need to randomize the order of
presentation to control for difficulty effects
I ended up creating three lesson videos for
6 different orders
the project, so there were ??
for presenting two videos
What do I mean by “creating lesson videos”?
Lesson Videos

In my original design, I would record
professors teaching mini-lessons (audio only)
–
–


Confidentiality
Bias effects (Rubin 1999)
I would then make Flash video animations of
these lessons, simulating professors writing
on a chalkboard
But recruiting professors turned out to be a
lot harder than I thought
Webcasts

On the advice of a friend, I looked around to
see if I could find any webcasts or open
courseware to fit my needs
–
–
–
–

At least moderate accent
Not too high-level, not too low-level
Not too long
Derivative works permitted
This turned out to eliminate just about
everything on the Web!
Lesson Videos

Using audio from the Discrete Structures
course, I chose three lessons:
–
–
–

Permutations and Combinations
The Proof That the Square Root of Two is
Irrational
Relations and the Cartesian Product
I then created three “videos” on PowerPoint,
with text appearing on the screen as Prof.
Kamala spoke it (chalkboard simulation)
Inter-Lesson Modules

Three groups, as defined by the inter-lesson
module they viewed: Control, Bias, and Training
–
–
–
Training module instructed students on accent
ideology, then specific features of Prof. Kamala’s
accent
Control module consisted of inert materials
Bias module contained an article, blog post, and
table of professor ratings meant to simulate
campus conversation by presenting professors in
a less-than-favorable light
Testing Groups

Each participant, then, had two different
treatment factors: their inter-lesson module
and the videos they viewed (and in what
order)
–
–


3 inter-lesson modules
6 permutations of 2 videos
These were independently randomized
This meant that there were 6 x 3 = 18 testing
groups
Testing Groups

Examples of testing groups:
–
–
–


Square Root of 2 > Bias > Perm-Comb
Relations > Control > Square Root of 2
Relations > Training > Square Root of 2
These would have been TERRIBLE names
Actual names for the above groups:
–
–
–
Iceberg
Sunflower
Fuchsia
Web-Based Content



Key parts of the project were hosted on the
Internet at my William and Mary webspace:
http://djvill.people.wm.edu
This helped avoid the need to download
lesson videos onto Learning Center
computers
In addition, this makes it easy to turn the
experiment into a fully web-based one
Testing Sessions




After students handed back consent forms, I
used a random number generator to
determine their testing group
Students first loaded the URL
http://djvill.people.wm.edu/%%%_lesson.html
This redirects the student to the page
containing the first lesson video for that
group
At the end of the video, the student receives
an assessment from a moderator
Testing Sessions, Cont’d



Once the student completes the assessment,
they are instructed to load
http://djvill.people.wm.edu/%%%_mid.html
This redirects the student to their group’s
inter-lesson module
Once that is complete, the student is
instructed to load
http://djvill.people.wm.edu/%%%_lesson2.html

This redirects the student to their lesson 2
Testing Sessions, Cont’d



Again, the student receives an assessment
corresponding to lesson 2
Once the student is done with the second
assessment, they are instructed to load
http://djvill.people.wm.edu/LPQ.html
This page is a Linguistic Profile
Questionnaire (on Google Docs)
–
–
–
How many Math classes taken in college
Where parents are from
Childhood exposure to accents
Discussion Sessions





In addition to testing sessions (collecting
mostly quantitative data), I held discussion
sessions to collect qualitative data
Nine discussion sessions between October
17-20, 60-90 minutes apiece
Between 4 and 11 participants in each
session
Segregated by testing group
Sandwiches!
Discussion Sessions, Cont’d

Three parts to session:
–
–
–

Reviewed ground rules and human subjects
protections
Actual discussion itself
Nuts and bolts of project
Discussion questions:
–
–
–
For starters, how many classes, if any, have you taken with
a foreign-born professor? Have any been Math classes?
Did his or her accent ever hurt your understanding of the
material?
Do you talk about professors’ accents a lot with your
friends?
Discussion Questions

More discussion questions:
–
–
–
–
–
Have you ever dropped a class or even changed your
academic plans because the professor had a foreign
accent? What about other students you know?
What do you think impedes communication between
students and professors the most, regardless of accent?
What do you wish professors (or even the College) would
do to deal with the issue of the communication gap between
undergraduates and mathematics professors?
Do you feel that you gained anything from this process?
Do you think that you are now at least somewhat better
equipped to deal with issues of accent in your instructors?
Discussion Session Ending


In discussing methodology, I finished by
walking the groups through the Training
module
This led to the final question:
–
Do you think that programs such as these would
be effective in dealing with the communication
gap?
Results/Discussion
Quantitative Results

I haven’t yet been able to do a full statistical
analysis of quantitative data
–

(cough, cough, Complex Analysis)
However, preliminary analyses are not
encouraging
Variable
Improvement
Macro-group
Bias
Control
Training
N
23
28
29
Mean
-1.09
0.79
-0.79
StDev
6.99
6.48
6.24
Minimum
-13.00
-15.00
-12.00
Median
-2.00
0.50
0.00
Maximum
10.00
12.00
12.00
IQR
12.00
9.50
6.00
Boxplot of Improvement
10
Improvement
5
0
-5
-10
-15
Bias
Control
Macro-group
Training
T-Tests

Remember, our null is that μC = μB = μT (But
we want μB < μC < μT)
Two-sample T for Improvement
Two-sample
Macro-groupT for
N Improvement
Mean StDev SE Mean
Bias
23 -1.09
6.99
1.5
Two-sample
T for
Improvement
Macro-group
N
Mean
StDev
SE
Mean
Training
29 -0.79
6.24
1.2
Bias
23 -1.09
6.99
1.5
Macro-group 28
N
Mean StDev
Control
0.79
6.48 SE Mean
1.2
Control
28
0.79
6.48
1.2
Difference = mu (Bias) - mu (Training)
Training
29 -0.79
6.24
1.2
Estimate for difference: -0.29
Difference
mu (Bias) -(-4.05,
mu (Control)
95% CI for =
difference:
3.46)
Estimate
for
difference:
-1.87
T-Test of difference = 0 (vs not =): T-Value = -0.16 P-Value = 0.875 DF = 44
Difference
= mu (Control)
- mu (Training)
95%
CI for difference:
(-5.71,
1.96)
Estimate
difference:
1.58
T-Test offor
difference
= 0 (vs
not =): T-Value = -0.98 P-Value = 0.330 DF = 45
95% CI for difference: (-1.80, 4.96)
T-Test of difference = 0 (vs not =): T-Value = 0.94 P-Value = 0.353 DF = 54
Fail to reject null
hypothesis
ANOVAs
Source
Macro-group
Error
Total
S = 6.546
Level
Bias
Control
Training
DF
2
77
79
SS
54.3
3299.3
3353.5
R-Sq = 1.62%
N
23
28
29
Mean
-1.087
0.786
-0.793
Pooled StDev = 6.546
MS
27.1
42.8
F
0.63
P
0.534
R-Sq(adj) = 0.00%
StDev
6.986
6.483
6.241
Individual 95% CIs For Mean Based on
Pooled StDev
---------+---------+---------+---------+
(-------------*------------)
(-----------*-----------)
(-----------*-----------)
---------+---------+---------+---------+
-2.0
0.0
2.0
4.0
ANOVAs
Source
Lesson Order
Error
Total
S = 4.284
Level
PC-R2
PC-Rel
R2-PC
R2-Rel
Rel-PC
Rel-R2
N
12
16
14
10
12
16
DF
5
74
79
SS
1995.4
1358.1
3353.6
R-Sq = 59.50%
Mean
-5.000
1.563
6.000
5.200
-0.667
-7.438
StDev
3.464
4.647
4.224
3.736
4.755
4.442
Pooled StDev = 4.284
MS
399.1
18.4
F
21.74
P
0.000
R-Sq(adj) = 56.76%
Individual 95% CIs For Mean Based on
Pooled StDev
---------+---------+---------+---------+
(----*----)
(---*---)
(----*----)
(----*-----)
(----*----)
(---*---)
---------+---------+---------+---------+
-5.0
0.0
5.0
10.0
Why these results?

Major discrepancy in difficulty of tests
Level
PermComb
Relations
Root2


N
28
28
24
Mean
12.714
14.857
8.375
StDev
3.184
2.663
4.009
Individual 95% CIs For Mean Based on
Pooled StDev
--+---------+---------+---------+-----(----*----)
(----*----)
(-----*----)
--+---------+---------+---------+-----7.5
10.0
12.5
15.0
Not enough participants for the number of
testing groups
Training module needs improvement
In the end…



That’s just how it goes with research in the
social sciences, especially with research on
humans
A project like this is often the first in a series
of many
Plenty of qualitative data gathered from
Linguistic Profile Questionnaire and
discussion sessions (about 6 ½ hours worth
of participants’ responses)
Questions?

Any questions or comments?
Homework

As part of your homework, you will simulate
being a participant in a testing session

http://djvill.people.wm.edu/<groupname-nocaps>_lesson.html

and follow the instructions from there
When you’re done Assessment 1, go to
http://djvill.people.wm.edu/<groupname-nocaps>_mid.html

Finally, there’s a discussion question for you
to answer on the BlackBoard discussion
board (not for extra credit)
Works Cited (in order of appearance)





Mooney, C. J. (1989, January 25). Uncertainty is Rampant as
Colleges Begin to Brace for Faculty Shortage Expected to Begin
in 1990’s. Chronicle of Higher Education, A14-A17.
Bailey, K. M. (1984). The “Foreign TA Problem”. In K. Bailey, F.
Pialorsi, J. Zukowski/Faust (Eds.), Foreign Teaching Assistants
in U.S. Universities (3-15).
Watts, M., & Lynch, G. (1989). The Principles Course Revisited.
The American Economic Review, 79, 236-241. Retrieved from
http://www.jstor.org/stable/1827763.
Gass, S. M., & Selinker, L. (2001). Second Language
Acquisition: An Introductory Course (2nd ed.). Mahwah, New
Jersey: Lawrence Erlbaum Associates.
Derwing, T. (2003). What Do ESL Students Say About Their
Accents? The Canadian Modern Language Review, 59, 547566.
Works Cited (in order of appearance)



Cassell, Edith Camilla (2007). Understanding community
linguistic diversity: An ecological approach to examining
language use patterns of international graduate students. Ph.D.
dissertation, Purdue University, United States -- Indiana.
Retrieved June 1, 2009, from Dissertations & Theses: Full Text
database. (Publication No. AAT 3287301).
Plakans, B. (1997). Undergraduates' Experiences with and
Attitudes toward International Teaching Assistants. TESOL
Quarterly, 31. Retrieved April 27, 2009, from
http://www.jstor.org/stable/3587976.
Davies, C. E., Tyler, A., & Koran, J. J., Jr. (1989). Face-to-Face
with English Speakers: An Advanced Training Class for
International Teaching Assistants. English for Specific
Purposes, 8, 139-153.
Works Cited (in order of appearance)






Tyler, A. (1992). Discourse Structure and the Perception of
Incoherence in International Teaching Assistants' Spoken Discourse.
TESOL Quarterly, 26, 713-726. Retrieved from
http://www.jstor.org/stable/3586870.
Young, R. (1989). Introduction. English for Specific Purposes, 8, 101107.
Halleck, G. B., & Moder, C. L. (1995). Testing Language and Teaching
Skills of International Teaching Assistants: The Limits of
Compensatory Strategies. TESOL Quarterly, 29, 733-758.
Gravois, J. (2005, April 8). Teach Impediment. Chronicle of Higher
Education, 51, A10. Retrieved from
http://chronicle.com/free/v51/i31/31a01001.htm.
Byrd, P., & Constantinides, J. (1992). The Language of Teaching
Mathematics: Implications for Training ITAs. TESOL Quarterly, 26,
163-167. Retrieved from http://www.jstor.org/stable/3587384.
Gass, S., & Varonis, E. (1984). The effect of familiarity on the
comprehensibility of nonnative speech. Language learning, 34, 65-89.