Transcript REACTIONS TO DIALANG AND ITS FEEDBACK

Users’ reactions to innovative
computerised feedback – the
case of DIALANG
Ari Huhta
Centre for Applied Language Studies
University of Jyväskylä
Finland
What is DIALANG?


computerized diagnostic language
assessment system in 14 languages
(www.dialang.org)
feedback (& activities) that help
learners to




diagnose their strengths and weaknesses
plan further language learning
raise awareness
unusually wide range of feedback for
a test; partly unique / innovative,
e.g. self-assessment
What is ’feedback’?
Why feedback?





In educational contexts FB can be:
”… any communication or procedure to inform a
learner of the accuracy of a response, usually to an
instructional question” (Knowledge of the Correct
Response)
but also ”… other information such as precision,
timeliness, learning guidance, motivational messages,
... critical comparisons, and learning focus”
”… any message or display that a computer presents
to the learner after a response” (Mory 2004)
improve performance & learning (directly)
but also to motivate, make learning more interesting,
encourage different learning, raise awareness, …
Why study (DIALANG) feedback?

feedback is an integral part of all
behaviour, performance and learning



social psychology, organizational theory,
education ...
feedback from tests is not much
researched (cf. teachers as
assessors)
feedback from computerized
(language) tests is particularly little
researched
DIALANG FEEDBACK
Extended
Advice
level
descriptions Level
(test result)
Self-assm
feedback
Info
about
SA
Immediate
item review
VSPT
feedback
Post-test
item review
+ subskill
Research topics / questions
This study:
 What are users’ reactions to different types of
feedback in DIALANG? How do new types of
feedback compare with more traditional
feedback?
 Are users’ reactions associated with their
background characteristics (e.g. sex, age,
language use, language proficiency)? How might
possible associations be explained?
-------------------------------------------------------
related, more in-depth studies users’ reactions to selfassessment and Vocabulary Size Placement Test
Research methods & data


Quantitative & qualitative (crosssectional)
Questionnaire




553 respondents in two countries
mostly non-language majors at universities
(typical users)
Interviews of users and their teachers
Written reports from users & teachers
Users’ ’reactions’?
Defined in terms of
 1) Reading / using particular
feedback?
 2) Interest in particular feedback?
 3) Perceived usefulness of particular
feedback?
 4) Other reactions (e.g. ease of use)


Likert scale + open-ended questions
cf. Technology Acceptance Model
Who were the respondents?
Gender of respondents
80
60
% 40
20
0
female
male
Age of respondents
Age of respondents
80
70
60
50
Percent 40
30
20
10
0
under 18
18 - 25
26 - 35
Age
36 - 45
46 - 55
# of languages & tests taken
Number of language taken
90
80
70
60
50
Percent
40
30
20
10
0
1
2
3
4
Number of languages
Number of tests taken
30
25
20
% 15
10
5
0
1
2
3
4
5
6
7
8
9
Number of tests
10 11 12 13 14
Most common test languages
Most common test languages
50
40
30
%
20
10
0
English
French
Swedish
German
Finnish
Results: Overall reaction to FB
- use & interest & usefulness
Level
Immediate
item review
Post-test item
review
Self-assessm.
FB
SA information
Extended
descriptions
Advice
VSPT
feedback
0,00
0,50
1,00
1,50
Mean
2,00
2,50
3,00
Reactions to Level (test result)
Level (interest & usefulness)
60
50
40
% 30
20
Level
10
Immediate
item review
Post-test item
review
Self-assessm.
FB
0
Clearly Somewhat Somewhat
Very
negative
negative positive on positive on
on average on average average
average
SA information
Extended
descriptions
Advice
VSPT
feedback
0,00
0,50
1,00
1,50
Mean
2,00
2,50
3,00
Reactions to Level (test result)



the most popular part of feedback
the fact that the meaning of A1 etc
was defined was appreciated
associations with user background:


correlated with proficiency: higher
proficiency – more positive reactions
men were less interested in this than
women
Reactions to Vocabulary Size
Placement Test feedback / result
VSPT feedback
50
Level
Immediate
item review
Post-test item
review
Self-assessm.
FB
40
30
%
20
SA information
Extended
descriptions
10
Advice
0
VSPT
feedback
0,00
0,50
1,00
1,50
Mean
2,00
2,50
3,00
Clearly Somewhat Somewhat
Very
negative on negative on posivite on positive on
average
average
average
average
Reactions to Vocabular Size Placement
Test & its feedback


VSPT and its result was the most controversial part
of DIALANG although, overall, the reactions were
positive
strong affective reactions, opinions rather divided



background: in general, those who got low VSPT
results regarded it very negatively, whereas those
who got good results regarded it very positively


“Interesting, I had never thought of a way like this to
test language proficiency. Fun to do!”
“Absolute nonsense! This can’t be my level!”
in particular, women at level C1/C2 liked VSPT
feedback
explanation: VSPT scoring is affected by test-taking
strategy and produces very low scores too easily
Immediate item review
50
40
%
30
20
10
0
Clearly SomewhatSomewhat Very
negative negative positive onpositive on
on
on
average average
average average
Level
Immediate
item review
Post-test item
review
Self-assessm.
FB
SA information
Post-test item review
Extended
descriptions
Advice
50
40
VSPT
feedback
%
0,00
0,50
1,00
1,50
Mean
2,00
2,50
3,00
30
20
10
0
Clearly SomewhatSomewhat Very
negative negative positive onpositive on
on
on
average average
average average
Reactions to immediate item review




almost 60% preferred immediate review, 30% posttest review, 10% preference
Why like: want to know immediately, remember better
if see the result right away, want to learn (during test)
Why don’t like: disturbs test-taking, can be depressing
background:




beginners liked it more than advanced learners
in particular, C-level (C2) learners considered it less
useful and interesting
C-level men used it particularly infrequently
possible explanation:


many beginners & intermediate learners want to learn
from tests??
advanced learners make so few mistakes that they may
feel immediate feedback is waste of time??
Reactions to post-test item review


less popular than immediate review but a
fair number of users absolutely want to
have it
Background:


more experienced users (# of languages used
& length of language studies) considered this
more interesting & useful
possible explanation (very tentative):

more experienced language learners may find
information about subskills interesting??
Reactions to self-assessment feedback
(comparison of SA and test result)
Self-assessment feedback
50
40
%
Level
Immediate
item review
Post-test item
review
Self-assessm.
FB
30
20
10
0
Clearly SomewhatSomewhat Very
negative negative positive onpositive on
on
on
average average
average average
SA information
Extended
descriptions
Advice
VSPT
feedback
0,00
0,50
1,00
1,50
Mean
2,00
2,50
3,00
Reactions to self-assessment feedback

according to users, SA is important part of
(a system like) DIALANG,


background:



as ‘popular’ as e.g. item review (joint second)
men had more often very negative opinions
compared with women
the youngest group (under 18) found it less
interesting than the older groups
possible explanation:

younger learners may not (yet) be so
interested in the kind of reflection on one’s
skills involved in self-assessment??
Reactions to self-assm. information (why
SA and test result may not match)
Information about SA
50
40
30
Level
%
20
Immediate
item review
Post-test item
review
Self-assessm.
FB
10
0
SA information
Clearly Somewhat Somewhat
Very
negative on negative on positive on positive on
average
average
average
average
Extended
descriptions
Advice
VSPT
feedback
0,00
0,50
1,00
1,50
Mean
2,00
2,50
3,00
Reactions to SA information


the least used / interesting / useful part of feedback
Background: negative reactions were associated with
higher proficiency, especially C1/C2 women
 more experience in lg learning (more lgs studied)
 male respondents (more very negative opinions)
 young age (under 18)
positive reactions were associated with:
 lower proficiency, especially A1/A2 men
 less experience in lg learning (fewer lgs studied)




men were divided in their opinions of SA information
possible explanations:


if SA and test agreed, no need to read this
beginners have less experience in what is described in SA
information and may find it new & potentially useful;
advanced learners may feel they already know all that
Reactions to extended descriptions
Extended level descriptions
50
40
Level
%
Immediate
item review
Post-test item
review
Self-assessm.
FB
30
20
10
0
Clearly Somewhat Somewhat
Very
negative negative positive on positive on
on average on average average
average
SA information
Extended
descriptions
Advice
VSPT
feedback
0,00
0,50
1,00
1,50
Mean
2,00
2,50
3,00
Reactions to extended descriptions


fairly interesting, useful and used
but less so than e.g. item review
background:

men made more often very negative
evaluations of this feedback than
women
Reactions to advice
Advice
50
40
%
Level
Immediate
item review
Post-test item
review
Self-assessm.
FB
30
20
10
0
Clearly SomewhatSomewhat Very
negative negative positive onpositive on
on
on
average average
average average
SA information
Extended
descriptions
Advice
VSPT
feedback
0,00
0,50
1,00
1,50
Mean
2,00
2,50
3,00
Reactions to advice


quite popular but not among all users –
surprise?
Background:




high proficiency (C1/C2) & more experience 
more negative views of advice
men had more very negative views than women
A1 / A2 learners were divided: they often chose it
as one of the 3 most useful types of FB but many
also chose it as one of the least useful types of
FB
possible explanation:

C-level learners no longer need advice, whereas
A-level learners do, but some of them may feel
the particular advice in DIALANG is not useful??
Implications / questions


implications for DIALANG and/or
other such computerized systems
users clearly like to have feedback
about their self-assessment


younger learners may need to be
motivated first
good to have two types of item
review: immediate and post-test,
based on choice

preferences differ; level of proficiency
Implications / questions

VSPT divides opinions: love – hate


scoring needs revising
advice appears useful for beginners in
particular
challenges
”I don’t bother to read the advice because I
know I’ll do things my way anyway”
”... you don’t remember them any more when
you study”



longitudinal studies
how best integrate systems like DIALANG
into teaching

(long-term) effectiveness of feedback?