Transcript Diapositive 1

EARLI 11th Biennial conference 2005
electronic Construction and
Quality Control in Standardized Testing:
the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello,
Marc Pirson, Mbassa Dabo & Hung Lê
http://www.smart.ulg.ac.be/earli-2005/e-CQCST.ppt
University of Liège - Belgium
I. Context
• Many European Countries have
to face a growing number of
higher education students
• The training professionals and
the human resources managers of companies are often
constrained to evaluate staff competencies, often acquired
outside the schools
• There is a worldwide movement of the introduction of quality
management in education and training activities
(Kaufman & al., 1995; Segers & Dochy, 1996; Ramsden, 1991;
Nightingale & O’Neil, 1994; Zink & Schmidt, 1995; Castaigne, Gilles &
Hansen, 2001; Gilles, 2002; Gilles et Lovinfosse, 2004; Phelps, 2005)
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
2
I. Context
• The quality approach suggested by the SMART team in this context
(Gilles, 2002)
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
3
II. Principals R&D objectives of the e-C&QCST project:
• To develop a web based platform which propose methods, procedures and
tools for the management of standardized tests.
• To set up economic valorizations of the results of our research with supply
of services like:
–
–
–
–
Audit of standardized testing systems
Training and coaching of the users of the e-C&QCST platform
Docimological services for the improvement of standardized tests
Test creation with the platform
• To propose a quality label for standardized testing
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
4
III. Method: the CCQCST model
First step:
1. ANALYSIS
1. Analysis
Points to evaluate (PE),
Priorities (P), Categories
of Performances (CP)
Specification table
(PE x CP)
- list of the Points to Evaluate (PE)
- their Priorities (P)
- and the Categories of Performances (CP)
of the learners or trainees
End product :
a test specification table where
Points to Evaluate (PE) are crossed with
Categories of Performances (CP)
 binomials (PE x CP).
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
5
III. Method: the CCQCST model
2. Design
1. Analysis
Points to evaluate (PE),
Priorities (P), Categories
of Performances (CP)
Second step:
Specification table
(PE x CP)
Modalities of the
Questions (MQ),
importance of each
MQ
(CP x MQ)
(PE x CP x MQ)
selection
2. DESIGN
The test developer must design the Modalities of the Questions (MQ) according
to the Categories of Performances (CP) previously defined at step 1.
End products :
- Binomials (CP x MQ) deriving from the Categories of Performances (CP)
associated with the Modalities of the Questions (MQ)
- Trinomials (PE x CP x MQ) deriving from the binomials (PE x CP), previously
defined at step 1, assembled with the binomials (CP x MQ) of this step 2
- Selection of the trinomials which should be linked to the questions of the test
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
6
III. Method: the CCQCST model
2. Design
1. Analysis
Points to evaluate (PE),
Priorities (P), Categories
of Performances (CP)
Specification table
(PE x CP)
Modalities of the
Questions (MQ),
importance of each
MQ
3. Questions
(CP x MQ)
(PE x CP x MQ)
selection
Selection or writing of
the questions which
are linked to the
selected trinomials
(PE x CP x MQ)
Item bank
Third step:
3. QUESTIONS
The test developer selects or writes the questions.
These questions are linked to the trinomials objects
selected for the test at the former step.
End product :
A series of questions related to the content and the
performances to be test. These questions are created
or selected in a secured item bank database.
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
7
III. Method: the CCQCST model
2. Design
1. Analysis
Points to evaluate (PE),
Priorities (P), Categories
of Performances (CP)
Specification table
(PE x CP)
Modalities of the
Questions (MQ),
importance of each
MQ
(CP x MQ)
(PE x CP x MQ)
selection
Fourth step:
3. Questions
Selection or writing of
the questions which
are linked to the
trinomials selected
Item bank
4. Training
4. TRAINING
The person in charge of the test can choose to
propose trainings on the modalities of the
questions previously defined.
…on the modalities of the
questions defined
Blank tests (paper or
web based)
End product :
“blank tests” (paper-pencil or web based)
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
8
III. Method: the CCQCST model
2. Design
1. Analysis
Points to evaluate (PE),
Priorities (P), Categories
of Performances (CP)
Specification table
(PE x CP)
Modalities of the
Questions (MQ),
importance of each
MQ
(CP x MQ)
(PE x CP x MQ)
selection
Fifth step:
3. Questions
Selection or writing of
the questions which
are linked to the
trinomials selected
Item bank
4. Training
5. TESTING
The questions created or selected at the third
step are put in the standardized test which is
proposed to the people who perform the test.
End products :
…on the modalities of the
questions defined
Blank tests (paper or
web based)
5. Testing
The standardized
test is proposed
Online exams,
paper tests linked
with OMR
- online exams (in a secured environment)
- questionnaires linked with OMR
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
9
III. Method: the CCQCST model
2. Design
Sixth step:
1. Analysis
Points to evaluate (PE),
Priorities (P), Categories
of Performances (CP)
6. CORRECTION
Specification table
The standardized test
(PE x CP)
developer carries out the
correction of the tests.
Modalities of the
Questions (MQ),
importance of each
MQ
(CP x MQ)
(PE x CP x MQ)
selection
Selection or writing of
the questions which
are linked to the
trinomials selected
Item bank
4. Training
…on the modalities of the
questions defined
Blank tests (paper or
web based)
End products :
Treatments are designed to
check the quality of the
questions before the final
version of the results
3. Questions
5. Testing
6. Correction
Set up of the
correction of the test
The standardized
test is proposed
Online exams,
paper tests linked
with OMR
Quality check and
discussion of the results
before the final version
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
10
III. Method: the CCQCST model
Seventh step:
2. Design
7. FEEDBACK
Modalities of the
Questions (MQ),
importance of each
MQ
Personalized and 1.
tailored
feedbacks are
Analysis
Points to evaluate (PE),
proposed on the basis
of the final version
(CP x MQ)
Priorities (P), Categories
(PE
x CP x MQ)
of
Performances
(CP)
of the results.
selection
Specification table
(PE x CP)
3. Questions
Selection or writing of
the questions which
are linked to the
trinomials selected
Item bank
End products :
Personalized and tailored feedbacks with
online access.
4. Training
…on the modalities of the
questions defined
Blank tests (paper or
web based)
7. Feedback
5. Testing
Personalized and
tailored feedbacks
are proposed on the
basis of final version
of the results
The standardized
test is proposed
6. Correction
Set up of the
correction of the test
online access FB
Online exams,
paper tests linked
with OMR
Quality check and
discussion of the results
before the final version
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
11
III. Method: the CCQCST model
Eight step:
2. Design
1. Analysis
Points to evaluate (PE),
Priorities (P), Categories
of Performances (CP)
Specification table
(PE x CP)
8.Regulation
analysis of the opiAdvices for the
nions of the people who
next test
intervened in the test
Modalities of the
Questions (MQ),
importance of each
MQ
8. REGULATION
the test developer collects
and analyses the
3. Questions
Selection
or writing
of
opinions of the all the
people
who
intervened
the questions which
(CP x MQ)
in the
test construction
process.
are linked
to the
(PE x CP x MQ)
selection
End products :
trinomials selected
Item bank
Advices in order to improve 4.
the
next
Training
the
modalities of the
standardized test that will …on
bequestions
created.
defined
Blank tests (paper or
web based)
7. Feedback
5. Testing
Personalized and
tailored feedbacks
are proposed on the
basis of final version
of the results
The standardized
test is proposed
6. Correction
Set up of the
correction of the test
online access FB
Online exams,
paper tests linked
with OMR
Quality check and
discussion of the results
before the final version
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
12
III. Method: the CCQCST model
2. Design
1. Analysis
Points to evaluate (PE),
Priorities (P), Categories
of Performances (CP)
Specification table
(PE x CP)
Modalities of the
Questions (MQ),
importance of each
MQ
(CP x MQ)
(PE x CP x MQ)
selection
3. Questions
Selection or writing of
the questions which
are linked to the
trinomials selected
Item bank
8.Regulation
4. Training
analysis of the opiAdvices for the
nions of the people who
next test
intervened in the test
…on the modalities of the
questions defined
Blank tests (paper or
web based)
7. Feedback
5. Testing
Personalized and
tailored feedbacks
are proposed on the
basis of final version
of the results
The standardized
test is proposed
6. Correction
Set up of the
correction of the test
online access FB
Online exams,
paper tests linked
with OMR
Quality check and
discussion of the results
before the final version
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
13
III. Method: the CCQCST model
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
14
V. Companies & organizations collaborating in this project
M. Bernard Van Parys
M. Hubert de Broqueville
Mme Régine Gendarme
Mme Catherine Deruisseau
Mme Rameault
Mme Maryse Riffont
M. Peter Vanderheyden
M. Alain Kock
M. Dominique Pierret
M. Jean-Marie Pairoux
Mme Catherine Warnant
M. Jean-Philippe Léonard
M. Francis Lajoie
M. Jean-Pierre Renault
Mme Sandra Schillemans
Mme Tania Delhaise
M. Pierre Ramelot
M. Yvan Lagage
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
15
VI. First results: Relevance of the CCQCST model
e-C&QCST
Research group
Presentation of
the 8 steps of the
CCQCST model
Requests for
immediate docimological
subcontracting
Services
(AXA, FOREM,
FORTIS, IFA)
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
16
VI. First results: Relevance of the CCQCST model
• Interviews of tests developers of the ten companies and organizations:
– Description of the actions;
– Collaborators concerned;
– Temporal dimension of the tests construction process;
– Other processes in which tests are inserted;
– Importance of the quality of the standardized tests.
– …
• First results of the interviews and the immediate subcontracting
services:
– The CCQCST model and its procedures are considered as relevant and well
accepted;
– The principal readjustment we have to plan consists of an integration of different
roles of tests developers working in teams.
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
17
VI. First results: Users’ roles within e-C&QCST platform
• Two difficulties for the users’ roles definition:
– They must be used in very different companies’ or non academic organizations’
contexts;
– They should be assigned by one person in charge of the test (the head of the
project) to a series of collaborators or subcontractors.
• We propose seven roles within the e-C&QCST platform:
– Role 1: “INSTALLER”  installs the software on the server, … ;
– Role 2: “ADMINISTRATOR”  defines the different tests supervisors, … ;
– Role 3: “TEST SUPERVISOR”  defines and supervises his collaborators &
subcontractors for the tasks related to the test construction, … ;
– Role 4: “TEST DEVELOPER”  realizes one or more tasks of one or more steps
of the CCQCST model, …;
– Role 5: “QUALITY SUPERVISOR”  defines and supervises his collaborators or
subcontractors in charge of the quality controls, …;
– Role 6: “QUALITY CONTROLLER”  realizes one or more tasks related to the
quality controls of the productions of the tests developers, …;
– Role 7: “ASSESSEE”  performs the standardized test.
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
18
VI. First results: The SAFES project
• Being given the success of the subcontracting services requested by four
companies, we filed in a FIRST SPIN-OFF project of the DGTRE.
• Our project entitled “Smart Advices For Evaluation Systems (SAFES)” was
well ranked by an independent scientific committee so that we are
currently, starting a two years feasibility study that will allows us :
– To create a method of audit of standardized testing systems;
– To improve the quality control procedures to be implemented;
– To define a typology of subcontracted services;
– To realize a market study.
• The creation of the spin-off company should coincide with the finalisation
of the e-C&QCST platform.
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
19
VI. First results: First interfaces of the e-C&QCST platform
• General ergonomics:
Main
menu with
the 8 steps
of the
CCQCST
model
Assessment
selector
Information
about the
user who is
connected
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
Multi
lingual
menu
Information
about the
assessment
which is
selected
(an assessment includes one or
more standardised
tests)
20
VI. First results: First interfaces of the e-C&QCST platform
• Management of the e-C&QCST users by the test supervisor:
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
21
VI. First results: First interfaces of the e-C&QCST platform
• 1st module “ANALYSIS” - specification table: PE are crossed with CP
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
22
VII. Conclusions
• Docimological framework:
The CCQCST model is relevant and well appreciated in the context of the ten
companies and organisations which are our partners in this project
• Ergonomics aspects:
The first results show that the ergonomics of the first interfaces facilitate the
production of specification tables related to the first step of the standardized
tests construction
• Economical valorization framework:
The surprising immediate docimological subcontracting services requested by
four of the ten companies (also the acceptation of the SAFES project) shows that
there are possibilities for economical valorisations of the results of this project.
• Users framework:
The seven categories of users’ roles we defined were validated by our partners.
This clarification should contribute to highlight the complementarities and the
responsibilities of the persons who are involved in standardized test creation.
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
23
Thank you for your attention !
For more information concerning the
electronic Construction and Quality Control
in Standardized Tests project
(e-C&QCST)
do not hesitate to contact me:
[email protected]
http://www.smart.ulg.ac.be/earli-2005/e-CQCST.ppt
EARLI 11th Biennial conference 2005 – electronic Construction and Quality Control in Standardized Testing: the e-C&QCST project
Jean-Luc Gilles, Sylvie-Anne Piette, Pascal Detroz, Salvatore Tinnirello, Marc Pirson, Mbassa Dabo & Hung Lê
24