Transcript PPT 77Kb
School of Technology Atgentive - Evaluation WP 4 Oxford Brookes University Dr Mary Zajicek 1 AUP INSEAD Ontdeknet Formative Evaluation UTA Summative Evaluation OBU CELN INSEAD School of Technology Atgentive Kick-Off Meeting 7 – 8 Dec. 2005 2 Objectives of WP4 • • to develop a robust and effective evaluation program • Evaluation of early prototyping of the concepts developed in WP1 and WP2 • Evaluation of final demonstrators. document the results of evaluation, for all concepts and models developed within the project Evaluation will be at three different levels • • • • Formative evaluation: the objective is to identify the users’ needs and to define or validate usage scenarios Summative evaluation: The objective is to evaluate the work that has been produced Substantive evaluation: The objective consists in assessing the substantive value of the knowledge (technical components or approaches) that has been generated in this project. Evaluation relies upon the specification of a well defined set of key indicators (qualitative or quantitative) that will be used to access the achievements of the project (including the value of the ideas and the knowledge assets generated in the project). School of Technology Atgentive Kick-Off Meeting 7 – 8 Dec. 2005 3 Formative Evaluation - Some First Thoughts Functions of attention software: • Detection of user’s current attentional state • Detection and evaluation of possible alternative states • Strategies for presentation to the user of these alternative states Each of these requires evaluation • Evaluate the success of strategies for evaluating attention and possibly redirecting it. • Evaluate the success of the new attention position • Evaluate the models and methodologies • Perceived and objective measurements School of Technology Atgentive Kick-Off Meeting 7 – 8 Dec. 2005 4 Work for WP4 T4.1: Formative evaluation: • elaborating high-level scenarios with users, and collecting user’s preliminary feedbacks • information will be used as input for the elaboration of the conceptual framework (relevance), and will be used for adjustment purpose in the design. T4.2: Elaboration evaluation frameworks for attention • (oriented toward effectiveness rather than efficiency) • for advanced user interface (with cognitive rather than ergonomic orientation) • focus on the evaluation of the substantive value contributed by each of the mechanisms (for instance protocols of tests with or without artificial characters, with out without support for attention and direction, could be considered) • will include the specification of a set of key indicators to assess the attention software T4.3: Definition of the evaluation plan (summative evaluation) • Development of a methodology (the characteristic of the test groups, specification of the evaluation action and of the schedule of execution) T4.4: Analysis of the result (summative evaluation) T4.5: Strategic Evaluation (Lesson learned, prospective, etc.). School of Technology Atgentive Kick-Off Meeting 7 – 8 Dec. 2005 5 WP 4 Deliverables D4.1 • Evaluation approach, methodology, document (including a first draft of the key indicators) – M06 D4.2 • Result of the formative evaluation – M09 D4.3 • Evaluation plan (including a consolidated version of the indicators) – M12 D4.4 • Final evaluation report (includes both the assessment of the result and the strategic evaluation) – M23 School of Technology Atgentive Kick-Off Meeting 7 – 8 Dec. 2005 6 WP4 Milestones M4.1 Evaluation approach. – M06 • The project has now a good idea of the methodological part of the evaluation (methods that will be used for the evaluation, balance between qualitative versus quantitative evaluation, first draft of the key indicators, etc.). M4.2 Formative evaluation has been completed – M09 • The result of this evaluation (first result available before M09) will validate, guide, and help to redifine the design. M4.3 Evaluation Plan is ready – M12 • This report will provide the information necessary for organising the work of the validation pilot. M4.4 Evaluation Report – M23 School of Technology Atgentive Kick-Off Meeting 7 – 8 Dec. 2005 7