Transcript Document
Analyzing Data from Formative Evaluations April 14, 2008 COM 535, S08
Theoretical Approaches to Evaluating FE Data • • Quantitative Relies on data collected from tallying users’ performance on pretests, practice tests, and post tests.
Numbers-oriented.
COM 535, S08 • • Qualitative Relies on data collected from observing users’ experience interacting with the instruction.
Narrative-oriented.
Analyzing Data from FE
• • • • Different formats of FE require different types of analysis and provide different kinds of data.
One-to-One Trials Small Group Trials Field Trials COM 535, S08
One-to-One Data
Qualitative Evaluation 1.
Learner characteristics and entry behaviors 2.
Direct responses to instruction 3.
4.
Learning time Responses to a user questionnaire.
COM 535, S08 Quantitative Evaluation 1.
Posttest performance
Revising from One-to One Data
• Revisions can be based on • • • Learner suggestions Learner performance Evaluator’s reactions/observations COM 535, S08
Challenges of Using One-to One Data • • Learners will be diverse.
Individuals will have unique experiences and responses.
• General summaries, conclusions will be difficult to make.
Use this data as a preview/frame for the data you will collect in the small group and field test collections.
COM 535, S08
Small Group/Field Test Data
• • • Qualitative Evaluation Responses to user questionnaire.
Learning and testing time.
Comments made directly on materials.
• • Quantitative Evaluation Pretest performance Posttest performance COM 535, S08
Compiling Small Group/Field Test Data • Create tables to compare: • • • Students’ scores for each performance objective (p. 318) Students’ scores for pretest/postest (p. 320) Students’ scores for achieving objectives mapped onto goal analysis (p. 321) COM 535, S08
Revising from Small Group/Field Test Data • Revisions can be based on • • • Learner suggestions Learner performance Evaluator’s reactions/observations COM 535, S08
Challenges in Using Small Group/Field Test Data • • • Observations of learner interaction will be more difficult to conduct.
Data will necessarily be more quantitative than qualitative.
Performance-oriented evaluation does not always suggest how the tests should be give you substantive information on user experience.
COM 535, S08
Interpreting FE Data
• • Once data is collected and summarized, look for trends in the following areas: • • • • • Entry Behaviors Pretests and Posttests Instructional Strategy Learning Time Instructional Procedures These categories can also be used to frame debriefing/survey questions.
COM 535, S08
Instructional Component Motivational, introductory material Pretest Sample Instructional Revision Plan Problem Proposed Change Evidence None None Video meeting was too quick to watch and fill out form successfully Add instructions to pause video while filling out form.
Learners reported good attention levels, clarity of purpose Comments from debriefing survey and observations COM 535, S08