Analyzing Data from Formative Evaluations April 14, 2008 COM 535, S08

Theoretical Approaches to Evaluating FE Data • • Quantitative Relies on data collected from tallying users’ performance on pretests, practice tests, and post tests.

Numbers-oriented.

COM 535, S08 • • Qualitative Relies on data collected from observing users’ experience interacting with the instruction.

Narrative-oriented.

Analyzing Data from FE

• • • • Different formats of FE require different types of analysis and provide different kinds of data.

One-to-One Trials Small Group Trials Field Trials COM 535, S08

One-to-One Data

Qualitative Evaluation 1.

Learner characteristics and entry behaviors 2.

Direct responses to instruction 3.

4.

Learning time Responses to a user questionnaire.

COM 535, S08 Quantitative Evaluation 1.

Posttest performance

Revising from One-to One Data

• Revisions can be based on • • • Learner suggestions Learner performance Evaluator’s reactions/observations COM 535, S08

Challenges of Using One-to One Data • • Learners will be diverse.

Individuals will have unique experiences and responses.

• General summaries, conclusions will be difficult to make.

Use this data as a preview/frame for the data you will collect in the small group and field test collections.

COM 535, S08

Small Group/Field Test Data

• • • Qualitative Evaluation Responses to user questionnaire.

Learning and testing time.

Comments made directly on materials.

• • Quantitative Evaluation Pretest performance Posttest performance COM 535, S08

Compiling Small Group/Field Test Data • Create tables to compare: • • • Students’ scores for each performance objective (p. 318) Students’ scores for pretest/postest (p. 320) Students’ scores for achieving objectives mapped onto goal analysis (p. 321) COM 535, S08

Revising from Small Group/Field Test Data • Revisions can be based on • • • Learner suggestions Learner performance Evaluator’s reactions/observations COM 535, S08

Challenges in Using Small Group/Field Test Data • • • Observations of learner interaction will be more difficult to conduct.

Data will necessarily be more quantitative than qualitative.

Performance-oriented evaluation does not always suggest how the tests should be give you substantive information on user experience.

COM 535, S08

Interpreting FE Data

• • Once data is collected and summarized, look for trends in the following areas: • • • • • Entry Behaviors Pretests and Posttests Instructional Strategy Learning Time Instructional Procedures These categories can also be used to frame debriefing/survey questions.

COM 535, S08

Instructional Component Motivational, introductory material Pretest Sample Instructional Revision Plan Problem Proposed Change Evidence None None Video meeting was too quick to watch and fill out form successfully Add instructions to pause video while filling out form.

Learners reported good attention levels, clarity of purpose Comments from debriefing survey and observations COM 535, S08