Improving Pedagogical Efficiency and Effectiveness using

Transcript Improving Pedagogical Efficiency and Effectiveness using

Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System

Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski University of British Columbia Okanagan

Introduction

Technology can improve teaching efficiency and delivery, but the most repetitive and time-consuming tasks of assignment and test preparation and marking are still mostly manual. Automated testing systems have the potential to dramatically reduce the amount of time spent on these repetitive tasks, freeing up instructor time for valuable student contact, and allowing for more uniformity on student and instructor achievement and evaluation. We have developed the

AutoEd

system that is an automated question generation and marking system and evaluated it in a first year Physics course.

Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Copyright© 2011 Page 2

Background – Testing Systems

There is a variety of technology for student testing:  clickers, web-based learning, course management systems, etc.

It is challenging to determine which technologies are appropriate for certain teaching styles and courses, and what are the correct products and systems that provide the appropriate payback for student learning and institutional deployment.

Most of the question systems are limited to

static

questions.

 There is no learning value in repetitive practice.  The questions do not encourage or improve learning; they simply test if previous learning has occurred.

Background – Question Templates

question template

is a question that tests a specific concept that can be re-used by varying its contents.

Example:

Calculate the distance travelled by a car in 2 hours that is travelling 60 km/h.

Template:

Calculate the distance travelled by a $x$ in $y$ that is travelling $z$.

Randomization:

For x, y, and z.

Answer:

y*z (with correct units)  Incorrect answers can also be encoded with feedback.

Previous Systems

Previous question template systems:  WWWAssign built by Dr. Larry Martin for physics courses. Currently a commercial product called WebAssign.

 Aplia - social sciences  Mastering Physics  Gradiance - computer science  Carnege Learning – mathematics These products have textbook questions and provide web hosting.

The cost is $15-$100 per student per course.

Course management systems such as WebCT/Blackboard have question systems and some support for randomized questions.

Previous Systems – Open Source

There have been several university led efforts on building these systems and providing access. Quest Service at The University of Texas at Austin  many science/math questions  outside users can get access by requesting UT Austin accounts LON-CAPA is an open source system built primarily at Michigan State University.  It has hundreds of thousands of resources and questions that are shared by multiple institutions. Institutions are part of a sharing network for distributing questions and resources.

Selecting a System

Selecting an appropriate system depends on one key decision:

Who pays?

1) Student pays  another cost in addition to textbooks paid by students  students will be reluctant to pay especially if not used extensively  commercial systems are great from an instructor point of view 2) Institution pays  Highly expensive for the institution so likely limited to open source systems. Both of which have their own restrictions.

 An in-house system is the only other choice.

AutoEd System Overview

The

AutoEd system

was designed and built by Alyosha Pushak (undergrad honours) and Dr. Lawrence in computer science.

 It is written completely in Java and uses open source systems for the database and web server.

 The database stores all user question and answers.

 Users interact with the system using a web browser.

Students can:  Practice questions and do assignments as given by the instructor. Students receive immediate feedback.

Instructors can:  Create practice questions and assignments.  Examine student assignment progress and marks at any time.

AutoEd – Student Test

AutoEd – Question Authoring

AutoEd – Student Completion Rate

Physics Evaluation

The AutoEd system was evaluated in Phys 112 in Fall 2010:  2 paper assignments, 2 electronic assignments, 1 choice  Electronic assignments were converted from paper assignments by adding randomization.

Surveys conducted at the first and last classes.

 First survey: 105 out of 111 surveys completed (95.5% return)  Second survey: 76 out of 89 surveys completed (85.4% return) Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Copyright© 2011 Page 12

Physics Evaluation Results



Format choice:

90% of the students used

AutoEd

for all three assignments and

92% used it for the last assignment

. 

Paper or electronic preference:

70% chose electronic while an additional 15% chose both as equal. 

Experience:

Over 72% of the students chose satisfactory or very satisfactory when asked about their experience using the system while only 3% rated it as very unsatisfactory.



Help learning:

AutoEd

Almost half of the students (45%) reported that helped their learning. 

Completion:

Students reported that on time (84%).

AutoEd

helped them complete their assignments (81%) and hand their assignments in 

Other uses:

Over 92% of the students thought

AutoEd

should be used again in Physics 112, and 87% felt that it could be used in other classes (chemistry, biology, mathematics, English). Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Copyright© 2011 Page 13

Insights and Improvements

1) Watch technical issues with the system and questions.  Issues with incorrect marking and significant figures.

 Question preparation takes time and all answers must be checked before releasing the assignment.

2) Users will use the Internet to find answers.

 5% of users wanted the system to block Internet searches.

 Copy-and-paste electronic question versus typing it in!

 Effectiveness as an evaluation tool as more answers are online?

3) Students really do wait until the last minute.

 By removing the overhead of assignment generation and collection/marking, more frequent, smaller assignments is better.

PHYS 122: Paper vs. Electronic

% handed in A1 A2 A3 2011 - online 93% 90% 93% Average grade [%] 2011 - online 84.4 71.3 77.5 Median grade [%] 2011 - online 100 100 100 A4 98% 86.3 100

PHYS 122: Question Writing

Q3: Using Kirchhoff’s rules for the circuit below find: a)The current in each resistor. [8 marks] b) The potential difference between points c and f [2 marks]

Randomize R 3 from 4.00 k Ω to 10.0 kΩ, increments 1.0 kΩ.

PHYS 122: Question Writing (2)

Q2. A conductor suspended by two flexible wires (as shown below) has a mass per unit length of

0.040 kg/m.

What current must exist in the conductor for the tension in the supporting wires to be zero when the magnetic field,

B = 3.60 T

points into the page? Give your answer with 3 sig. figs. [5 marks] Randomize B from 2.60 T to 4.60 T, intervals 0.10 T. Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System UBCO Research Week Presentation 2011 University of British Columbia Copyright© 2011 Page 17

Future Work and Conclusions

It is possible to build a customized question answer system with minimal cost. Although the upfront costs and time is higher, the payback in terms of student costs and instructor flexibility is significant. We are currently evaluating the system in several other courses and will release it as open source to the community.

The system will be maintained and extended for many years.

Limited time offer:  We will create the questions for you over the summer for any courses that you may be interested in using the system for!

AutoEd – Demonstation

Let's have a quick tour of the AutoEd system.

Improving Pedagogical Efficiency and Effectiveness using

Transcript Improving Pedagogical Efficiency and Effectiveness using