Transcript Part II: The 3 T’s of Online Assessment: Tools, Techniques, and (Saving) Time Curtis J.

Part II: The 3 T’s of Online
Assessment: Tools, Techniques, and
(Saving) Time
Curtis J. Bonk, Professor, Indiana University
President, CourseShare
http://php.indiana.edu/~cjbonk; [email protected]
Vanessa Dennen, Assistant Professor
Florida State University
[email protected]
Do you have a strategic plan for
evaluation and assessment?
Bonk et al.’s (in press) Online Learning
Assessment and Evaluation Model
Considerations in Evaluation Plan
8. University
or
Organization
7. Program
6. Course
5. Tech Tool
1. Student
2. Instructor
3. Training
4. Task
Bonk, C. J., Wisher, R. A. & Champagne, M. V. (in press).
Toward a comprehensive model of e-learning evaluation: The Components.
What to Evaluate?
1.
2.
3.
4.
5.
6.
7.
8.
Learner—attitudes, learning, jobs.
Instructor—popularity, course enrollments.
Training—internal and external.
Task--relevance, interactivity, collaborative.
Tool--usable, learner-centered, friendly,
supportive.
Course—interactivity, completion rates.
Program—growth, long-range plans.
Organization or University—cost-benefit,
policies, vision.
1.







Measures of Student Success
(Focus groups, interviews, observations,
surveys, exams, records)
Increased Comprehension & Achievement
High Student Attitudes
High Retention, Completion Rates in Program
Jobs Obtained, Internships
Enrollment Trends for Next Semester
Grades, Achievement, Certifications
Computer Log Activity; e.g., Number of Posts,
Participation, Messages/day, Time in System
1. Student High-End Success






Message complexity, depth,
interactivity, questioning
Collaboration skills
Problem finding/solving and critical
thinking
Challenging and debating others
Case-based reasoning, critical thinking
measures
Portfolios, performances, PBL activities
Assessments Possible









Quizzes and Tests
Peer Feedback, Mentoring, Responsiveness
Tasks Attempted or Completed, Usage, etc.
Discussion/Forum Participation
Writing, Blogs, Weekly Reflections
Cases and Problems
Group Work
Web Resource Explorations & Evaluations
Performances, Portfolios, etc.
Issues to Consider…
1.
2.
3.
4.
5.
6.
7.
Bonus pts for participation?
Pts for peer evaluation of work?
Assess improvement?
Are tests timed? Allow retakes if lose
connection? How many retakes?
Give unlimited time to complete?
Cheating? Is it really that student?
How measure competency and
demonstrate learning online?
2. Instructor Success




High student evals, more signing
up, student recommendations
High student completion rates
Utilize Web to share teaching
Course recognized with awards
3. Training: Outside Support






Training (FacultyTraining.net)
Courses & Certificates (JIU, e-education)
Reports, Newsletters, & Pubs
Aggregators of Info (CourseShare, Merlot)
Global Forums (FacultyOnline.com; GEN)
Resources, Guides/Tips, Link
Collections, Online Journals, Library
Resources
3. Training
Inside Support…






Instructional Consulting
Mentoring (strategic planning $)
Small Pots of Funding
Laptops
Summer and Year Round Workshops
Colloquiums, Tech Showcases, Guest
Speakers, Awards, Recognitions

Newsletters, guides, active learning grants, annual
reports, faculty development, brown bags
5
3
RIDIC -ULO US
Technology Use
Model of
4. Tasks (RIDIC):





Relevance
Individualization
Depth of Discussion
Interactivity
Collaboration-Control-ChoiceConstructivistic-Community
RIDIC5-ULO3US Model
of Technology Use
5. Tech Tools (ULOUS):





Utility/Usable
Learner-Centeredness
Opportunities with Outsiders Online
Ultra Friendly
Supportive
6. Course Success






Few technological glitches/bugs
Adequate online support
Increasing enrollment trends
Course quality (interactivity rating)
Monies paid
Accepted by other programs
7. Online Program or Course Budget
(i.e., how pay, how large is course, tech fees charged, # of
courses, tuition rate, etc.)

Indirect Costs: learner disk space,
phone, accreditation, integration with
existing technology, library resources, on
site orientation & tech training, faculty
training, office space

Direct Costs: courseware,
instructor, help desk, books, seat time,
bandwidth and data communications,
server, server back-up, course developers,
postage
7. Program:
Online Content Considerations





Self-Paced or Live mentors?
Interactive or content dumping?
Individual or Collaborative?
Lecture or problem-based
learning?
Factual or performance
assessment?
8. Institutional Success

E-Enrollments from






new students, alumni, existing students
Additional grants, new State monies
Press, publication, partners,
attention
Cost-Benefit model, ROI
Faculty attitudes, training, support
Acceptable policies, ADL compliant
Let’s Focus at the Individual Level
What Online Testing Tools Do You Use?
What Can Online Tests Do?





Assess student progress
Allow for self-assessment
Provide standards for success
Timed testing and retesting
Opportunity for instructor
commenting
Test Selection Criteria
(Hezel, 1999)







Easy to Configure Items and Test
Handle Symbols
Scheduling of Feedback (immediate?)
Provides Clear Input of Dates for
Exam
Easy to Pick Items for Randomizing
Randomize Answers Within a Question
Weighting of Answer Options
More Test Selection Criteria





Recording of Multiple
Submissions
Timed Tests
Comprehensive Statistics
Summarize in Portfolio and/or
Gradebook
Confirmation of Test Submission
More Test Selection Criteria
(Perry & Colon, 2001)





Supports multiple items types—multiple
choice, true-false, essay, keyword
Can easily modify or delete items
Incorporate graphic or audio elements?
Control over number of times students
can submit an activity or test
Provides feedback for each response
More Test Selection Criteria
(Perry & Colon, 2001)




Flexible scoring—score first, last,
or average submission
Flexible reporting—by individual
or by item and cross tabulations.
Outputs data for further analysis
Provides item analysis statistics
(e.g., Test Item Frequency
Distributions).
How Use Technology to Assess?
What Assess Online?
1.
2.
3.
4.
Traditional Online Exams
Self-Test Exams
Learner-Content Interactions
Guided Explorations: Virtual Tours and
Timelines
5. Cases and Vignettes
6. Blogs
7. Chats with Visual Representations
8. Soft Skill Simulations
9. Virtual Reality and Role Play Simulations
10.Cyber Fashion Shows & Music Performance
1. Traditional Online Testing
Using WebCT Quizzes in a HighDemand Environment (Brothen & Wambach,
Technology Source, May/June 2003)
“Several reviews and metaanalyses…have found superior
student learning in PSI compared
to traditional lecture/discussion
methods.” Here, students read a
textbook and when they are
ready, they take chapter quizzes;
after they master one chapter,
they move on to the next.”
2. Online Self-Testing
3. Learner-Content Interactions
(Option 6 Designers)
4. Guided Explorations: Virtual
Tours and Timelines
4. Guided Explorations: Interactive
Adventure Content (Andrew Revkin, New York
Times, May 25, 2003)
5. Case-Based Learning:
My Patient.com and SimTeacher
6. Blogs (diaries, writing)
7. Visual with Chat:
Learningbydoing.net
Participants: a facilitator of online therapy,
students at all levels, a doctoral candidate
in DE, administrators, teachers, lecturers,
researchers, a physicists, a professor of
Psychology, a professor of Mathematics, a
consultant in training, an HR trainer, and a
psychotherapist. We were located in
Herzelia, a beach town north of Tel Aviv,
Stanford California, Baltimore, Montreal,
and Ismir, Turkey.
Games and Simulations
“There’s something new on the
horizon, though: computerbased soft skills simulations,
which let learners practice skills
such as negotiation and team
building.”
Clark Aldrich, The State of Simulations, Sept. 2001,
Online Learning
8. Simulations: Virtual Univ
Adminstrator & Virtual Leader
9. Virtual Worlds/Virtual Reality
Avatars--representations
of people
Objects--representations of objects
Maps--the landscape which can be explored
Bots--artificial intelligence
10. Online Performances
(e.g., Cyber Fashion Shows)
Which might you use?
How would you use?
Online Survey
Tools for
Assessment
Sample Survey Tools



Zoomerang
(http://www.zoomerang.com)
SurveyMonkey
(http://surveymonkey.com)
QuestionMark
(http://www.questionmark.com/home.html)

Survey Solutions from Perseus
(http://www.perseusdevelopment.com/fromsurv.htm)

Infopoll (http://www.infopoll.com)
Sample Survey Tools

Active Feedback

(http://www.activefeedback.com/af)

SurveyKey
(http://www.surveykey.com)
EZSurvey from Raosoft
(http://www.raosoft.com/)

SurveyShare



(http://SurveyShare.com; from
Courseshare.com)
Survey Student Opinions
(e.g., InfoPoll, SurveySolutions,
Zoomerang, SurveyShare.com)
Online Survey in Blackboard
Web-Based Survey Advantages






Faster collection of data
Standardized collection format
Computer graphics may reduce
fatigue
Computer controlled branching
and skip sections
Easy to answer clicking
Wider distribution of respondents
Why Conduct Online Surveys




Formative assessment of class
Increase student voice and
ownership in class
Involve students from other
locations
Quickly gather answers to
questions
Web-Based Survey
Problems: Why Lower
Response Rates?






Low response rate
Lack of time
Unclear instructions
Too lengthy
Too many steps
Can’t find URL
Survey Tool Features







Support different types of items
(Likert,
multiple choice, forced ranking, paired comparisons,
etc.)
Maintain email lists and email
invitations
Conduct polls
Adaptive branching and cross
tabulations
Modifiable templates & library of past
surveys
Publish reports
Different types of accounts—hosted,
corporate, professional, etc.
Web-Based Survey
Solutions: Some Tips…








Send second request
Make URL link prominent
Offer incentives near top of request
Shorten survey, make attractive, easy
to read
Credible sponsorship—e.g., university
Disclose purpose, use, and privacy
E-mail cover letters
Prenotify of intent to survey
Plagiarism
Increasing Cheating Online
($7-$30/page, http://www.syllabus.com/ January, 2002,
Phillip Long, Plagiarism: IT-Enabled Tools for Deceit?)

http://www.academictermpapers.com/
http://www.termpapers-on-file.com/
http://www.nocheaters.com/

http://www.cheathouse.com




http://www.realpapers.com/
http://www.pinkmonkey.com/
(“you’ll never buy Cliffnotes again”)
Sample Paper Assistance
(i.e., Cheating) Sites
Sample Paper Assistance
(i.e., Cheating) Sites

“In recent years there has been a
sharp increase in students cutting
and pasting material into papers
from Web sites without attribution,
or purchasing term papers from
online term-paper mills.”
Cell Phones to Cheat on
Accounting Exam (1/30/03)
Plagiarizing Plagiarism
(June 8, 2003; Dartmouth Review)


“Michael Heberling—an author at the
Chronicle and president of Baker
College for Graduate Studies—read an
article written in Syllabus magazine
that looked remarkably like his own
piece. The subject of the plagiarist’s
piece: plagiarism.”
the offending author, a graduate
student, apologized for any passages
accidentally cut-and-pasted into her
own article. Her excuse: ‘deadline
pressure.’
When Values Collide, NY
Times, p. 4, July 15, 2003


“Some of the tutors
make questionable
decisions. Writing
papers for the
players, if that is true,
is very questionable.”
“I guess it is difficult,
if not impossible, to
stop the faculty
members that are
devout fans and give
any grade they want.”
Ok, Million Dollar Question: How
can you reduce cheating online?
1. Reducing Cheating Online
http://www.syllabus.com/ January, 2002, Phillip
Long, Plagiarism: IT-Enabled Tools for Deceit?)
http://www.syllabus.com/article.asp?id=5916






Ask yourself, why are they cheating?
Do they value the assignment?
Are tasks relevant and challenging?
What happens to the task after
submitted—reused, woven in, posted?
Due at end of term? Real audience?
Look at pedagogy b4 calling plagiarism
police!
Ways to Reduce Cheating Online
(How to
Proctor from a distance, Dan Carnevale, Chronicle of Higher
Ed, Dec., 12, 1999)
2. Use proctored exams
3. Require cameras during discussions
and tests
4. Get to know your students better
through discussions and chats
5. Give random quizzes using chat tool
6. Write a long essay at start of semester
to serve as an index of writing style
7. When in doubt, place document into a
search engine (Eve2 or Turnitin.com)
Resources for Reducing Cheating Online
($7-$30/page, http://www.syllabus.com/ January, 2002,
Phillip Long, Plagiarism: IT-Enabled Tools for Deceit?)






http://www.turnitin.com/ (software, $100,
free 30 day demo/trial)
http://www.copycatch.freeserve.co.uk/
(free in UK)
http://www.canexus.com/ (Eve2 software;
essay verification engine, $19.95)
http://www.plagiarism.org/ (resource)
http://www.academicintegrity.org/
(assoc.)
http://sja.ucdavis.edu/avoid.htm (guide)
Turnitin.com
Turnitin Testimonials
"Many of my students believe that if they do not
submit their essays, I will not discover their
plagiarism. I will often type a paragraph or two
of their work in myself if I suspect plagiarism.
Every time, there was a "hit." Many students
were successful plagiarists in high school. A
service like this is needed to teach them that such
practices are no longer acceptable and certainly
not ethical!”
Check Papers with Eve2
Do Dennen & Bonk Plagiarize?
More Ways to Reduce Cheating Online
(How to Proctor from a distance, Dan Carnevale, Chronicle
of Higher Ed, Dec., 12, 1999)
8. Vary items in exam
9. Have timed exams
10. Make course too hard to cheat
11. Random selection of items for item
pool
12. Use mastery learning for some tasks
13. Assign collaborative tasks
Still More Ways to Reduce
Cheating Online
14. Use test passwords
15. Rely on computer IP# screening
16. Set expectations (e.g., scholarly
integrity, syllabus procedures)
17. Emphasize consequences (e.g.,
stories of past offenders)
18. Have students make a vow of no
cheating (e.g., University of Virginia)
Academic Honor Codes!
More Ideas:
(Promoting Academic Integrity)
(June 16, 2003, TLTL, IU Anti-Plagiarism Strats)
19. Help students recognize citation probs
 Examples, cases, Web sites, 1 credit course
20. Help Faculty and AIs recognize problems
 Examples, cases, practice grading , discuss
21. Provide practice in citing properly
 Interactive workshops, discussions
22. Design assignments to minimize dishonesty
and poor citation
 Provide frameworks, well designed prompt
23. Hold students accountable
24. Plan for special situations
25. Use available support
Plagiarism Test
(Ted Frick, Indiana University)
Still More Ways to Reduce Cheating Online
(McMurty (2001) E-Cheating: Combating a 21st Century
Challenge, Tech Horizons in Ed, 29(4), 36-41.
26. Require prewriting steps such as an
outline and rough draft
27. Require electronic paper submission
28. Design writing assignments with
high specificity, not open-ended
(harder to find a match in a pool)
29. Use Google to search for phrases
that do not appear to be from a
particular student
30. Peruse paper help (i.e., cheat) sites
Tips on
Authentification
31. Check e-mail access
against list
32. Use password access
33. Provide keycode, PIN,
or ID #
34. Futuristic Other: Palm
Print, fingerprint,
voice recognition, iris
scanning, facial
scanning, handwriting
recognition, picture
ID
The Ultimate in Plagiarism???
Fake Degrees (Oakley, 2003)
Fake Degrees
(Burks Oakley, 2003)

http://www.fakedegrees.com/
http://www.myodegrees.com/

http://www.boxfreeconcepts.com/magicmill/





http://www.blackmarket-press.com/
http://www.bogusphd.com/
http://www.coastalretreat.com/degrees/funny.html
www.counterfeitlibrary.com/cl/qual.asp
Sample Fake Degree Sites
Imagine the Assessments!
American InterContinental University (AIU)
Final advice…whatever you do…