Online Survey and Assessment Techniques Dr. Curtis J. Bonk Associate Professor, Indiana University President, CourseShare.com http://php.indiana.edu/~cjbonk, [email protected].
Download ReportTranscript Online Survey and Assessment Techniques Dr. Curtis J. Bonk Associate Professor, Indiana University President, CourseShare.com http://php.indiana.edu/~cjbonk, [email protected].
Online Survey and Assessment Techniques Dr. Curtis J. Bonk Associate Professor, Indiana University President, CourseShare.com http://php.indiana.edu/~cjbonk, [email protected] Pause… What tools do you use or know about? Collecting Evaluation Data & Online Evaluation Tools Types of Data Collection • Direct Observation in Work Setting – By supervisor, co-workers, subordinates, clients • Collect Data By Surveys, Interviews, Focus Groups – Supervisors, Co-workers, Subordinates, Clients • • • • Self-Report by learners or teams Email and Chat Online Tests E-Portfolios or Learning Logs I. Online Testing Tools (see: http://www.indiana.edu/~best/) Choice: Select companies that specialize in online assessment. Or: Use what the courseware package gives ya… Some things in life are free…! Test Selection Criteria (Hezel, 1999) • • • • • • • Easy to Configure Items and Test Handle Symbols Scheduling of Feedback (immediate?) Provides Clear Input of Exam Dates Easy to Pick Items for Randomizing Randomize Answers Within a Question Weighting of Answer Options More Test Selection Criteria • Recording of Multiple Submissions • Timed Tests • Comprehensive Statistics • Summarize in Portfolio and/or Gradebook • Confirmation of Test Submission More Test Selection Criteria (Perry & Colon, 2001) • Supports multiple items types—multiple choice, true-false, essay, keyword • Can easily modify or delete items • Incorporate graphic or audio elements? • Control over number of times students can submit an activity or test • Provides feedback for each response More Test Selection Criteria (Perry & Colon, 2001) • Flexible scoring—score first, last, or average submission • Flexible reporting—by individual or by item and cross tabulations. • Outputs data for further analysis • Provides item analysis statistics (e.g., Test Item Frequency Distributions). Web Resource: http://www.indiana.edu/~best/ II. Electronic Gradebooks (Vockell & Fiore, 1993) • Calculate scores, store info • Weight scores • Flag students with certain characteristics • Print reports by individual or group • Provide prompt feedback • But inflexible, impersonal, & can be incorrect III. Online Survey Tools for Assessment Perhaps you have heard of my survey techniques? Survey Tool Features • • • • • • • • Maintain email lists and email invitations Conduct polls Adaptive branching and cross tabulations Modifiable templates Maintain library of past surveys Publish reports Technical support, chat advice Different types of accounts—hosted, corporate, professional, etc. Still More Online Survey Features • Online help, FAQs, demos, tutorials, white papers, glossaries, site maps, tips, overviews • Website evaluation tools or options • Supports different types of items (Likert, multiple choice, forced ranking, paired comparisons, etc.) • Customizable audience lists • Launch and close surveys More Online Survey Features • Online viewing of assessment results and question analysis (including various graphical displays) • Easy to author surveys • Instant feedback to participants • Custom reporting, editable images and background, personalized greetings • Storage of results in Access or SQL Still More Features • • • • • • • Ability to start and stop a survey Reports on subgroups Randomize q’s to reduce bias Easy email reminders Personalized banner or logo Expanded question options Ability to copy surveys Web-Based Survey Advantages • Faster collection of data • Standardized collection format • Computer controlled branching and skip sections • Graphics can reduce fatigue • Easy to answer clicking • Wider distribution of respondents Web-Based Survey Problems: Why Lower Response Rates? • • • • • • • Low response rate Lack of time Unclear instructions Too lengthy Too many steps Can’t find URL Perceived as aggressive Web-Based Survey Solutions: Some Tips… • • • • • • • • Send second request Make URL link prominent Offer incentives near top of request Shorten survey, make attractive, easy to read Credible sponsorship—e.g., university Disclose purpose, use, and privacy E-mail cover letters Prenotify of intent to survey Tips on Authentification • • • • Check e-mail access against list Use password access Provide keycode, PIN, or ID # (Futuristic Other: Palm Print, fingerprint, voice recognition, iris scanning, facial scanning, handwriting recognition, picture ID) Sample Survey Tools • • • Zoomerang (http://www.zoomerang.com) IOTA Solutions (http://www.iotasolutions.com) QuestionMark • (http://www.questionmark.com/home.ht ml) Survey Solutions from Perseus • Infopoll (http://www.infopoll.com) (http://www.perseusdevelopment.com/fromsurv. htm) Sample Survey Tools • Active Feedback • (http://www.activefeedback.com/af) • • • • Majon International (http://www.coolsurveys.com) EZSurvey from Raosoft (http://www.raosoft.com/) • SurveyShare (http://SurveyShare.com; from Courseshare.com) Zoomerang Steps 1. Register: Sign Up for Basic Membership (SKIP FOR NOW!) 2. Click on new survey. 3. Click on survey template (e.g., “Education” and Professor) 4. Delete, modify, or add 1-2 items. 5. Submit survey. Zoomerang Steps 6. 7. 8. 9. Modify Web greeting and submit. Pick option #1 for submission. Type Survey invitation & email message Enter email addresses for potential survey respondents 10. Decide on how share results Zoomerang Steps 11. Launch survey 12. Click on “My Surveys” and browse 13. Click on “Email List” and browse 14. Click on “Account Info” 15. Other explorations… SurveyShare Steps 1. Register: Sign Up (perhaps take Quick Tour, Flash version) 2. Login 3. Click on Create Survey 4. Check out existing templates 5. Select one of the templates 6. Add one item, delete one item, modify one item SurveyShare Steps 7. Edit Survey Introduction and Title 8. Edit Closing and Submit 9. Select Survey Invitation Via Email 10. Type emails to complete the survey 11. Click add (to move emails to right) 12. Give email subject and message SurveyShare Steps 13. Enter email subject line and text 14. Preview survey 15. Activate survey 16. Go to MySurveys 17. Click on Survey Info 18. Go to mailing lists SurveyShare Steps 19. Add a person 20. Add list 21. Check out resources (books) 22. Check out help, quick tour, and sitemap 23. Perhaps create a survey from scratch 24. Perhaps create a survey from “My Previous Surveys” Any Reactions to Online Surveys??? Fitting Surveys into Evaluations Plans Bonk’s Evaluation Plan… Considerations in Evaluation Plan 8. University or Organization 7. Program 6. Course 5. Tech Tool 1. Student 2. Instructor 3. Training 4. Task What to Evaluate? 1. Learner—attitudes, learning, use, performance. 2. Instructor—popularity, course enrollments. 3. Training—internal and external components. 4. Task--relevance, interactivity, collaborative. 5. Tool--usable, learner-centered, friendly, supportive. 6. Course—interactivity, participation, completion. 7. Program—growth, long-range plans. 8. Organization—cost-benefit, policies, vision. The 4 Levels • Reaction • Learning • Behavior • Results Percent of Respondents Figure 26. How Respondent Organizations Measure Success of Web-Based Learning 90 80 70 60 50 40 30 20 10 0 Learner satisfaction Change in knowledge, skill, atttitude Job performance Kirkpatrick's Evaluation Level ROI Ziegler, April 2002, eLearning “…the key is not to measure every possible angle, but rather to focus on metrics that are pragmatic and relevant to both human and business performance at the same time.” Some Final Advice… Or Maybe Some Questions???