Transcript Data Collection Techniques

NECC 2007 - MP356
Creating Web-Based Online
Surveys
Jeff Sun
[email protected]
 www.sun-associates.com/necc2007
You?
Who’s working on projects now?
Existing survey work?
2
Objectives
To understand the role of data collection in
an overall evaluation process
To understand the role of surveys as one
tool in an evaluation data collection
process
To become initially familiar with tools for
creating online surveys
Others?
3
Why Evaluate?
To fulfill program requirements
Project sustainability
Generation of new and improved project
ideas
4
By Definition, Evaluation…
 Is both formative and summative
 Helps clarify project goals, processes, products
 Should be tied to indicators of success written for your
project’s goals
 Is not a “test” or simply a checklist of completed
activities
 Qualitatively, are you achieving your goals?
 What adjustments can be made to your project to
realize greater success?
5
Data Collection vs. Evaluation
Evaluation is more than data collection
Evaluation is about…
Creating questions
Creating indicators
Collecting data
Analyzing and using data
Data collection occurs within the context
of a broader evaluation effort
6
An Iterative Process
Evaluation breaks your vision down into
increasingly observable and measurable
pieces.
7
Tools and Techniques
Surveys, Focus Groups and Interviews
Measuring satisfaction and determining attitudes
Observations
Artifact Analysis
Triangulation with data from administrators and
staff
Do other groups confirm that teachers are being
served?
8
Surveys
Online vs. Paper
Is there sufficient connectivity?
Often works best if people complete the
instruments all at the same time
Same goes for paper surveys
Online surveys provide immediate data
Spreadsheets which can be exported to a
variety of different programs for analysis
9
Surveys
Online
VIVED
Profiler
LoTi
Zoomerang
SurveyMonkey.com
10
Make Your Own!
www.sun-associates.com/surveyws/surveys.html
 www.sun-associates.com/necc2007/monday/y2survey.html
Based on a CGI script on your web server
Outputs to a text file, readable by Excel
Works with yes/no, choose from a list, and free text
input (no branching)
11
The Survey Script
 www.bignosebird.com/carchive/bnbform.shtml
You need access to your WWW server’s directory
structure
If you do not have this, you should use some of the
commercial survey services discussed earlier
You need FTP access to your WWW server
The bnbform.cgi script goes in your /cgi directory
The readme.txt file contains all the installation info
necessary
12
Once Installed
Surveys are HTML forms that make use of
the POST command
<FORM METHOD="POST" ACTION="/cgibin/bnbform.cgi" ENCTYPE="x-www-formurlencoded">
Let’s examine an example…
 www.sun-associates.com/necc2007/monday/y2survey.html
13
For Today’s Exercise
Create a folder called <yourinit>testsite
on your desktop
In that folder, create a folder called www
In the /www folder, create a folder called
<yourinit>test
14
 www.sun-associates.com/necc2007/monday/y2survey.html
Save this as “sample.html” as Page Source
Place that file in your <yourinit>test folder
15
Start Dreamweaver
/Site/New Site
Local Root Folder = <yourinit>testsite
Http address = www.sun-associates.com
Remote Info
Access = FTP
Host = ftp.business.earthlink.net
Host Dir = leave blank
Login = wfp44289%sun-associates.com
Pswd = neccmp356
16
Creating Forms
Dreamweaver (or another HTML authoring
tool) is recommended
It’s best to start with a completed survey
form and then modify
That way, you get all of the correct <form>
tags and hidden files
You can use the previous example as your
template (just Save As… to save the html)
Let’s do that now
17
Things to Do
 (in “Design” view) Delete text between first line on the
page and the text just before the SUBMIT button
 Create new form elements in the resulting space
Pull Down List
Text Area
Radio Button Pair (yes/no)
 Modify the hidden elements (in “Code” view) at the end
of the source code
<P><!-- SCRIPT CONFIGURATION SECTION -->
<INPUT TYPE="hidden"NAME="data_order" VALUE="district,other_dist,…8i">
<INPUT TYPE="hidden" NAME="outputfile" VALUE="neccwsy2.log">
<INPUT TYPE="hidden" NAME="form_id" VALUE="SAMPLE Y2 Expectations Survey">
<INPUT TYPE="hidden" NAME="ok_url" VALUE="http://www.sun-associates.com/thanks.html">
<INPUT TYPE="hidden" NAME="not_ok_url" VALUE="http://www.sun-associates.com/sorry.html">
<!-- END OF SCRIPT CONFIGURATION SECTION -->
18
Survey Considerations
There actually is a science involved in
survey creation
Issues of validity both in questions and
instruments
Sample size
Attaining a sufficiently large sample to test
validity
Piloting
19
Survey Tips
Keep them short (under 10 minutes)
Avoid huge long checklists
Allow for text comments
Support anonymity
But allow for categorical identifications -school, job function, grade, etc.
20
Coordinate and support survey
administration
Avoid the “mailbox stuffer”
Work with building leaders
Provide clear deadlines
21
Three Big Points
Surveys alone mean nothing
TRIANGULATE!
100% response rate is virtually impossible
On the other hand, nearly 100% is very
possible if you follow our tips!
Share the data
No one wants to fill in forms for no purpose
22
Data Dissemination
Compile the report
Determine how to share the report
School committee presentation
Press releases
Community meetings
23
10 Tips for Data Collection
Ground your data collection within a larger
evaluation framework
Know why you’re asking what you’re asking
Not all data is quantitative
Cast a wide net
It’s all about stakeholders
Dig deep
Try to collect the data that can’t easily be observed or
counted
24
Use confirming sources
Triangulate! Surveys alone do nothing.
Have multiple writers
Stakeholders and different perspectives
Think before you collect
Choose questions carefully and with regard
to what you really expect to find
25
Set (reasonable) expectations for
participation
Time and effort
Forget about mailbox surveys
Usually waste more time than their value
Report back
Don’t be a data collection black hole!
26
More Information
[email protected]
978-251-1600 ext. 204
www.sun-associates.com/evaluation
www.sun-associates.com/necc2007
www.edtechevaluation.com
27