Internet Research: Issues, Problems and an Example Technique
Download
Report
Transcript Internet Research: Issues, Problems and an Example Technique
Web-Based Research:
Issues, Problems and an
Example Technique
John H. Krantz
Hanover College
Outline
Brief History
Issues in Internet Research
An Example Method
Problems and Solutions
History
Email for 2-3 Decades
1995
Krantz, Ballard, & Scher (1997) Within Subjects
Reips (1997) Between Subjects
First True Web Experiments
July 1995 List of Online Psychological
Research – sponsored by APS for awhile
http://psych.hanover.edu/research/exponnet.html
Almost 200 studies and study sites listed
Why Do Internet Research
Musch & Reips (2000)
Sample Size
Statistical Power
High Speed
Ability to reach participants in other countries
High external or ecological validity
Low cost
Ability to replicate a lab experiment with more
power
Special populations
Issues
Is the Sample Representative
Yes and No
Important to remember base of comparison
Ideal vs. Lab
Krantz & Dalal (2000)
Much more diverse that most lab samples
Especially in age and education range
Race is generally limited as is nationality
Sample Characteristics:
Gender
Are Internet Samples Male Dominated?
Krantz & Dalal (2000)
GVU 1st (1994)
Reips (1996) English:
German:
Krantz, et al. (1997)
Pasveer & Ellard (1998) 3rd Study
% Female
5%
43%
18%
44%
71%
More Recent
Caddell & Utt (2004)
Meyerson & Tryon (2003)
77%
45%
Sample Characteristics: Age
Are we still testing college sophomores?
Krantz, et al. (1997)
43% > 30
Smith & Leigh (1997)
35% > 30
Pasveer & Ellard (1998)
45% > 25
GVU 1st
36% > 30
Caddell & Utt (2004)
60% > 30
Pattison & Rouse(2003)
16% > 30
Is the College Sophomore be
Making a Comeback
Plot of activity of Psychological Research on the Net
50000
# Hits
40000
30000
2003
20000
10000
Monlth
Pattison & Rouse (2004) 76% 18-22
ec
D
ct
N
ov
O
ar
A
pr
M
ay
Ju
n
Ju
ly
A
ug
Se
p
M
Fe
b
Ja
n
0
Sample Characteristics: Race
How diverse are the samples?
Unfortunately in general samples, diversity is still
limited
Krantz et al. (1997)
89% White
Smith & Leigh (1997)
86% White
GVU 10th (1998b)
87% White
O’Neil, Penrod, & Bornstein (2003) 82% White
Meyerson & Tryon (2003)
93% White
However, as will be discussed later, web can make
it possible to access special populations
Sample Characteristics:
Nationality
Where to do the subjects come from?
Largely North American, even US, even in
some European studies
Krantz, et al. (1997)
86% N. Am.
Senior, et al (1999)
>80% N. Am
This study was conducted in England.
Is the Data Any Good
Emphatically Yes
Data Quality: Direct
Comparisons
A number of studies, still, run both laboratory
and internet samples.
Krantz, et al. (1997)
Regression of web means on laboratory
samples (even though different types of
samples):
lab mean = 1*(Web mean) + 0
r2 = .99
Compare to Established Data
Compare web results to previously published
data sets
Myerson & Tryon (2003)
Studied Sexual Boredom Scale of Watt &
Ewing (1996)
Matched sample characteristics
Found same internal consistency
Form of administration was not a significant
factor
Direct Validity Comparison
Use same techniques to validate results
Pasveer & Ellard (1998)
Developed new scale
Internal consistency
Psychomectric properties
Data Validity: Arguments
Reips (2000)
Statistical Power
Limited Sample Population
Limited External Validity
Less than Optimal Voluntariness
Motivational Confounding
Experimenter Bias
Nontransparency
Limitations of what is feasible to research
Interesting Deviations
Stern & Faber (1997)
Milgram’s lost letter technique
Milgram, sent on
Stern & Faber, returned to sender
From doing favor to doing easy task
Often effects are smaller
Getting Subjects: General
http://psych.hanover.edu/research/exponnet.
html
Other Pages:
Social Psychology:
http://www.socialpsychology.org/expts.htm
The Web Experiment List: http://genpsylabwexlist.unizh.ch/
Getting Subjects: Special
Populations
Advertising
Email Groups
Careful, get permission
Can be thought spam
Netiquette
Discussion Groups
Same, permission
Sample Method for Doing
Survey
Birnbaum, M. H. (2001). Introduction to
Behavioral Research on the Internet. Upper
Saddle River, NJ: Prentice Hall.
Lots of resources at http://ati.fullerton.edu/
NSF sponsored classes at Fullerton on Social
Psychology
One more next January, look for the
announcement at this site
First Step: Where to put it
Setting up a server can be easy
Apache web server
http://www.apache.org
Most popular server
Freeware
Any computer connected to web is now a
server
May have issues of firewalls & dynamic IP
addresses
Second Step: SurveyWiz
In Birnbaum (2001)
Also linked to from http://ati.fullerton.edu/
Under Birnbaum’s page
Freely available
We will build a very short survey to illustrate
Forms
SurveyWiz uses web forms to collect data
What are forms:
Elements to allow web page viewers to input
This data is then sent back to the server to be
processed
SurveyWiz
What the page looks like My Take
Can edit it with any web page editor
Examples I have done this
year
Caddell & Utt
Pattison & Reese
Third Step: Collecting the data
First way, let Birnbaum do it for you
SurveyWiz is set up this way
<FORM action=http://psych.fullerton.edu/cgiwin/polyform.exe/generic method=post>
FTP://guest:[email protected]
You can type in the address box of your browser.
Note that this is FTP site. At the present time,
password is guest99, as shown above, and
permits download only.
Keeping the Data Local
Second, get this perl script (found at
http://psych.fullerton.edu/mbirnbaum/program
s/PERL_script2.htm)
The script and instructions are at this site
Written by Billy Schmidt
Also change code in form command
At my school the line reads:
<FORM action=http://psych.hanover.edu/cgibin/survey_wiz.pl method=post>
What is CGI
Common Gateway Interface
Method for other programs to interact with
web servers
In this case, this perl program takes data
from web forms and stores them in a file
Stores data in Excel or SPSS readable files
Fourth Step: Example Data File
Data File
Data coding scheme: all variable names
begin two numbers, see example
The perl program sorts the variables by
numerical order
Another Method of Survey
Generation
Schmit’s Survey Assistant
He manages data
More flexibility
http://or.psychology.dal.ca/~wcs/hidden/home.htm
l
What Do I Need?
At NSF ATI site
List of software resources
What Problems are important
Musch & Reips (2000) – 5 point scale
No control over participant’s behavior
No control over motivation
Inability of participants to ask questions
Nonrepresentative sample
Manipulation and fraud
Ethical problems
3.6
3.4
3.3
2.9
2.4
1.5
Problems
Eliminating Multiple Entries
Dropout
Security
Data Integrity
Multiple Entries
People submit too fast, while waiting for
feedback
People will fake being two people
Usually have same IP address
SurveyWiz and most other methods, sends
the IP address of machine where survey is
being taken
Can eliminate more than one from same IP,
e.g. Schmidt (1997)
Security & Data Integrity
If on public server, others can access data
and download
Others might fake pages to send data
(unlikely)
Keep data in non-public directories so only
researchers have access
Have CGI check for origin of survey to make
sure it is yours (Schmidt, 1997)
Dropout
People will come but not finish
Or data is incomplete
Survey will sends a complete signal to help
you track
Can use: (O’Neil, Penrod, & Bornstein, 2003)
Short sweet
Financial incentives.
If multiple pages to survey, several pages warmup so not drop-out during
Ethical Issues
Should not think of Ethical Issues of Web
Research Alone
Compare to Ethics of doing Traditional
Research
Not let our comfort with what we have done
blind us to those problems
Conclusion
Many Benefits
Easy Methods
Easy Resources
Not alternative to traditional methods
A new tool still needs to be used thoughtfully