Transcript Research Methods - De Montfort University

Research Methods
Howell Istance
Department of Computer Science
De Montfort University
© De Montfort University, 2001
1
This session….
•
•
•
•
•
•
Why do you need this….
Online resources
Literature reviews
The nature of research
Questionnaire design
Structured investigations
© De Montfort University, 2001
2
Why do you need this…?
• To understand content of directed reading within taught
modules
• To be able to write essays or reviews of published
literature
• To be able to plan and conduct your project where there is
an element of research in the investigation or the
evaluation of what you have done
© De Montfort University, 2001
3
Library student study skills books…
• http://www.library.dmu.ac.uk/Skills/Study/
• Collection of self study books relevant to research and
projects
• Most available in .pdf or .html format (free download)
• Series includes…
–
–
–
–
–
–
–
Research Methods
Designing a questionnaire
Information Citation and Control
Dissertation preparation and presentation
Research interviews
Planning a sample survey
Thinking it through: an introduction to critical thinking
© De Montfort University, 2001
4
MSc projects
• All projects are required to have at least one academic
objective, usually an investigation of literature relevant to
some aspect of your project
• Most projects will include some form of evaluation
– E.g the production of a computer-based tool to teach letters of the
alphabet to nursery level children
– A piece of software which purports to do this will be of limited
value without some evidence of how effective it is in doing so.
– Demonstrating its effectiveness requires a structured study
© De Montfort University, 2001
5
Project Skills…
• There is a non-assessed module in the second semester,
which covers
–
–
–
–
–
–
Personal time management
Project management
Project Selection
Written and verbal communication
Giving demonstrations
Learning Strategies
© De Montfort University, 2001
6
Literature reviews
• Conducted to ensure a researcher is familiar with ‘all’ of
the what is known about a particular field
• Often published in order to bring other researchers (and
MSc students) up to speed quickly in an unfamiliar field
• Need to have the scope of the review carefully defined
– Not too big such that adequate coverage is infeasible, and there is
too much literature to review and the review becomes unfocussed
(and thereby not useful)
– Not too narrow such that there are too few papers to include
© De Montfort University, 2001
7
Form of a literature review…
• Requires reading literature from a variety of sources
• Forming some form of taxonomy or structure for your
review
• Identifying where in your taxonomy the various
contributions from the literature fall
• Critically reviewing the literature
– Identifying different approaches, contradictions between
contributions, analysis of strengths and weaknesses
– Not simply pasting quotes from different papers
• Drawing your own conclusions, particularly concerning
completeness of coverage
• Highlight implications for your work (if appropriate)
© De Montfort University, 2001
8
Example of a literature review
Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics
Forum, 16(5): 269-281. (Dec 1997)
(Abstract)
Recent gains in the performance of 3D graphics hardware and rendering
systems have not been matched by a corresponding improvement
in our knowledge of how to interact with the virtual environments we
create; therefore there is a need to examine these further if we are to
improve the overall quality of our interactive 3D systems. This paper
examines some of the interaction techniques which have been
developed for object manipulation, navigation and application control
in 3D virtual environments. The use of both mouse-based techniques
and 3D input devices is considered, along with the role of feedback
and some aspects of tools and widgets.
© De Montfort University, 2001
9
Example of a literature review
Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics
Forum, 16(5): 269-281. (Dec 1997)
(Abstract)
Recent gains in the performance of 3D graphics hardware and rendering
systems have not been matched by a corresponding improvement
in our knowledge of how to interact with the virtual environments we
create; therefore there is a need to examine these further if we are to
improve the overall quality of our interactive 3D systems. This paper
examines some of the interaction techniques which have been
developed for object manipulation, navigation and application control
in 3D virtual environments. The use of both mouse-based techniques
and 3D input devices is considered, along with the role of feedback
and some aspects of tools and widgets.
rationale
© De Montfort University, 2001
10
Example of a literature review
Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics
Forum, 16(5): 269-281. (Dec 1997)
(Abstract)
Recent gains in the performance of 3D graphics hardware and rendering
systems have not been matched by a corresponding improvement in
our knowledge of how to interact with the virtual environments we
create; therefore there is a need to examine these further if we are to
improve the overall quality of our interactive 3D systems. This paper
examines some of the interaction techniques which have been
developed for object manipulation, navigation and application control
in 3D virtual environments. The use of both mouse-based techniques
and 3D input devices is considered, along with the role of feedback
and some aspects of tools and widgets.
Scope of review
© De Montfort University, 2001
11
Example of a literature review
Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics
Forum, 16(5): 269-281. (Dec 1997)
(Abstract)
Recent gains in the performance of 3D graphics hardware and rendering
systems have not been matched by a corresponding improvement in
our knowledge of how to interact with the virtual environments we
create; therefore there is a need to examine these further if we are to
improve the overall quality of our interactive 3D systems. This paper
examines some of the interaction techniques which have been
developed for object manipulation, navigation and application control
in 3D virtual environments. The use of both mouse-based techniques
and 3D input devices is considered, along with the role of feedback
and some aspects of tools and widgets.
Indication of taxonomy
© De Montfort University, 2001
12
Sources of information
•
•
•
•
Scientific journals
Conference proceedings (refereed and unrefereed)
Magazines, newspapers
WWW
• Important to differentiate between reported investigations
and reported opinion
• Conclusions of any investigation contain some element of
informed opinion or judgement
• Sources of information must be cited appropriately
© De Montfort University, 2001
13
Project Academic Objectives
• For example.. ‘Review methods of usability evaluation
appropriate to the evaluation of internet-based groupware
systems for use by students undertaking a course by
distance learning’
– Requires review of methods described in the literature, using the
original sources (not just someone else’s review)
– Implications of problems/needs associated with evaluation of
groupware systems
– Comparisons and contrasts between methods in light of identified
needs of groupware applications
– Your conclusions about which method(s) to use and how
© De Montfort University, 2001
14
How to do it…
• Collect and read current papers and reviews of a field
• From the list of references cited in these, get copies of
those which appear relevant
• Start to classify the papers you collect in one or several
ways to form the basis of a taxonomy
• Identify authors who are prominent and check what else
they have published (WWW useful here)
• Read contents of recent relevant journals in library
• Subscribe to mailing lists for coming conferences and look
at contributors
• search on-line bibliographic archives (e.g bids.ac.uk)
© De Montfort University, 2001
15
Constructively criticising research..
• Researchers have a vested interest in making a piece of
work appear significant and worthy of publication
• They may not be as forthcoming about the limitations of
the work as about the benefits
• Results should be reported in sufficient detail to enable the
reader to draw his/her own conclusions and thereby judge
the validity of the conclusions drawn by the author
• Are the conclusions drawn justified by the evidence
provided?
• Is the method of investigation appropriate or flawed?
• Are the investigators aware of other similar contributions
in the field?
© De Montfort University, 2001
16
What is research?
• A systematic enquiry, which is reported in a form that
allows the research methods and outcomes to be
accessible to others
• Concerned with seeking solutions to problems or answers
to meaningful questions
• Meaningful questions are expressed in a way that indicates
what you will accept as an answer
• Non-meaningful (in research terms) questions are not
answerable as a result of enquiry alone (eg judgemental or
metaphysical questions)
• Positivism versus phenomalism
© De Montfort University, 2001
17
Nature of research - positivist
• Deals with positive facts and observable phenomena
• Subscribes to the ‘scientific method’
• Primary goal is not only description but prediction and
explanation
• Classification of substances and events, and observation of
these, provide the basis for descriptive laws based on
consistencies in patterns and properties
• Characterised by absolute or varying degree of
generalisability
• Quantitative, as it draws on measurable evidence
© De Montfort University, 2001
18
Postulates in Positivist Research
• postulate of natural kinds: all instances of classes and
categories of phenomena exhibit the same properties
• postulate of constancy: all phenomena remain the same or
change only very slowly over time
• postulate of determinism: there is orderliness and
regularity in nature, constancy in terms of cause and effect
© De Montfort University, 2001
19
Nature of research - phenomenalist
• Considers that each phenomena is unique and is controlled
by variables such as time, location and culture
• No two situations are identical
• No reliance on postulates of natural kinds, constancy or
determinism
• Essentially subjective, where the content of research and
the way it is pursued is indicative of researchers intention
• Outcomes are descriptions which are expressed in
narrative and mainly in qualitative terms
© De Montfort University, 2001
20
Stockholm archipelago
© De Montfort University, 2001
21
Example of both approaches…
Can the study of critical incidents (as opposed to accidents) in
marine navigation in the Stockholm archipelago provide
the basis for improvements in sea safety in the area?
• Positivist approach: collect data via interview, classify
types of incidents, produce analyses, make
recommendations based on analysis
• Phenominalist approach: analyse interviews in depth, seek
to draw conclusions about causal factors
© De Montfort University, 2001
22
Positivist research methods include...
Descriptive research
• Anything that is variable, varies to a defined degree, and
thus can be measured
• Surveys, case studies, causal comparative studies,
correlational studies, developmental studies, trend studies
Experimental research
• Deliberate manipulation of certain factors under highly
controlled conditions
• Purpose is to identify causal connections through keeping
the levels of some variables constant and manipulating
others
© De Montfort University, 2001
23
Questionnaire design
• Relevant to requirements gathering for, and evaluation of,
projects
• Self study pack by Arthur Rothwell covers:
–
–
–
–
Planning and logistics of questionnaire design
Layout issues
Forms of questions
Contents of questions
© De Montfort University, 2001
24
Planning and logistics of questionnaire
design
•
•
•
•
•
•
•
•
•
Quantitative or qualitative?
Legal requirements: the Data Protection Act
Confidentiality and anonymity
Sample size
Volunteer respondents
Identifying subject areas
Determining appropriate length
Typical time scale
Main components of questionnaires
© De Montfort University, 2001
25
Forms of question
•
•
•
•
•
Open and closed questions
Dichotomous items
Scaled items
Mid-point or no mid-point?
Ranking to show preference
© De Montfort University, 2001
26
Content of items
• Avoiding response set
• Components of attitudes
• Common types of faulty items
–
–
–
–
–
–
leading questions
context effects
double barelled questions
vague and ambiguous terminology
hidden assumptions
social desirability
© De Montfort University, 2001
27
Leading questions and context effects
Would you agree that the governments policies on health are
unfair?
• Item wordings should not contain value judgements
How many pints of beer did you drink last night?
• Think how the context of the study would affect the
response, say in a
– survey of young peoples life styles
– survey of health behaviour and heart disease
© De Montfort University, 2001
28
Double barreled questions
Do you believe the training programme was a good one and
effective in teaching you new skills?
• avoid questions that involve multiple premises
© De Montfort University, 2001
29
Vague and ambiguous terminology
How often do you clean your teeth?
–
–
–
–
Frequently
often
infrequently
never
• what does ‘frequently’ mean?
• Give quantifiers to ensure all respondents understand the
same thing by the response categories
© De Montfort University, 2001
30
Hidden assumptions, social desirability
When did you last borrow a video tape?
• Avoid hidden assumptions - what are these?
Do you ever give to charity?
• May lead to a positive response as otherwise something
negative about the respondent is being conveyed
© De Montfort University, 2001
31
Structured Investigations - 1
• An MSc project has produced a computer-based tool to
teach letters of the alphabet to nursery level children
• Talk to the person next to you and discuss how you would
demonstrate the effectiveness of the tool through some
form of structured investigation
• Consider the resources required and the feasibility of
conducting the study within a 3-month project
© De Montfort University, 2001
32
Ways of approaching this….
• You need access to people with expertise and, hopefully, a
group of nursery children
• The product needs to be tested, and has to be ready in time
for this to happen
• Subjective assessment by teachers comparing the tool with
– Other computer based tools
– Paper-based methods
• Objective approaches
– Before and after letter recognition tests with two matched groups
of children
– Use of non-parametric statistical techniques
© De Montfort University, 2001
33
Structured Investigations - 2
• A project has produced a web-based presentation of a
college offering educational courses both as a 2D
collection of pictures of the building, and as a 3D virtual
building which you can ‘walk through’
• How would you compare the relative efficiencies of each
approach?
• Again, discuss with the person next to you
– What you are trying to assess
– How to carry out the assessment
© De Montfort University, 2001
34
Ways of approaching this….
• What to assess?
– How much information a visitor to the site obtains with each type
of presentation
– Usability issues – e.g ease of navigation through the
representations of the college building
• How to assess this?
– Unstructured user trials, where the amount of building explored is
recorded
– Structured user trials, where a task is given to find pieces of
information
– Subjective assessment of visit by participant
– Observed usability problems
© De Montfort University, 2001
35