Transcript Chapter 12 Critiquing Nursing Studies

Chapter 12
Critical Appraisal of Quantitative and
Qualitative Research for Nursing
Practice
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
1
Intellectual Research Critique

A careful examination of all aspects of a study
to judge:




Merits
Limitations
Meaning
Significance
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
2
Intellectual Critique Questions
Was the research problem significant?
What are the major strengths of the study?
What are the major weaknesses of the
study?
4. Did the researchers use sound
methodology?
5. Do the findings accurately reflect reality?
6. Are the findings consistent with those from
previous studies?
1.
2.
3.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
3
Intellectual Critique Questions (cont’d)
7.
8.
Can the study be replicated by others?
What are the implications of the findings?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
4
Intellectual Critique Guidelines
Read and critique the entire study.
Examine the organization and presentation
of the research report.
3. Examine the significance of the probe,
studied for nursing practice.
4. Identify strengths and weaknesses of a
study.
5. Be objective and realistic in identifying the
study’s strengths and weaknesses.
1.
2.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
5
Intellectual Critique Guidelines
(cont’d)
Provide specific examples of the strengths
and weaknesses.
7. Provide a rationale for your critique.
8. Suggest modifications for future studies.
9. Discuss feasibility of replication of the study.
10. Discuss usefulness of findings for practice.
6.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
6
Critique Process for Quantitative
Studies




Phase 1—Comprehension
Phase 2—Comparison
Phase 3—Analysis
Phase 4—Evaluation
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
7
Phase 1—Comprehension




Read the article carefully.
Identify terms you do not understand and
determine their meaning in a dictionary or the
glossary of Burns and Grove.
Read the article a second time.
Highlight each step of the research process.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
8
Phase 2—Comparison



Requires knowledge of what each step of the
research process should be like
The ideal is compared with the real.
Must examine the extent to which the
researcher followed the rules for an ideal
study
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
9
Phase 3—Analysis


Involves a critique of logical links connecting
one study element with another
Overall logical development of the study must
be examined.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
10
Critique Guidelines or Comparison
and Analysis



Review research text(s).
Compare the steps in the study you are
analyzing with the criteria in the research
text(s).
Analyze the logical links among the steps of
the study.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
11
Research Problem and Purpose





Is the problem sufficiently delimited in scope
without being trivial?
Is the problem significant to nursing?
Is there evidence of researcher biases?
Does the purpose narrow and clarify the aim
of the study?
Was the study feasible in terms of funding,
expertise, subjects, facility, equipment, and
ethical considerations?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
12
Literature Review




Does it demonstrate progressive
development of ideas through previous
research?
Is a theoretical knowledge base developed
for the problem and purpose?
Does the literature review provide rationale
and direction for the study?
Is a clear, concise summary presented of the
current empirical and theoretical knowledge
in the area of study?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
13
Study Framework




Is the framework presented with clarity?
If a map or model is presented, is it adequate
to explain the phenomenon of concern?
Is the framework linked to the research
purpose?
Would another framework fit more logically
with the study?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
14
Study Framework (cont’d)


Is the framework related to nursing
knowledge?
If a proposition from a theory is tested, is the
proposition clearly identified and linked to the
study hypotheses?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
15
Research Objectives, Questions,
or Hypotheses



Are the objectives, questions, or hypotheses
expressed clearly?
Are the objectives, questions, or hypotheses
logically linked to the research purpose and
framework?
Are the research objectives, questions, or
hypotheses linked to concepts and
relationships from the framework?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
16
Variables



Do the variables reflect the concepts
identified in the framework?
Are the variables clearly defined?
Is the conceptual definition of a variable
consistent with the operational definition?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
17
Design




Is the design used the most appropriate to
obtain the needed data?
Does the design provide a means to examine
all of the objectives, questions, or
hypotheses?
Have threats to design validity been
minimized?
Is the design logically linked to the sampling
method and statistical analyses?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
18
Design (cont’d)



Is the treatment clearly described?
Was a protocol developed to promote
consistent implementation of the treatment?
Did the researcher monitor the
implementation of the treatment to ensure
consistency?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
19
Sample, Population, and Setting



Is the sampling method adequate to produce
a representative sample?
What are the potential biases in the sampling
method?
Are any subjects excluded from the study
based on age, socioeconomic status, or race,
without a sound rationale?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
20
Sample, Population, and Setting
(cont’d)



Were the sampling criteria appropriate for the
type of study conducted?
Is the sample size sufficient to avoid a type II
error?
If more than one group is used, do the groups
appear equivalent?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
21
Sample, Population, and Setting
(cont’d)



Are the rights of human subjects protected?
Is the setting used in the study typical of
clinical settings?
Was sample mortality a problem? If so, how
might this influence the findings?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
22
Measurements



Do the instruments adequately measure the
study variables?
Are the instruments sufficiently sensitive to
detect small differences?
Does the instrument have adequate validity
and reliability?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
23
Scales and Questionnaires




Are the instruments clearly described?
Are techniques to complete and score the
instruments provided?
Are validity and reliability of the instruments
described?
If the instrument was developed for the study,
is the instrument development process
described?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
24
Observation



Are the phenomena to be observed clearly
identified and defined?
Is interrater and intrarater reliability
described?
Are the techniques for recording observations
described?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
25
Interviews




Do the interview questions address concerns
expressed in the research problem?
Are the interview questions relevant for the
research purpose and objectives, questions,
or hypotheses?
Does the design of the questions tend to bias
subjects’ responses?
Does the sequence of questions tend to bias
subjects’ responses?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
26
Physiological Measures


Are the physiological measures or
instruments clearly described? If appropriate,
are brand names of instruments identified?
Are the accuracy, selectivity, precision,
sensitivity, and error of the instruments
discussed?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
27
Physiological Measures (cont’d)



Are the physiological measures appropriate
for the research purpose and objectives,
questions, or hypotheses?
Are the methods for recording data from the
physiological measures clearly described?
Is the recording of data consistent?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
28
Data Collection


Is the data collection process clearly
described?
Is the training of data collectors clearly
described and adequate?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
29
Data Collection (cont’d)



Is the data collection process conducted in a
consistent manner?
Are the data collection methods ethical?
Do the collected data address the research
objectives, questions, or hypotheses?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
30
Data Analysis





Are data analysis procedures appropriate to
the type of data collected?
Are data analysis procedures clearly
described?
Are the results presented in an
understandable way?
Do data analyses address each research
objective, question, or hypothesis?
Are the analyses interpreted appropriately?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
31
Data Analysis (cont’d)



Are the statistical analyses logically linked to
the design?
Is the sample size sufficient to detect
significant differences?
Was power analysis used to determine
sample size?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
32
Interpretation of Findings





Are findings discussed in relation to each
objective, question, or hypothesis?
Are the findings clinically significant?
Do the conclusions fit the findings from the
analyses?
Are conclusions based on statistically
significant and clinically significant results?
Are there limitations the researcher did not
identify?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
33
Phase 4—Evaluation



Involves determining the meaning and
significance of the study by examining the
links among the study process, study
findings, and previous studies
Study findings are examined in light of
previous study findings.
Evaluation builds on conclusions reached
during the first three stages of the critique
and provides the basis for the fifth step—
conceptual clustering.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
34
Phase 4—Evaluation (cont’d)

The steps of the study are evaluated based
on previous studies.

Present hypotheses are based on previous
hypotheses.
 Present design is based on previous designs.
 Present methods of measurement are based on
previous measurement.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
35
Critique Guidelines for Evaluation




What rival hypotheses can be suggested for
the findings?
How much confidence can be placed in the
study findings?
To what populations can the findings be
generalized?
What questions emerge from the findings,
and are these identified by the researcher?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
36
Critique Guidelines for Evaluation
(cont’d)



What future research can be envisioned?
Could the limitations of the study have been
corrected?
When the findings are examined based on
previous studies, what is now known and not
known about the phenomenon under study?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
37
Critique Guidelines for Evaluation—
Examination of Previous Studies




Are the findings of previous studies used to
generate the research problem and purpose?
Do the findings build on findings of previous
studies?
Is the design an advance over previous
designs?
Do sampling strategies show an improvement
over previous studies?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
38
Critique Guidelines for Evaluation—
Examination of Previous Studies (cont’d)



Does the sample selection have the potential
for adding diversity to samples previously
studied?
Does the current research build on previous
measurement strategies so that
measurement is more precise or more
reflective of the variables?
How do statistical analysis techniques
compare with those used in previous studies?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
39
Critique Guidelines for Evaluation—
Examination of Previous Studies (cont’d)


Is the current knowledge in this area
identified?
Does the author indicate the implication of the
findings for practice?
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
40
Skills Needed to Critique Qualitative
Studies




Context flexibility
Inductive reasoning
Conceptualization, theoretical modeling, and
theory analysis
Transforming ideas across levels of
abstraction
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
41
Context Flexibility


Definition: the capacity to switch from one
context or worldview to another, to shift
perception so as to see things from a different
perspective
It is not necessary to become committed to a
perspective to follow or apply its logical
structure.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
42
Context Flexibility (cont’d)

All scholarly work requires a willingness and
ability to examine and evaluate works from
diverse perspectives.

For example, analysis of the internal structure of a
theory requires context flexibility.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
43
Inductive Reasoning Skills



Necessary so as to follow the logic of a
qualitative researcher
Used in the transformation process during
data analysis
Revealed in the move from concrete
descriptions to the abstract level of science
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
44
Standards for Qualitative Critique

Standard 1:
Standard 2:
Standard 3:

Standard 4:

Standard 5:


Descriptive vividness
Methodological congruence
Analytical and interpretative
precision
Philosophical or theoretical
connectedness
Heuristic relevance
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
45
Standard 1—Descriptive Vividness


Description of the site and subjects, the
experience of collecting the data, and the
thinking of the researcher during the process
need to be presented so clearly that the
reader has the sense of personally
experiencing the event.
Because one of the assumptions of
qualitative research is that all data are
context specific, the evaluator of a study must
understand the context of that study.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
46
Descriptive Vividness (cont’d)

A contextual understanding of the whole is
essential and prerequisite to the capability of
the reviewer to evaluate the study in the light
of the other four standards.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
47
Threats to Descriptive Vividness




Failure to include essential descriptive
information
Lack of clarity and/or depth of description
Inadequate skills in writing descriptive
narrative
Reluctance to reveal self in written material
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
48
Standard 2—Methodological
Congruence


Reviewer must have knowledge of philosophy
and methodological approach used by the
researcher.
Four dimensions:




Rigor in documentation
Procedural rigor
Ethical rigor
Auditability
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
49
Rigor in Documentation



The reviewer examines if the researcher
clearly and concisely presents study
elements.
The reviewer examines the study elements
for completeness and clarity.
The reviewer identifies any threats to rigor in
documentation.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
50
Qualitative Study Elements







Phenomenon
Purpose
Research question
Significance of phenomenon
Identification of assumptions
Identification of philosophy
Researcher credentials
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
51
Qualitative Study Elements (cont’d)






Context
Role of researcher
Ethical implications
Sampling and subjects
Data-gathering strategies
Data analysis strategies
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
52
Qualitative Study Elements (cont’d)






Theoretical development
Conclusions
Implications
Suggestions for further study
Suggestions for practice
Literature review
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
53
Threats to Rigor in Documentation


Failure to present all elements or steps of the
study
Failure to present all elements or steps of the
study accurately or clearly
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
54
Procedural Rigor



Definition: the rigor of the researcher in
applying selected procedures for the study
The researcher needs to make clear the
steps taken to ensure data were accurately
recorded and that the data obtained are
representative of data as a whole.
When critiquing a qualitative study, the
reviewer examines the description of the data
collection process and study findings for
threats to procedural rigor.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
55
Threats to Procedural Rigor


The researcher asked the wrong questions.
The questions need to tap the subjects’
experiences, not their theoretical knowledge
of the phenomenon.
The questions included terminology from the
theoretical orientation of the researcher.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
56
Threats to Procedural Rigor (cont’d)

The informant might have misinformed the
researcher.



Ulterior motive
Other individuals present might have inhibited the
informant.
The informant might have wanted to impress the
researcher by giving the most desirable response.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
57
Threats to Procedural Rigor (cont’d)


The informant did not observe the details
requested or was not able to recall the event
and substituted instead what he or she
supposed happened.
The researcher placed more weight on data
obtained from well-informed, articulate, highstatus individuals than on data from those
who were less articulate, obstinate, or of
lower status.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
58
Threats to Procedural Rigor (cont’d)




The presence of the researcher distorted the
event being observed.
The researcher’s involvement with the
subject-participants distorted the data.
Atypical events were interpreted as typical
The informants lacked credibility.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
59
Threats to Procedural Rigor (cont’d)




An insufficient amount of data were gathered.
An insufficient length of time was spent in
data gathering.
Approaches for gaining access to the site
and/or subjects were inappropriate.
The researcher failed to keep in-depth field
notes.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
60
Ethical Rigor




Requires recognition and discussion by
researcher of ethical implications related to
study
Consent is obtained from subjects and
documented.
Report must indicate that the researcher
ensured rights of subjects were protected.
Reviewer examines data-gathering process
and identifies potential threats to ethical rigor.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
61
Threats to Ethical Rigor



Researcher failed to inform subjects of their
rights.
Researcher failed to obtain consent from
subjects.
Researcher failed to ensure protection of
subjects’ rights.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
62
Auditability




Definition: the rigorous development of a decision
trail
To achieve this, the researcher must report all
decisions involved in transformation of data to
theoretical schema.
A second researcher, using original data and decision
trail, should be able to arrive at conclusions similar to
those of original researcher.
Reviewer examines decision trail for threats to
auditability.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
63
Threats to Auditability



Description of data collection process was
inadequate.
Researcher failed to develop or identify
decision rules for arriving at ratings or
judgments.
Researcher failed to record nature of
decisions, data on which they were based,
and reasoning that entered into decisions.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
64
Threats to Auditability (cont’d)


Evidence for conclusions was not presented.
Other researchers were unable to arrive at
similar conclusions after applying decision
rules to the data.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
65
Standard 3—Analytical and
Interpretative Preciseness

Requires researcher to make intense efforts
to identify and record decision-making
processes through which data
transformations are made
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
66
Analytical Preciseness




Premature patterning may occur before the
researcher can logically fit all of the data within the
emerging schema.
The consequence may be a poor fit between data
and theoretical schema.
It is critical to test the schema by rechecking the fit
between the schema and the original data.
When critiquing study, the reviewer examines
decision-making processes and theoretical schema
to detect threats to analytical preciseness.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
67
Threats to Analytical Preciseness





Interpretive statements do not correspond
with findings.
Categories, themes, or common elements are
not logical.
Samples are not representative of the class of
joint acts referred to by the researcher.
Set of categories, themes, or common
elements fails to set forth a whole picture.
Set of categories, themes, or common
elements is not inclusive of data that exist.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
68
Threats to Analytical Preciseness
(cont’d)




Data are inappropriately assigned to
categories, themes, or common elements.
Inclusion and exclusion criteria for categories,
themes, or common elements are not
consistently followed.
Working hypotheses or propositions are not
identified or cannot be verified by data.
Various sources of evidence fail to provide
convergence.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
69
Threats to Analytical Preciseness
(cont’d)




There is incongruence of evidence.
Subject-participants fail to validate findings.
Conclusions are not data based or do not
encompass all the data.
Data are made to appear more patterned,
regular, or congruent than they are.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
70
Standard 4—Philosophical or
Theoretical Connectedness

Requires that theoretical schema developed
from the study be:




Clearly expressed
Logically consistent
Reflective of the data
Compatible with knowledge base of nursing
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
71
Threats to Theoretical Connectedness





Findings are trivialized.
Concepts are inadequately refined.
Concepts are not validated by data.
Set of concepts lacks commonality.
Relationships between concepts are not
clearly expressed.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
72
Threats to Theoretical Connectedness
(cont’d)





Proposed relationships among concepts are
not validated by data.
Working propositions are not validated by
data.
Data are distorted during development of the
theoretical schema.
Theoretical schema fails to yield a meaningful
picture of the phenomenon studied.
A conceptual framework or map is not derived
from the data.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
73
Standard 5—Heuristic Relevance


Reflects reader’s capacity to recognize
phenomenon described in the study, its
theoretical significance, applicability to
nursing practice situations, and influence on
future research activities
Intuitive recognition: when individuals are
confronted with theoretical schema derived
from the data, it has meaning within their
personal knowledge base
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
74
Threats to Intuitive Recognition



Reader is unable to recognize phenomenon.
Description is not consistent with common
meanings.
Theoretical connectedness is lacking.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
75
Relationship to Existing Body of
Knowledge


There should be intersubjectivity with
existing theoretical knowledge in nursing and
previous research.
When critiquing, reviewer examines strength
of the link of study findings to existing
knowledge.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
76
Threats to Relationship to Existing
Body of Knowledge



Researcher failed to examine existing body of
knowledge.
Process studied was not related to nursing
and health.
There is a lack of correspondence with
existing knowledge base in nursing.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
77
Applicability to Nursing Practice,
Research, or Education


Applicability: nurses need to be able to
integrate the findings into their knowledge
base and apply them to nursing practice
situations
Findings also need to contribute to theory
development.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
78
Threats to Applicability




Findings are not significant for discipline of
nursing.
There is a failure to achieve methodological
congruence.
There is a failure to achieve analytical
precision.
There is a failure to achieve theoretical
connectedness.
Copyright © 2011 by Saunders, an imprint of Elsevier Inc.
79