Transcript Designing Scientific Experiments Dr. Gail P. Taylor MBRS-RISE Coordinator UT San Antonio 08/2006 References • CRITICAL THINKING, THE SCIENTIFIC METHOD, AND PAGE 25* OF GILBERT Dany S.

Designing Scientific
Experiments
Dr. Gail P. Taylor
MBRS-RISE Coordinator
UT San Antonio
08/2006
References
• CRITICAL THINKING, THE SCIENTIFIC METHOD, AND
PAGE 25* OF GILBERT Dany S. Adams, Department of
Biology, Smith College, Northampton, MA 01063
http://www.sdbonline.org/SDBEduca/dany_adams/critical_thinki
ng.html#blurb
• Validity: http://carbon.cudenver.edu/~lsherry/rem/validity.html
• At the Bench, A Laboratory Navigator; updated edition. Kathy
Barker, Cold Spring Harbor Press, 2005
Scientific Method
•
•
•
•
•
•
Observe phenomenon & conceive ideas
Make predictions/develop a hypothesis
Devise a test/formulate experiment
Carry out experiments
Draw conclusions from results
Reject or support hypothesis
Types of Experiments
• Science does not generally deal with
facts, but rather with evidence
• Each experiment weakens or
strengthens a hypothesis
• All evidence is not equal
• Try to discern cause and effect!
Planning Experiments I
• What ideas have you come up with?
• Why is your idea important?
• Have other people tested this idea
before?
– http://www.pubmed.org
• What type of background information is
available?
• Define question/Develop hypothesis
(and null hypothesis)
Determining Causality
• Causality can be difficult to prove
• Different experimental designs impact differently
• Correlative Evidence (weak evidence)
Found together in time or space
• Loss of Function (stronger evidence)
– Blocked a phenomenon
• Gain of Function (strongest evidence)
– Initiation of event leads to second event (additional function)
Example – Protein X may be
involved in Cellular Aggregation
• “Show it”
• Correlative evidence (time and space):
– Antibody used to detect:
• Found in particular microorganism when aggregating
(and not when free living)
• Found in area where cells are contacting one another
during aggregation
– No causality; nothing beyond inference about
function
• Clumping could cause the protein expression
• Clumping and protein expression could be induced by
same causative agent
• Could be completely coincidental
Example – Protein X may be
Necessary for Cell Aggregation
• “Block it”
• Loss of Function - What does its loss do to
clumping?
– Antibody to protein used to block it from
functioning.
– Or knock out gene
– Clumping no longer takes place
• Need controls– Clumping specifically and only being inhibited
– cells not dying
– May support “real” clumping agent to function
– Therefore it is necessary for clumping
Example – Protein X may be
Sufficient for Cellular Aggregation
• “Move it”
• Gain of Function
– In organism that does not normally clump…
• Artificially introduce required protein
• Or artificially turn it on at all times (constituitively express)
– Aggregation now takes place
• Therefore is sufficient to induce clumping
Progression to Necessary and
Sufficient
• Often you will see this progression
through Biological scientific papers
– What is it?
– Yes, it’s there
– Yes, it’s in the right place
– It’s loss produces this response
– It’s addition produces this response…
Planning Experiments II
• Consider statistical methodologies during
planning stages
• Look in prior papers for ideas about statistics.
• Statistical analysis will generally discern that
likelihood that a result occurred by chance
• Consult mentor or statistician for confirmation
– Compare 1 treatment and control: t-test
• Decide on p (Probability value) p < 0.05 or 0.01
– Compare many treatment groups: ANOVA
– Many more…
Planning Experiments III
• Variables
– Independent (manipulated)
– Dependent (outcome)
• # of samples (minimum 2, 3 better)
• # repetitions (minimum 2x)
Internal Validity
• Cause and Effect- Did the experimental treatment,
and only the experimental treatment, cause the
effect!
– Controls (Be Careful!!!)
– Prevent additional variables from sneaking into your
experiment
– Must control for:
• Selection: Anything that makes treatment and control groups
different at beginning (random assignment)
• History: What different things may happen between expt. And
control groups between initial treatment and measurement
• Maturation: Natural changes in subjects (aging)
• Instrumentation: All tests/equipment/reagents must stay
same throughout experiment
• Testing: “incoming” may “teach” the subject
• Mortality: Subjects may leave or die (contamination)
• Regression: If initial test scores were high, on average, will
naturally move towards mean
External Validity
• The extent to which the findings of the study can be applied,
reproduced, or generalized to another setting or systems.
i.e., techniques to ensure that these groups correspond to
general population
•
•
•
•
•
•
•
Unrepresentative Sample: Sample members not representative of
general population.
Clear Description of the Treatment or Protocol (replicability)
Hawthorne Effect: Subjects know that they are being studied and it
influences behavior
Novelty Effect: Particularly in humans…enjoy experiment, then
possibly don’t.
Pretest Sensitization: If the pretest is part of the treatment, it will
obviously affect the results or findings.
History and Treatment Interaction: something else happened which
influenced results, for all participants
Measurement of the Dependent Variable: Treatment and data
collection must be the same every time!
Types of Controls
• Experimental
– Standards/calibration
– Animal/Cell selection/care
– Positive controls
• See what a positive response looks like and that it can
be obtained. (positively expressing cells…)
– Negative controls
• Shows what a zero response looks like
• Treatment controls
– All groups treated identically except for indep.
Variable
– If two treatments combined, show individual
– All time points must be covered
– Multiple samples
Keeping it Simple
• Your mentor wants to look at the time course
effects of a possible cancer suppressor on
proliferation and mRNA expression in six breast
cancer cell lines. Wants to look at 0, 12, 24, 36,
48, 72, 96h
The beauty of Small
experiments….
• Mega Experiment
– Ex: 6 types of cells, 7 time points, treated
and untreated (2), in triplicate (3).
– 6x7x2x3 = 252 plates
• Plan Strategically and Break it down…
– 1 cell line, treated and untreated, duplicate,
7 time points = 28. Or postpone
duplicates.
Results from Small
Experiments
•
•
•
•
Low possibility for confusion
Reasonable workload
Reasonable use of resources
Ability to “assess as progress”
– Easy to interpret
– Can change directions on fly
• Easy to create discrete graphs
Chasing the Big Problem
• For a Publication…
– Need a Big Picture of what you are
pursuing; tell a good story
– Start with correlation
– Get additional information
– Knock it out/Add it back/Overexpress
• Slight modifications, depending on field
How to do Experiment – Obtain
Protocol
• Instructions for carrying out a particular
technique
• If followed, will produce desired results
• Best if it’s a proven protocol
– Designing your own is time-consuming
– Obtain from another investigator
• In lab, best
– A book of protocols, from web, from kit
• Will need fine-tuning for your local circumstances
– Methods section from published papers (least
reliable)
Review Protocol
• Read and do dry run-through
• May find logic gaps
• May find references to “common”
procedures you do not know
Personalize Protocol
• Rewrite (keeping same steps, etc) to
make more sense to you.
• Add notes about own equipment
required
Fully Prepare before Experiment
• Buy all required materials
– Radioisotopes
• Make all solutions and buffers
• Reserve machine time if needed
Follow Protocol exactly, first time
through
• If it doesn’t work, you can assume it’s
you.
• Do again. Not work?
• Can get help from person who provided
Modify Protocol
•
•
•
•
Once protocol is working, modify.
Make notations of changes
Rewrite for next run though
Good if type into computer…can record
changes and re-print.
During Experiment
• Record which media, temperature or
date-sensitive agents you used
• Record any procedural deviation
– Dropped something
– Delayed
– Calibration questions
• Put in lab notebook in a timely fashion
Interpreting Results I
• Did the Experiment work?
– Examine procedural (markers, cells lived)
– Examine positive control (yes, antibody
working)
– Examine negative control (No, did not have
everything come up positive)
Interpreting Results II
• What were the results?
– Compared to controls, did you see effect?
– Graph your data
– How big was effect?
– Did effect vary over time?
Interpreting Results III
• What does the experiment mean?
– Do the results make sense?
– Was the result what you expected?
– Can you explain spurious results?
– What additional controls may you need?
Interpreting Results IV
• What do other investigators think?
– Talk to lab members
– Discuss results with someone versed in
technique
– Run through background papers again
– Repeat results
Interpreting Results V
• Are the results repeatable?
– Do experiment again
– Add any additional controls
Agh! It didn’t work!
1. Check notes.
2. Redo the experiment
3. Focus on individual parts of expt.
1. + and – controls…
4. Do partial expt. to insure it’s fixed
5. When you’ve done all…try again
several times
6. If external protocol, may want to
switch
Switching Projects
•
•
•
•
Never can reproduce data
Project has little support from PI
Direction of project has changed
Not technically possible to do
experiments well
• Project too difficult or involved