Statistical Considerations for Studies Involving Mice (et al.)

E L I Z A B E T H G A R R E T T - M A Y E R P R O F E S S O R O F B I O S T A T I S T I C S D I R E C T O R , B I O S T A T I S T I C S S H A R E D R E S O U R C E H O L L I N G S C A N C E R C E N T E R

What started this? ARRIVE

  

Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research*

The gist:   careful consideration should be given to planning experiments with animals results should provide details of experimental designs, including sample sizes, of mouse experiments.

“A wealth of evidence shows that across many areas, the reporting of biomedical research is often inadequate, leading to the view that even if the science is sound, in many cases the publications themselves are not ‘‘fit for purpose,’’ meaning that incomplete reporting of relevant information effectively renders many publications of limited value as instruments to inform policy or clinical and scientific practice.” *Kilkenny et al., PLOS Biology, June 2010.

Why are specifics not reported?

 Lots of reasons  space     lack of understanding of their importance lack of confidence in methods used or inability to articulate methods recognition that results may look less important if all details are included they are not required!

 peer reviewers don’t care or understand  novel approaches often get panned because it isn’t a t-test, for example.

Why are specifics not reported?

 If you don’t have good rationale for your planned experimental design, what are you going to say?

 “we experimented on 3 mice, and the p-value was non significant. So, we continued adding 3 mice to the experiment until we got a significant p-value.”  “We chose 6 mice per group because we always use 6 mice per group.”  “We chose 8 mice per group because that was all the budget allowed.”  “We chose to report the results of differences in tumor size at day 45 because differences at all other days were not statistically significant.”

ARRIVE guidelines: 20 item checklist

TITLE (1)  Provide as accurate and concise a description of the content of the article as possible.

ABSTRACT (2)  Provide an accurate summary of the background, research objectives (including details of the species or strain of animal used), key methods, principal findings, and conclusions of the study.

INTRODUCTION   Background   (3 a). Include sufficient scientific background (including relevant references to previous work) to understand the motivation and context for the study, and explain the experimental approach and rationale.

(3 b.) Explain how and why the animal species and model being used can address the scientific objectives and, where appropriate, the study’s relevance to human biology.

Objectives (4) Clearly describe the primary and any secondary objectives of the study, or specific hypotheses being tested.

ARRIVE guidelines: 20 item checklist

METHODS  Ethical statement (5) Indicate the nature of the ethical review permissions, relevant licences (e.g. Animal [Scientific Procedures] Act 1986), and national or institutional guidelines for the care and use of animals, that cover the research.

 Study design (6) For each experiment, give brief details of the study design, including:   a. The number of experimental and control groups.

b. Any steps taken to minimise the effects of subjective bias when allocating animals to treatment (e.g., randomisation procedure) and when assessing results (e.g., if done, describe who was blinded and when).

  c. The experimental unit (e.g. a single animal, group, or cage of animals). A time-line diagram or flow chart can be useful to illustrate how complex study designs were carried out.

Experimental procedures (7) For each experiment and each experimental group, including controls, provide precise details of all procedures carried out. For example:  a. How (e.g., drug formulation and dose, site and route of administration, anaesthesia and analgesia used [including monitoring], surgical procedure, method of euthanasia). Provide details of any specialist equipment used, including supplier(s).

    b. When (e.g., time of day).

c. Where (e.g., home cage, laboratory, water maze).

d. Why (e.g., rationale for choice of specific anaesthetic, route of administration, drug dose used).

Experimental animals (8)   a. Provide details of the animals used, including species, strain, sex, developmental stage (e.g., mean or median age plus age range), and weight (e.g., mean or median weight plus weight range).

b. Provide further relevant information such as the source of animals, international strain nomenclature, genetic modification status (e.g. knock-out or transgenic), genotype, health/immune status, drug- or testnaıve, previous procedures, etc.

ARRIVE guidelines: 20 item checklist

    Housing and husbandry (9) Provide details of:    a. Housing (e.g., type of facility, e.g., specific pathogen free (SPF); type of cage or housing; beddingmaterial; number of cage companions; tank shape and material etc. for fish).

b. Husbandry conditions (e.g., breeding programme, light/dark cycle, temperature, quality of water etc. for fish, type of food, access to food and water, environmental enrichment).

c. Welfare-related assessments and interventions that were carried out before, during, or after the experiment.

Sample size (10)    a. Specify the total number of animals used in each experiment and the number of animals in each experimental group.

b. Explain how the number of animals was decided. Provide details of any sample size calculation used.

c. Indicate the number of independent replications of each experiment, if relevant. Allocating animals to experimental groups (11)   a. Give full details of how animals were allocated to experimental groups, including randomisation or matching if done.

b. Describe the order in which the animals in the different experimental groups were treated and assessed.

Experimental outcomes (12) Clearly define the primary and secondary experimental outcomes assessed (e.g., cell death, molecular markers, behavioural changes).

ARRIVE guidelines: 20 item checklist

 Statistical methods (13)  a. Provide details of the statistical methods used for each analysis.

  b. Specify the unit of analysis for each dataset (e.g. single animal, group of animals, single neuron).

c. Describe any methods used to assess whether the data met the assumptions of the statistical approach.

RESULTS   Baseline data (14) For each experimental group, report relevant characteristics and health status of animals (e.g., weight, microbiological status, and drug- or test naıve) before treatment or testing (this information can often be tabulated).

Numbers analysed (15)     a. Report the number of animals in each group included in each analysis. Report absolute numbers (e.g. 10/20, not 50%a).

b. If any animals or data were not included in the analysis, explain why.

Outcomes and estimation (16) Report the results for each analysis carried out, with a measure of precision (e.g., standard error or confidence interval).

Adverse events (17)   a. Give details of all important adverse events in each experimental group.

b. Describe any modifications to the experimental protocols made to reduce adverse events.

ARRIVE guidelines: 20 item checklist

DISCUSSION  Interpretation/scientific implications (18)    a. Interpret the results, taking into account the study objectives and hypotheses, current theory, and other relevant studies in the literature.

b. Comment on the study limitations including any potential sources of bias, any limitations of the animal model, and the imprecision associated with the resultsa.

c. Describe any implications of your experimental methods or findings for the replacement, refinement, or reduction (the 3Rs) of the use of animals in research.

  Generalisability/translation (19)  Comment on whether, and how, the findings of this study are likely to translate to other species or systems, including any relevance to human biology.

Funding (20)  List all funding sources (including grant number) and the role of the funder(s) in the study.

Manuscripts vs. grants

  Similar principles, but very limited space in grants Convince the reviewers   you have a clear question you have a method for gathering relevant data   you can measure the outcomes of interest you know what to do with the data  the results will answer the clear question

1. You have a clear question

   Stated objective and rationale should make it clear what your scientific question is.

Stating your hypothesis can be very helpful.   E.g., the knockout model will have faster tumor growth than the wildtype model.

E.g., gene expression will be lower in the mice treated with inhibitor compared to untreated mice.

This is standard in clinical research but not as common in basic/translational research.

2. You have a method for gathering relevant data

      Experimental design!!

Details….we need details.

If you cannot explain the design to the reviewer, he cannot understand the data that is generated.

Be very careful of ‘bias’s’ in your design Example: evaluating metastases.

  design says mice will be followed until death or primary tumor reaches xx mm 3 .

Differential follow-up time. Example: comparing tumor size   design says you will compare tumor volumes at day 60 preliminary data suggests that you will have had to sacrifice most of the animals in your control group by day 50. how can you compare tumors when the mouse died 10+ days ago?

2. You have a method for gathering relevant data

  Randomization?

 important to consider “confounders”  confounder: a ‘variable’ that might affect your outcome that is not related to the experimental conditions   e.g., shipping batch e.g., diet/temperature/location of cage Blinding?

 inherent biases when you know group assignment!

 important to NOT know if the mouse being evaluated is in the group you expect to do better/worse.

 subconcious effects. in clinical research, taken for granted.

2. You have a method for gathering relevant data

 Sources of variation?

 transgenic vs. xenografts models?

     how many cell lines? Or, are you using (multiple) primary tumors?

fresh vs. frozen tissue?

reference gene? is it really ‘stable’?

Sampling   longitudinal? same mouse measured repeatedly over time separate cohorts? sac’ing different cohorts over time Measurement   same vs. different ‘raters’?

different measurement approaches?

3. how you measure the outcome

  Very often see  “we will evaluate differences in antitumor efficacy”   tumor size at time t?

tumor growth rate?

   presence/absence of metastases?

tumor take rate?

caliper vs. imaging measures Other measures: gene expression, methylation, mutations, histology, etc.

  assays need to be included type of measure should be included  continuous expression?

  positive vs. negative?

IHC, for example, can be expressed in many ways.

4. You know what to do with data

    Example:    You have measured the tumor volume every 5 days for 100 days on 100 mice. 21 measures per mouse x 100 mice = 2100 measures.

Why would you look only at the measures at one time point?

Statistical analysis plan is very important.

There are different approaches for    continuous outcomes (e.g. tumor volume at time t).

binary/categorical outcomes (e.g, presence of metastases) time to event outcomes (e.g., time to death/sacrifice) There are different approaches for   longitudinal data comparing vs. estimating vs. dose finding

Example

 Moussa O, Ashton AW, Fraig M, Garrett-Mayer E, Ghoneim MA, Halushka PV, Watson DK. Novel role of thromboxane receptors beta isoform in bladder cancer pathogenesis. Cancer Research, 2008, Jun 1; 68(11): 4097-4104.

Xenograft mouse model. TCC-SUP tumorigenic human bladder cancer cells were selected as they express TP-β receptor and were used for the drug combination studies. Immortalized nontransformed normal urothelial SV-HUC cells were selected because they express the TP-α. These cells were stably transfected with pcDNA3, TP-α, or TP-β for cell transformation studies. Both cell lines were used in a s.c. model in immunocompromised (nu/nu) mice. TCC-SUP cells (5 × 10 injection 6 ) or SV-HUC cells (5 × 10 7 ) in Matrigel (BD Bioscience, Inc.) were injected s.c. into the right and left flanks of anesthetized mice. Tumor growth was monitored in these mice twice a week. For mice injected with TCC-SUP, GR32191 or vehicle control was administered daily (20 mg/kg) by gavage with treatment initiated 24 h after initial injection. Two cycles of cisplatin [single high dose (5 mg/kg) or single low dose (0.5 mg/kg)] were administrated at day 4 and day 11 post-tumor cell

The data

4000 3000 2000 1000 600 300 100 30 0 0 High Cis+GR High Cis Low Cis+GR Low Cis GR Control 20 40 Time (days) 60 80

What are our questions?

 Is the time to tumor initiation different across treatment groups?

 is onset later in the cisplatin groups than in the other groups?

 Is the growth rate different across treatment groups?

 is the growth rate for high cisplatin smaller or larger than for low cisplatin in the GR group?

 These are questions that can be addressed statistically.

 Vague questions:   Is tumor size different? (when?) Which treatment is the most effective?

(using what metric?)

Time to tumor initiation: antitumor effects of GR32191 and cisplatin treatment in immunocompromised mice. Subcutaneous tumors from TCC-SUP human bladder cancer cells were treated with vehicle control (12 mice), GR32191 (15 mice), 5 mg/Kg cisplatin (cisplatin high; 12 mice), 5 mg/kg cisplatin in combination with GR32191 (13 mice), or 0.5 mg/kg (single low-dose cisplatin) alone (10 mice), or 0.5 mg/kg cisplatin in combination with GR32191 (10 mice). Tumor size was measured over time. Kaplan Meier curves showing time to tumor onset across the treatment groups.

High Cis+GR High Cis Low Cis+GR Low Cis GR noGR 10 30 Time to Tumor (Days) 50 70

Table 2: Hazard ratios comparing time to tumor in treatment groups. A hazard ratio greater than 1.00 implies that the first treatment has shorter time to tumor than the second in the comparison. For example, the hazard ratio comparing Low Cis vs. Low Cis + GR is 23.5. This implies that, at any given time for mice who haven’t yet developed a tumor, mice treated with Low Cis were 23 times more likely to have tumor incidence than mice treated with Low Cisp + GR. Hazard ratios less than 1 imply a protective effect. For example, the hazard ratio for GR vs. Low is 0.51. This implies that for mice who haven’t yet developed a tumor, those treated with GR are 0.51 times as likely to have tumor incidence as mice treated on Low Cis at any given point in time.

Comparison p-value No GR vs. GR No GR vs. Low No GR vs. Low + GR No GR vs. High No GR vs. High + GR GR vs. Low GR vs. Low + GR GR vs. High GR vs. High + GR Low vs. Low + GR Low vs. High Low vs. High + GR Low + GR vs. High Low + GR vs. High + GR High vs. High + GR Hazard Ratio 2.42

1.24

29.1

304.1

556.5

0.51

12.0

125.6

229.8

23.5

245.7

449.6

10.5

19.1

1.83

0.02

0.63

<0.0001

<0.0001

<0.0001

0.12

0.0001

<0.0001

<0.0001

<0.0001

<0.0001

<0.0001

<0.0001

<0.0001

0.19

95% Confidence interval for hazard ratio 1.12, 5.22

0.51, 2.99

8.70, 97.2

65.4, 1414.2

113.7, 2722.7

0.22, 1.20

3.29, 43.79

25.2 626.4

43.9, 1203.3

5.88, 93.9

45.7, 1320.5

79.8, 2531.5

3.36, 32.6

5.77, 63.4

0.73, 4.61

4000 3000 2000

Tumor growth rate: how to compare?

High Cis+GR High Cis Low Cis+GR Low Cis GR Control 1000 600 300 100 30 0 0 20 40 Time (days) 60 80

4000 3000 2000 Data used for tumor growth analysis: notice that for mice With no tumor onset, they are included from day 60+. Remaining Mice all have data shown for volumes>0.

High Cis+GR High Cis Low Cis+GR Low Cis GR Control 1000 600 300 100 30 0 0 20 40 60 Time from Tumor Initiation (days) or Day 60 80

4000 3000 2000 Fitted regression lines per mouse, by treatment group (stage 1 of Two stage analysis).

High Cis+GR High Cis Low Cis+GR Low Cis GR Control 1000 600 300 100 30 0 0 20 40 Time from Injection (days) 60 80

4000 3000 2000 Estimated regression lines per treatment group (result of stage 2 of Two stage analysis) High Cis+GR High Cis Low Cis+GR Low Cis GR Control 1000 600 300 100 30 0 0 20 40 Time from Injection (days) 60 80

Comparisons of slopes

Table 3: P-values for comparing slopes of tumor growth Control GR Low Cis Low Cis + GR High Cis GR 0.004

Low Cis 0.0001

0.75

Low Cis + GR <0.0001

0.47

0.58

High Cis 0.14

0.08

0.01

0.003

High Cis + GR 0.0006

0.89

0.84

0.49

0.03

Simpler ways to deal with the data?

  Simple comparisons across groups Example:  two groups of mice have tail vein injections to establish tumors. they are followed for a fixed amount of time and then are sacrificed.

 Question 1: All mice get tumors. you want to compare tumor burden in the two groups of mice. What test would you use to compare them?

a) b) c) d) e) t-test Wilcoxon rank sum test Anova Fisher’s exact test it depends

Simpler ways to deal with the data?

  Simple comparisons across groups Example:  two groups of mice have tail vein injections to establish tumors. they are followed for a fixed amount of time and then are sacrificed.

 Question 3: SOME mice get metastases. You want to compare incidence of metastases. What test should you use?

a) b) c) d) e) f) Chi-square test Fisher’s exact test Anova Kaplan Meier Signed rank test it depends

Simpler ways to deal with the data?

  Simple comparisons across groups Example:  two groups of mice have tail vein injections to establish tumors. they are followed for a fixed amount of time and then are sacrificed.

 Question 2: All mice get tumors. you want to compare tumor burden in the two groups of mice. What are you comparing if you use a t-test?

a) b) c) d) e) the distribution of tumor volume the distribution of the log of tumor volume the mean of tumor volume the mean of log tumor volume It depends

Simpler way to deal with the data?

     By using simple approaches, you might be oversimplifying This can hurt you. Example: you average triplicate values  naively assuming that there is 1/3 of the data as truly exists  this will make your standard errors larger than they should be This can invalidate your results.

Example: repeated measures on the same mouse assumed to be independent   naively assuming that there is more data than truly exists.

this will make your standard errors smaller than they should be

5. Lastly…how many mice should you use?

 Stay tuned for next week’s talk by EKG.

Quick aside….interpreting pvalues

 Definition: the p-value is the probability of getting a result as or more extreme than you observed if the null hypothesis is true  Can anyone translate that?

Hypotheses

  Ho: mean gene expression is the same in two groups H1: mean gene expression is different in two groups  Next: we do an experiment, we collect data (e.g. gene expression), we perform a test.

 What will affect the p-value?

 which hypothesis is actually true   the variance of the values in each group the SAMPLE SIZE!!!

What if…?

 What if we find a p-value of 0.02 comparing the mean gene expression in two groups? Your conclusion is….

  i need more information including:  the effect size! how different is the gene expression?

 the sample size! how many samples/animals were in each group?

And, it would be nice to ‘see’ the data either in a figure of by knowing the means and standard deviations in the two groups

Two scenarios

 Scenario 1: 5 mice per group. Gene expression is 4 times higher on average in the KO group compared to the WT group. P-value is p=0.02.

 Scenario 2: Tissue microarray. Gene expression is 1.2 times higher in late stage cancers (n=100) compared to the normals (n=50) (p=0.02). There is no significant difference (fold change = 1.05) between early stage cancers (n=100) and normals (p=0.45).

Take home points

1.

2.

3.

4.

Never interpret a p-value without additional information statisticians can help you. at a minimum, we help you clarify your experimental design and definition of outcomes.

statisticians have many tools in our toolboxes. A t test is not the hammer and every experiment is not a nail.

sample size justification will be next….very important (maybe most important) piece of statistical considerations in your grant.