Category Archives: lecture

Test next week (Oct 22, 2015)

Reminder, test next week, Oct 22 (our last meeting before Halloween).


Test details
Three parts, 100 points, total worth 25% of total grade in course.
I will give you a few choices for the final section (problem solving).  Duration: 3h.
synthesis of best experimental design to solve problems.

Sample test questions

What is independence from an experimental design perspective? Explain some simple solutions to deal with this issue
What are independent data points from an experimental design perspective and provide at least 2 solutions and give a real world example.
What is random variation from an experimental design perspective and provide an example.
What is the difference between individual variation, inter-individual variation, and intra-individual variation?
What is intra-individual variation, why do we care about and how can we use it?
What is a confounding effect and provide a real world example.
What is the difference between a confounding factor and a 3rd variable?
What is a hypothesis, provide a real world example and explain the difference between a hypothesis and prediction
What are some common pitfalls associated with making good hypotheses and good predictions?
What is a pilot experiment, what are the benefits, and what are the elements of an effective pilot experiment?
What is a null hypothesis and do we ever use them?
What is the difference between a confounding and indirect variable?
What is an indirect measure and why do we ever use them?
What is the difference between mensurative and manipulative experiment? Provide a decision tree for when you would use each (2 reasons for each).
Where do good ideas come from and how does this relate to critical scientific thinking?
The global goals are an inspiration. We need them. Design an experiment that can provide the evidence for one of the 17 goals listed
You are Matt Damon stranded on Mars. Alone. Design an experiment using only potatoes to test whether Martian soil can supply necessary nutrients. You have 120 potatoes with eyes, all the soil you can lug into your space station, lights, and a water production system. You should test not only whether the potatoes can be propagated to support you but whether they are nutritious. You have all the usual equipment to analyze both soils and organic matter an astronaut scientists has access to.
VW lied to the public about TDI cars. The programmed a cheat code that allows the car to pass aircare tests.   However, when driving it produces 4-40x above acceptable emissions. Design an experiment to determine the frequency that this cheat was applied to their TDI models (3 models with TDI engines), if they are set to cheat – the amount they pollute, and the extent that the cheat activates in testing within each model (i.e. is their sensitivity in the cheat turning on).
What is replication and why is it an important consideration is designing effective experiments?
What is pseudoreplication and what are the common sources?
When is pseudoreplication not an issue in designing experiments?
What are cohort effects?
What is a randomization sampling design?
Can you use random and regular sampling concurrently in an experiment?
What are the different sampling methods available in designing sample collection?
Why do you need to randomize the order that you treat replicates?
What are some tools that you can use to decide on how the extent that you replicate?
Why are most scientific study/findings are false?
What is an effect size?
What is an effect size and how did they influence the value of the finding in science?
What is the difference between a Type I and II error. Provide a real-world example following your definitions.
What are the different types of control and do we always need controls?
What is a procedural control?
What is a placebo effect? Explain with reference to types of controls.
What is the difference between factor and levels?
What is a blind or masked procedure, why do we do them, and what are the different kinds?
Calculate the total number of sample units needed for the following two experiments. Orthogonal .. and an unbalanced/incomplete factor design
p87 figure
p89 figure
Why do we sometimes use blocks and sometimes use paired designs in experiments?
What is the primary advantange of a paired design in an experiment?
What is a within-subject design and why is not pseudoreplication?
What are the advantages and disadvantages of a within-subject design?
Why are we sometimes forced to do split-plot designs? Explain split-plot designs in your answer as well using a figure.
What is the difference between a nominal, ordinal, and interval scale?
Calibration of scales makes good sense. However, how might you calibrate other measurement devices such as human observers. Develop a calibration plan including an indication how often you would do it.
What is the difference between precision and accuracy? Explain in words and with a figure too please.
You are managing a running race. Design a validation experiment to demonstrate that the system you have devised minimizes bias and inaccuracies. You have a limited budget for equipment but many volunteers.
Intra-observer reliability can be a significant challenge in designing some experiments. Propose three solutions associated with these classes of errors.
Errors in measurement often scale by magnitude. How do you test this phenomenom and solve?
Provide a visual decision tree associated with how you decide on the resolution of the measurements you take in an experiment?
Recording and storing data from experiments also pose challenges. List two challenges, why they are important, and propose solutions.
Why are experiments that are subject to floor or ceiling effects a problem?
What is the difference between sensitivity and specificity in measurements?
What do you really need to know to most effectively interpret the outcome of an experiment/test even if you know the specificity and sensitivity? Explain and provide a real-world experiment.
You have been told you have a marker in your blood associated with arthitis.   You have completed an experimental design course. What do you need to know to effectively interpret the likelihood that you will soon develop arthitis?
How do decide between two versus many levels for a given factor in designing an experiment? Use figures to show how you decide and provide a brief explanation in text form too. Make a general recommendation on the most likely design to best capture variation across many levels but still estimates variation/protects against drop-outs too.
Subsampling wthin a set of populations or across samples is a solution to solving challenges we sometimes face in designing experiments. Explain how you decide on what extent to subsample, describe trade-offs and when it is appropriate at some levels, and use a figure to illustrate what you are describing as well.
It is sometimes useful to unbalance replicates across groups or levels for ethical or practical reasons. Provide a set of guidelines including a checklist of considerations for researchers.   Clearly explain and show when you want to unbalance in favour of controls versus treatments and the converse.
Explain the most appropriate use for sequential sampling in designing experiments.
Explain stratified sampling and the gold standard in using it to sample heterogeneous populations/systems.
What is a latin squares design? Explain and include a short table listing its strengths and weaknesses.
Covariates can interact too. No surprise. Provide an example set of plots showing statistical interactions between a factor (continuous) and covariate (continuous) and no interactions and another set showing interactions/no interactions for a factor (two levels only) and another factor (three levels). Explain each set of plots in a brief figure legend.
A version of question 6.2 but for sleep and a study aid.
Provide a plot showing Simpson’s paradox for male versus female applications to a university.
Test a few terms from the biomedical literature.
Provide a personal version of a flowchart from your time training in ‘best experimental design’ (BED).


Power & effect sizes

Here is the power and effect size material I mentioned in class. It will really help you answer the test questions we prepped in class and capture the ideas more effectively.

Statistical power analysis.

Online power calculators: many. Here is a useful one. Many more though. You can use them to calculate power or to calculate the sample size you will need given your estimate of power.Unknown

Effect size
Super important idea. Read up online.
Here is great paper why it matters!
Why most published research findings are false.