Test analytics

Hi designers
I popped you each an individual email with your breakdowns.

m = memory
c = concepts
s = synthesis question

Here are some basic analytics:
1. There was no statistical relationship between study time and total test score.

2.  Students scored highest in memory, then concepts, then synthesis  (using the means).  However, this is a bit deceptive.  See plot below. Synthesis answers were the most positively skewed.  A few individuals brought down the mean for that section because they did not complete designing the third question for that section.

3250 test

3. The test is worth 25% so ensure you scale your calculations to that. However, out of 100 points, the median was near 70.

test total

Readings for Thursday

This Thursday I will be lecturing you on open source ecology/open science/big data, as well as formal synthesis tools such as systematic reviews and meta analyses.

Please have these four readings read BEFORE lecture on Thursday so we can discuss them in class:

Open-Source Ecology Takes Root Across the World
How to critically read meta-analyses.
Formalized synthesis opportunities
Uses and misuses of meta-analysis.

In lab on Thursday I will go into more depth on how to actually conduct a systematic review/meta analysis, and what will be involved for your particular assignment.

Take care, and be sure to read the post below about the due date for the lab report!

PS. If you’re interested, this paper is a good guide to types of reviews, and the review process: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3024725/



Hello designers,

I have received several requests from you guys for extensions on the lab report due Thursday.

While you have had a considerable amount of time to complete it, we know you are all very busy this time of year, so have decided to give everyone  a short extension until FRIDAY at midnight. 

Any report (or figshare) submitted after this time will be considered late, with a 20%/day penalty. As the weekends do not count, there will be a 20% penalty for any report submitted late on Friday, Saturday, Sunday, or Monday before 2:30 PM.

See you all Thursday!





FYI for anyone who missed lab today

Your paper is due Nov 5th by 2:30 PM, on Turnitin (see blog lost below). One report per individual, should NOT be the same (in any way) as any of your group members.

Report  – 20%

  • 5-8 pages of text (double spaced, size 12 font). Maximum 10 pages of text, then a few pages of figures (use the whole page please) and title/references pages
  • Here are the required sections:
    • title page
    • abstract
    • introduction
    • methods (experimental and statistical)
    • results (text and figures with captions)
    • discussion and conclusion
    • references (min. 5, please cite throughout in APA or CSE; name date format for in-text)

Figshare – 5%

  • data
  • metadata
  • tags (BIOL3250, ExperimentalDesign2015, RuttanED2015), plus additional tags
  • description (methods)

See you in two weeks!


Lab Report

Today in lab I will be going over your lab report expectations and the marking scheme, including things to avoid doing.

ALL LAB REPORTS WILL BE SUBMITTED THROUGH TURNITIN. We are trying to keep this course as paper-free as possible :). Computer/internet problems are not an excuse for late lab reports, please plan accordingly and submit a day or two early to check if there are any issues with your turnitin account if this is a concern for you.

If you have not already enrolled in the course through turnitin, the login info is:

Class ID: 10717724

Password: design

Additionally, as stats is a pre-req for this course, we will not be teaching it in labs. But, for your convenience, Alex Filazzola, the lab coordinator for BIOL2050, has made a few great tutorials that may be very helpful to you. These are available here:


See you all in lab!


Test next week (Oct 22, 2015)

Reminder, test next week, Oct 22 (our last meeting before Halloween).


Test details
Three parts, 100 points, total worth 25% of total grade in course.
I will give you a few choices for the final section (problem solving).  Duration: 3h.
synthesis of best experimental design to solve problems.

Sample test questions

What is independence from an experimental design perspective? Explain some simple solutions to deal with this issue
What are independent data points from an experimental design perspective and provide at least 2 solutions and give a real world example.
What is random variation from an experimental design perspective and provide an example.
What is the difference between individual variation, inter-individual variation, and intra-individual variation?
What is intra-individual variation, why do we care about and how can we use it?
What is a confounding effect and provide a real world example.
What is the difference between a confounding factor and a 3rd variable?
What is a hypothesis, provide a real world example and explain the difference between a hypothesis and prediction
What are some common pitfalls associated with making good hypotheses and good predictions?
What is a pilot experiment, what are the benefits, and what are the elements of an effective pilot experiment?
What is a null hypothesis and do we ever use them?
What is the difference between a confounding and indirect variable?
What is an indirect measure and why do we ever use them?
What is the difference between mensurative and manipulative experiment? Provide a decision tree for when you would use each (2 reasons for each).
Where do good ideas come from and how does this relate to critical scientific thinking?
The global goals are an inspiration. We need them. Design an experiment that can provide the evidence for one of the 17 goals listed
You are Matt Damon stranded on Mars. Alone. Design an experiment using only potatoes to test whether Martian soil can supply necessary nutrients. You have 120 potatoes with eyes, all the soil you can lug into your space station, lights, and a water production system. You should test not only whether the potatoes can be propagated to support you but whether they are nutritious. You have all the usual equipment to analyze both soils and organic matter an astronaut scientists has access to.
VW lied to the public about TDI cars. The programmed a cheat code that allows the car to pass aircare tests.   However, when driving it produces 4-40x above acceptable emissions. Design an experiment to determine the frequency that this cheat was applied to their TDI models (3 models with TDI engines), if they are set to cheat – the amount they pollute, and the extent that the cheat activates in testing within each model (i.e. is their sensitivity in the cheat turning on).
What is replication and why is it an important consideration is designing effective experiments?
What is pseudoreplication and what are the common sources?
When is pseudoreplication not an issue in designing experiments?
What are cohort effects?
What is a randomization sampling design?
Can you use random and regular sampling concurrently in an experiment?
What are the different sampling methods available in designing sample collection?
Why do you need to randomize the order that you treat replicates?
What are some tools that you can use to decide on how the extent that you replicate?
Why are most scientific study/findings are false?
What is an effect size?
What is an effect size and how did they influence the value of the finding in science?
What is the difference between a Type I and II error. Provide a real-world example following your definitions.
What are the different types of control and do we always need controls?
What is a procedural control?
What is a placebo effect? Explain with reference to types of controls.
What is the difference between factor and levels?
What is a blind or masked procedure, why do we do them, and what are the different kinds?
Calculate the total number of sample units needed for the following two experiments. Orthogonal .. and an unbalanced/incomplete factor design
p87 figure
p89 figure
Why do we sometimes use blocks and sometimes use paired designs in experiments?
What is the primary advantange of a paired design in an experiment?
What is a within-subject design and why is not pseudoreplication?
What are the advantages and disadvantages of a within-subject design?
Why are we sometimes forced to do split-plot designs? Explain split-plot designs in your answer as well using a figure.
What is the difference between a nominal, ordinal, and interval scale?
Calibration of scales makes good sense. However, how might you calibrate other measurement devices such as human observers. Develop a calibration plan including an indication how often you would do it.
What is the difference between precision and accuracy? Explain in words and with a figure too please.
You are managing a running race. Design a validation experiment to demonstrate that the system you have devised minimizes bias and inaccuracies. You have a limited budget for equipment but many volunteers.
Intra-observer reliability can be a significant challenge in designing some experiments. Propose three solutions associated with these classes of errors.
Errors in measurement often scale by magnitude. How do you test this phenomenom and solve?
Provide a visual decision tree associated with how you decide on the resolution of the measurements you take in an experiment?
Recording and storing data from experiments also pose challenges. List two challenges, why they are important, and propose solutions.
Why are experiments that are subject to floor or ceiling effects a problem?
What is the difference between sensitivity and specificity in measurements?
What do you really need to know to most effectively interpret the outcome of an experiment/test even if you know the specificity and sensitivity? Explain and provide a real-world experiment.
You have been told you have a marker in your blood associated with arthitis.   You have completed an experimental design course. What do you need to know to effectively interpret the likelihood that you will soon develop arthitis?
How do decide between two versus many levels for a given factor in designing an experiment? Use figures to show how you decide and provide a brief explanation in text form too. Make a general recommendation on the most likely design to best capture variation across many levels but still estimates variation/protects against drop-outs too.
Subsampling wthin a set of populations or across samples is a solution to solving challenges we sometimes face in designing experiments. Explain how you decide on what extent to subsample, describe trade-offs and when it is appropriate at some levels, and use a figure to illustrate what you are describing as well.
It is sometimes useful to unbalance replicates across groups or levels for ethical or practical reasons. Provide a set of guidelines including a checklist of considerations for researchers.   Clearly explain and show when you want to unbalance in favour of controls versus treatments and the converse.
Explain the most appropriate use for sequential sampling in designing experiments.
Explain stratified sampling and the gold standard in using it to sample heterogeneous populations/systems.
What is a latin squares design? Explain and include a short table listing its strengths and weaknesses.
Covariates can interact too. No surprise. Provide an example set of plots showing statistical interactions between a factor (continuous) and covariate (continuous) and no interactions and another set showing interactions/no interactions for a factor (two levels only) and another factor (three levels). Explain each set of plots in a brief figure legend.
A version of question 6.2 but for sleep and a study aid.
Provide a plot showing Simpson’s paradox for male versus female applications to a university.
Test a few terms from the biomedical literature.
Provide a personal version of a flowchart from your time training in ‘best experimental design’ (BED).


unofficial blog for experimental design @yorku