Notes From an Experienced Reviewer: Pay Attention to Study Design
If your experiment needs statistics, you ought to have done a better experiment. – Lord Ernest Rutherford
A Peer Reviewer’s Conundrum: Sample size of 1
In the journal Neuroimage, in 2009, a group of investigators reported an MRI study of one salmon which was 18 inches long. The fish, which was dead, was shown photographs of people in social situations with particular emotions and asked to determine what emotion that individual showed. Several regions in the brain were shown to be significantly activated by this task. This showed that random noise can yield spurious results even when statistical thresholds suck as P<0.001 are used. Multiple comparison corrections are needed. If comparisons are made and the number of participants is small, as in this study of one dead fish, the chances of false-positive findings are great.
Is it a False Positive or False Negative?
It is important to consider both false-positive and false-negative results. False-negative results occur when the data incorrectly indicate that a particular condition is absent. False positive results, on the other hand, occur when data suggest that a condition is present when it is not. The various forms of bias could contribute to a failure to consider these opportunities for error.
One man’s noise is another man’s signal – Sir Bernard Katz
Peer Reviewer’s Red Flags: Bias and Placebo Effects
Consider how the data were acquired and whether experimenter bias influenced the results. Placebo effects may also be involved. The magnitude of placebo effects is directly proportional to the cost, pain and difficulty of an intervention. This is why traditional remedies often contain bitter spices to make them appear to be powerful. As a moonlighting physician many years ago, I was doing evening house calls in Queens, New York. The manager of the practice had a class on how to complete home visits. He told me that every patient needed to get vitamin B12 injection and that I needed to show each patient the solution I was injecting, which was blue, because it would imply that the medication administered was effective. (I never gave the B12 shots because they were not indicated).
Is the Study Really What it Claims?
If a research study claims to be double-blinded, is that the case? In a placebo controlled double blind randomized trial, the participants as well as the investigators will not be told if the subject is getting the active drug or a placebo. However, the double-blind status only operates if the patient cannot correctly guess which group they are in. This is a quote form a patient in a trial of a medication for amyotrophic lateral sclerosis: I just had my one month….trial visit….today. I am guessing that I am not on placebo because each time I ramped up dosage I was nauseous for 3 days and that’s a known side effect.” This is called a nocebo effect (the opposite of a placebo effect, a negative outcome occurring due to the belief that the intervention is harmful or not effective).
Rather than looking at the statistics, look at the data – Robert Friedland, MD
Final tip for the Reviewer: Look at ALL of the Data
The first step of data analysis is looking at the data. All the data. Look for things that are obvious. Don’t rely on the summaries, such as mean and standard deviation, correlation coefficients, and significant or insignificant differences. Rather than looking at statistics, look at the data. The best results are those that are clear without the need of statistics. If data analysis in a clinical trial takes six months to complete and involves many statisticians, the results should be questioned for fear of data massaging (also referred to as data cleansing and data scrubbing).
This excerpt was republished with permission from Lesson 89 from the book “Ninety-Nine Lessons in Critical Thinking”, Oxford University Press, by Dr. Robert Friedland, MD, the author of this essay.
