Saturday, March 14, 2009

Another rant on bad science

A new paper by researchers in the Netherlands has gotten attention from the New Scientist, BBC and even Freakonomics for showing "a causal link between exposure to drinking models and alcohol commercials on acute alcohol consumption". The paper though seems to have a few curious problems.

The research design is very interesting. Individual males were invited to bring a friend to a relaxed lab setting to watch clips from movies where soda and alcohol were provided for free. When they showed up, they were randomly divided into 2 groups that were shown 1 hour long clips from either American Pie 2 and 40 Days and 40 Nights. Each of these groups were randomly divided into 2 more groups, with one being shown commercials involving alcohol use, the other not. The people were observed during the film clips to record alcohol use, and a questionnaire was given at the end to measure appreciation and familiarity with the movie and self reported previous levels of alcohol consumption. The results suggest that those that watched the "alcoholic movie" (American Pie 2) and alcohol commercials drank twice as much as those that watched the "non-alcoholic movie" (40 Days and 40 Nights) and no alcohol commercials.

While a nice enough study design, I think there are a few important problems with the study that the researchers (and media) have not talked about. First off, they claim a sample size of 80 people, though these were made up of pairs of people arriving. Even controlling for group clustering, there is no way the sample is greater than 40 since the partners were pre-selected by the individuals. The researchers even give away this fact on page 2 when they theorize that people are more comfortable drinking in groups.

Then there is the movie choice. While one is labeled as alcoholic and the other not, both contained a lot of drinking. American Pie 2 had 23 scenes of alcohol, and 40 Days and 40 Nights had 15. Any results must then be somehow interpreted that 8 more scenes of alcohol is somehow significant.

Their participants were also already heavy drinkers: "36.3% reported to have heavy drinking occasions once or twice a week and 17.5% more than twice a week. The average weekly consumption was 21.05 (SD=15.12) glasses".

Even with random assignment, balance was not very good: "The frequency of 6+ drinking did not differ significantly between the conditions, but last week’s alcohol consumption was higher in the AM/AC than in the NM/NC condition". This difference is, strangely enough, exactly divided along the lines of their results. That is, those shown the "alcoholic movie" and alcohol commercials had drunk more in the last week than those that watched the "non-alcoholic movie".

Then there is the difference in taste for the movie. The researchers claim that "American Pie 2 had a score of 3.52 (SD = 0.65) on a 5-point Likert scale ranging from 1 to 5, whereas ‘40 days and 40 nights’ had a score of 3.69 (SD = 0.72) ... indicating that differences in actual alcohol consumption cannot be attributed to differences in movie appreciation". This is a completely false conclusion: on average appreciation across movies was the same, but we don't know what this meant for individuals.

Most of these problems could be easily controlled for, but most concerning of all, the researchers do not use any of the information they have on people in their statistical analysis. They don't control for appreciation of the movie or previous drinking history. Despite having useful information, they ignore a large quantity of data.

The conclusion that exposure to drinking may increase drinking may be true, but this paper is done so badly, I can't believe the results at all.

No comments: