More on the dark side of statisitcal work

This one comes from a group of researchers at the RAND corporation on the relationship between TV viewing and sexual activity.

The paper, in the journal Pediatrics, can be found here. An earlier paper, which seems to say the same thing and is published in the same journal, can be found here. So, to begin with, we have a problem that the data is being recyled, even by the same journal, without any clearly value added content. Is anyone surprised they came up with the same results?

The paper came to my attention through a Washington Post article, followed by a discussion on the website. To those of you interested in causal arguments, I encourage a close look at the discussion. A lot of good points were raised during it, and the answers to the issues were not very convincing. The most common answers were that the researchers controled for many variables and used a longitudinal (meaning they followed the same people over time) study of individuals. Interestingly, both of these answers are completely wrong. Controlling for other correlates does not solve the causation problem, and, while time series designs can be pretty strong evidence of causation, in cases, such as this, where baseline selection (such as teens with more interest in sex or more free time spend more time watching TV) are so important for the outcome, it is definitely not a solution.

Far from causal information, I interpret the results as finding a strong correlation between youth that report watching TV with sexual content and youth that are willing to admit to having ever been pregnant over the phone.

I submitted a comment, but it was not chosen. Like so many others in the discussion, I have some concerns about the methodology that leads the researchers to makecomments such as this from the abstract:

"Limiting adolescent exposure to the sexual content on television and balancing portrayals of sex in the media with information about possible negative consequences might reduce the risk of teen pregnancy."

This line was also often repeated in the discussion, so the authors are trying to play up the causal effects of TV. There are though some very important problems with this statement:

First, the data collection is problematic. Surveys on sexual activity have been found in the social sciences to have many forms of bias in them, and phone surveys are even worse as parental ease-dropping is even more likely, and self dillusion becomes even more important when there's a human voice on the other end of the line. For instance, it is very unlikely that a youth who has had a secret abortion would basically admit such information over the phone. Again, social scientists know that this kind of information is very difficult to get in the best survey methodology, and this is not the best by far.

Second, the researchers don't ask "are you pregnant" but "have you ever been", meaning mis-reporting of pregnancy history is even more easy for the subjects. If you have any doubts about mis-reporting being a problem, keep in mind that in completely anonymous sex surveys men report two-to-four times as many sexual partners as do women, when the number should be the same.

Third, using baseline measures of sexual content does not solve for selection of individuals who are more interested in watching and engaging in sex. Think of it this way: if someone has a lot of free time to watch TV, maybe they have a lot of free time for sex. Or, if you are someone who really enjoys sex, perhaps you watch more sexy TV. The authors never solve these issues.

Fourth, why do they not have a control for religion? For many families, this will be the biggest indicator of how much sex is portrayed in TV shows they watch (for instance, Mormons and Baptists are known to be more focused on "family" TV, but still have issues with teen pregnancy).

Fifth, the selection of TV shows is not clear from the paper, and, from the Washington Post article, the authors aren't saying what the shows are.

For me, the biggest problem here though is the media attention this is getting. Including a causal argument where one is not warrented, and then beginning a media campaign from it, is one of the reasons the public becomes confused and tired by conflicting scientific research.