Monday, February 3, 2014

Worthless Journalism

This morning I came across a news story purporting to show that vaccination against the HPV virus does not make girls more likely to be sexually active. After reading it, I have no more reason to believe the conclusion than before reading it, making it a good example of bad journalism.

The article describes an experiment in which more than 300 girls between 13 and 21 were interviewed about their sexual behavior before getting the shot and at intervals thereafter. Before getting the shot 42% of the girls said they were sexually inexperienced. Six months after, of the 99 who had been inexperienced, 20 no longer were. That is the sum total of information about whether the vaccination makes girls more likely to be active provided by the article—there is also a little more information, almost equally worthless, about the effect on the behavior of those already active. 

So far as I can tell from the article, there was no control group. The article offers no information about how many girls from a group with the age distribution of those studied would be expected, absent the vaccination, to become sexually active over a period of six months. The only thing it tells us relevant to what it is supposed to be about is that the researchers say that their research produced the conclusion that, pretty obviously from the tone of the article, they wanted it to. No information by which a reader can decide for himself whether it's true.

Just for fun, I tried doing a back of the envelope calculation to see how plausible their claim was. To get an upper bound on the rate at which girls become active, assume that 100% go from inexperienced to experienced between age 13 and 21, making the average rate 11%/year. The story does not give the age distribution of the group studied, but if we assume an equal number of each age and take the group size to be 300, they should have been going from inexperienced to experienced at a rate of about 33/year. According to the article, the rate was actually 20 in six months. Insofar as the calculation tells us anything, it suggests that the rate for those vaccinated was higher than would have been expected—the opposite of the conclusion of the article and, presumably, the study.

That is not very strong evidence that the study's results were bogus, since we do not know the actual age distribution. It could have been concentrated in an age range where the average rate was as high as the rate observed. I have not read the study itself, so have no way of knowing whether it produced evidence for (or against) its conclusion. But the article on it is pure puffery.

The article mentioned, at the end, a previous study on a smaller scale that did use a control group. There too, it gave no information by which the reader could tell whether the reported conclusion was true. "That study ... found no significant differences" presumably means that the any differences found were not statistically significant, which sounds as though it means that there were no differences that matter—provided the reader does not know what "statistically significant" means.

----

P.S. A commenter (via Google+) provides me a link to the study. What the authors were looking at was not the relation between vaccination and sexual behavior but the effect of vaccination on perceived risk of sexually transmitted infections and the relation between that perception and sexual behavior. The authors report that for a majority of participants vaccination did not affect perception; they do not say what the effect was on the minority. They found no evidence that a reduction in perceived risk resulted in an increase in sexual activity. Their study produced no information on whether vaccination affected sexual behavior through mechanisms other than its effect on reported perception of risk—for example by implying that it was normal and accepted for girls to be sexually active.

The oddest thing in the paper is the theory they seem to be testing, which they refer to as risk homeostasis. That theory apparently holds that individuals act to maintain a constant level of risk. In the sexual context, that would imply that any reduction in the riskiness of intercourse would be exactly balanced by an increase in frequency.

From the economist's point of view, that is equivalent to assuming that the total amount a consumer spends on a good is independent of its price, that if the price falls in half the consumer doubles his consumption. It makes me wonder if risk homeostasis is the result of someone reading a comment by an economist and not understanding it.

No comments:

Post a Comment