the probability of his results being due to chance ...

I recently came across a sentence in a reasonably well balanced article on climate issues that nicely illustrates the problem of interpreting statistical claims. The relevant passage:

Despite the relative simplicity of his model, Crowley found good agreement between the temperature fluctuations it calculated for the years 1000 AD to 1850 AD and the fluctuations actually measured from tree rings during that interval. Over that 850-year period, fluctuations in solar intensity along with volcanic eruptions could account for roughly 50 percent of the variation seen in the tree-ring record -- give or take 10 percent. 

Something happened, however, after 1850. Crowley's model could only account for about 25 percent of the observed temperature changes. Something else was needed -- volcanic eruptions and solar variability were not enough.

Crowley then introduced a human-triggered greenhouse effect to the model and it produced a much better match.
....

In the case of Crowley's study, statistical tests show that the probability of his results being due to chance is less than 1 percent.

(Emphasis mine)

There are two things wrong with the final sentence. The first is linguistic ambiguity. "the probability of his results being due to chance" sounds as though it means "given the results he got, the probability that the cause was chance is less than 1 percent." What it actually means, however, is "if the results were due to chance, the probability of their occurring would be less than 1 percent."

To see the difference between the two readings of the sentence, consider a simpler case. I pull a coin out of my pocket and, without examining it, flip it twice. It comes up heads both times. My null hypothesis is that it is a fair coin, my alternative hypothesis is that it is a double headed coin.

Given the null hypothesis, the probability that the result would occur, that the coin would come up heads both times, is only .25. It does not follow that the probability that the result was due to chance is only .25 since, in my simple example, that would mean a .75 chance that the coin is double headed. To calculate the latter probability you would need to take account of the fact that double headed coins are very uncommon, hence, even after getting two heads, the odds are overwhelmingly against the coin being double headed.

Similarly for Crowley's result. It is a statement about how likely it is that his model would work as badly as it did for the period after 1850 if nothing had changed, not the probability that a change in what determined climate was responsible for its working as badly as it did.

But it doesn't even tell us that, because there is an additional assumption built into the argument—that the data from 1000 A.D. to 1850 A.D. provide a complete model of what determines climate. Suppose there is some cause of climate change that occurs rarely enough so that it did not occur in the period Crowley used to build his model, giving him no data at all on how likely it was or how large its effects. If it happened to hit around 1850, it could explain the divergence from his model thereafter, even though that divergence was entirely due to natural effects. Probability of that occurring? Unknown.

And in fact, if we accept Crowley's interpretation of his evidence, that is precisely what did happen—with "natural" interpreted to include the results of human activity. Every year for the past tens of thousands, there was a tiny probability of an industrial revolution producing enough CO2 to affect climate. It did not happen from 1000 A.D. to 1850, so did not show up in Crowley's data and the model built from that data. It did happen thereafter, producing results that look very unlikely given that model.

An earlier post on these issues, in the context of Lovejoy's similar 1% claim.

Earlier posts on the first problem in other contexts—World of Warcraft and Egyptian mummies.