My presentation and summary all about the published paper on the link between coffee and Parkinson's disease is becoming more and more interesting.
The whole assignment started in January, when we were all sent away to choose a primary research paper from a limited choice of recent journals. We couldn't choose one that another student had already chosen, it couldn't be a review or summary of other research, and the choice had to be made to a deadline. I spent a while looking through the journals, but these things are tricky to skim, time was short, and inevitably a choice had to be made without fully exploring every angle of the chosen article. So I plumped for the results of a Finnish survey that suggested that high levels of coffee intake were associated with lower risk of Parkinson's disease.
The first task was to choose a magazine, any magazine, and write an article about the research paper that was pitched at the right level and designed to appeal to readers of that magazine. I chose the BBC Good Food Magazine, given that I've never read any other magazine regularly except if you count the supplements that come with the Sunday papers. [I mentioned this article in my last post - I saw the lecturer today to query my mark and all is well, as I suspected she hadn't seen the second page and is happy to re-mark.]
The next job was to create a presentation about the research paper, and write a handout to accompany it, this time pitched at a professional audience of dietitians. I've been working on that over the last week or two, and it has been extraordinarily interesting, but not because the paper is interesting, or the task is interesting. It's because, despite having read every word of this research paper multiple times, it wasn't until yesterday that I realised that none of the results was significant.
That's 'significant' in the statistical sense, which involves crunching the numbers in some arcane formula that spews out a 'p-value' - the probability that the results you obtained were distributed that way entirely by chance. If you toss a coin 100 times, you probably won't get 50 heads and 50 tails, but if you got 45 heads and 55 tails then the probability of this happening by chance is strong, and p will be large, approaching 1.0. If you got 7 heads and 93 tails, you'd suspect that something was wrong with the coin and your p value would be small. That's not to say that 93 tails out of 100 is impossible, just that the odds of it happening with a non-dodgy coin are small.
You can set the cut-off point at anything you like, but most of the scientific research I've come across uses a p-value of 0.05. Below this - there's something going on, something has been done to the coin to affect the way it falls. Above this - the coin may have been treated in some way, but not enough to say that the treatment has worked in a way that satisfies the scientific community.
Back to the coffee and Parkinson's disease paper. What the researchers did was go back to the results of a lifestyle survey administered in the 1970's in Finland, and look through the results for what people said about the number of cups of coffee drunk at that time, and cases of Parkinson's disease in the next 22 years. You can see several of the major flaws straight away: how big is a cup? what strength is your coffee? did coffee consumption stay the same over 22 years? what about other caffeinated drinks?
The subjects gave a lot of other information that might have an effect on coffee drinkers' risk of Parkinson's disease, so the researchers thought they'd look for other factors too. Age, sex, smoking habits, body mass index, blood pressure, cholesterol level and more were thrown into the calculations, but it was important to adjust the numbers first. People who drink lots of coffee may be more likely to be smokers; people who smoke might tend to drink a lot of coffee, so if coffee drinkers are found to have fewer cases of Parkinson's disease, might it actually be due to smoking?
This was going well; I'd written most of the handout and the presentation was coming along nicely. What happened yesterday was that I read Ben Goldacre's 25 April blog post about sub-group analysis. What he describes very eloquently is that in a completely random situation, it's possible to apply some criterion and find a significant result. "The coins are randomly distributed throughout your christmas pudding. If you x-ray it, and follow a very complex path with your knife, you will be able to create one piece with more coins in it than the others: but that means nothing. The coins are still randomly distributed in the cake."
This seemed to be directly applicable to this study. If you examine numbers about coffee drinking, Parkinson's disease and a whole load of other factors that are actually distributed completely randomly, you still might find something significant.
They found an association between coffee drinking and incidence of Parkinson's disease once they'd adjusted for everything, but it wasn't quite significant (p=0.18). Then they looked at all the different factors they'd used, and found two that were significant: people who said 22 years ago that they drank more than 4 indeterminate sized cups of indeterminate strength coffee per day and who subsequently were diagnosed with Parkinson's disease were a) more likely to be overweight (p=0.04), or b) more likely to have low cholesterol levels (p=0.03). Bingo! They'd sliced up the pudding every which way they could, and found a cluster of coins.
So my presentation and handout recommend that we take no notice whatever of this research paper. The strange nature of statistics means that they may be right, but until someone does a study starting from scratch using the proposition "the risk of Parkinson's disease in coffee drinkers is modified by body mass index/cholesterol," we won't know for sure. And even then, I'd have my doubts.
Hooray for Ben Goldacre! I strongly recommend his blog, and his book "Bad Science." He has posted a whole chapter on his blog that couldn't be included in the book because the subject was still suing him while it was being published. Go and read it. If I were not married, I would be tempted. Perhaps he would consider Lola II.