The whole assignment started in January, when we were all sent away to choose a primary research paper from a limited choice of recent journals. We couldn't choose one that another student had already chosen, it couldn't be a review or summary of other research, and the choice had to be made to a deadline. I spent a while looking through the journals, but these things are tricky to skim, time was short, and inevitably a choice had to be made without fully exploring every angle of the chosen article. So I plumped for the results of a Finnish survey that suggested that high levels of coffee intake were associated with lower risk of Parkinson's disease.
The first task was to choose a magazine, any magazine, and write an article about the research paper that was pitched at the right level and designed to appeal to readers of that magazine. I chose the BBC Good Food Magazine, given that I've never read any other magazine regularly except if you count the supplements that come with the Sunday papers. [I mentioned this article in my last post - I saw the lecturer today to query my mark and all is well, as I suspected she hadn't seen the second page and is happy to re-mark.]
The next job was to create a presentation about the research paper, and write a handout to accompany it, this time pitched at a professional audience of dietitians. I've been working on that over the last week or two, and it has been extraordinarily interesting, but not because the paper is interesting, or the task is interesting. It's because, despite having read every word of this research paper multiple times, it wasn't until yesterday that I realised that none of the results was significant.
That's 'significant' in the statistical sense, which involves crunching the numbers in some arcane formula that spews out a 'p-value' - the probability that the results you obtained were distributed that way entirely by chance. If you toss a coin 100 times, you probably won't get 50 heads and 50 tails, but if you got 45 heads and 55 tails then the probability of this happening by chance is strong, and p will be large, approaching 1.0. If you got 7 heads and 93 tails, you'd suspect that something was wrong with the coin and your p value would be small. That's not to say that 93 tails out of 100 is impossible, just that the odds of it happening with a non-dodgy coin are small.
You can set the cut-off point at anything you like, but most of the scientific research I've come across uses a p-value of 0.05. Below this - there's something going on, something has been done to the coin to affect the way it falls. Above this - the coin may have been treated in some way, but not enough to say that the treatment has worked in a way that satisfies the scientific community.
Back to the coffee and Parkinson's disease paper. What the researchers did was go back to the results of a lifestyle survey administered in the 1970's in Finland, and look through the results for what people said about the number of cups of coffee drunk at that time, and cases of Parkinson's disease in the next 22 years. You can see several of the major flaws straight away: how big is a cup? what strength is your coffee? did coffee consumption stay the same over 22 years? what about other caffeinated drinks?
The subjects gave a lot of other information that might have an effect on coffee drinkers' risk of Parkinson's disease, so the researchers thought they'd look for other factors too. Age, sex, smoking habits, body mass index, blood pressure, cholesterol level and more were thrown into the calculations, but it was important to adjust the numbers first. People who drink lots of coffee may be more likely to be smokers; people who smoke might tend to drink a lot of coffee, so if coffee drinkers are found to have fewer cases of Parkinson's disease, might it actually be due to smoking?
This was going well; I'd written most of the handout and the presentation was coming along nicely. What happened yesterday was that I read Ben Goldacre's 25 April blog post about sub-group analysis. What he describes very eloquently is that in a completely random situation, it's possible to apply some criterion and find a significant result. "The coins are randomly distributed throughout your christmas pudding. If you x-ray it, and follow a very complex path with your knife, you will be able to create one piece with more coins in it than the others: but that means nothing. The coins are still randomly distributed in the cake."
This seemed to be directly applicable to this study. If you examine numbers about coffee drinking, Parkinson's disease and a whole load of other factors that are actually distributed completely randomly, you still might find something significant.
They found an association between coffee drinking and incidence of Parkinson's disease once they'd adjusted for everything, but it wasn't quite significant (p=0.18). Then they looked at all the different factors they'd used, and found two that were significant: people who said 22 years ago that they drank more than 4 indeterminate sized cups of indeterminate strength coffee per day and who subsequently were diagnosed with Parkinson's disease were a) more likely to be overweight (p=0.04), or b) more likely to have low cholesterol levels (p=0.03). Bingo! They'd sliced up the pudding every which way they could, and found a cluster of coins.
So my presentation and handout recommend that we take no notice whatever of this research paper. The strange nature of statistics means that they may be right, but until someone does a study starting from scratch using the proposition "the risk of Parkinson's disease in coffee drinkers is modified by body mass index/cholesterol," we won't know for sure. And even then, I'd have my doubts.Hooray for Ben Goldacre! I strongly recommend his blog, and his book "Bad Science." He has posted a whole chapter on his blog that couldn't be included in the book because the subject was still suing him while it was being published. Go and read it. If I were not married, I would be tempted. Perhaps he would consider Lola II.
5 comments:
My p-value is two mugs of redbush and a pint of ribena. I don't think Mr Goldacre would be interested.
What a fascinating study you've got going there Lola.
If it didn't require all the previous course work and needing all that math and science stuff - I could really get interested in doing something like that! :0)
On another note - I noted that research in celiac disease is very limited because of the lack of funding. The doctor they quoted said that the lack of funding is due to the fact that less people donate money to this research because all celiacs are spending all of their time looking for gluten-free food that is (of course) expensive. I know of which he speaks.
I like the pudding and coin analogy! I wonder if you can put the same theory to the test about some of the dubious links that there are between being overweight and certain medical conditions. Some obviously make sense but others don't and I do have a bee in my bonnet about it.
Sub-group analysis sounds a bit like what we call the "look-elsewhere" effect. If you look at a distribution, you can maybe find one small region where the prediction and the measurement are a long way apart. But to assess how unlikely this is, you have to take into account that the deviation could have happened anywhere in the distribution, and this can have a HUGE effect on whether something is significant or not. Same thing if you bring in more and more variables and look for sub-groups of the sample. Then you can find a sub-group with a very significant outcome.. but you have to correct that significance for the number of sub-groups you created, and the fact most of them looked insignificant.
Sal - there's obviously a ton of research going on about the effects of being overweight and what to do about it, and we cover quite a lot of it in the course.
Fat cells used to be thought of as just a place where the body stores lipids, but more recently they've discovered that adipose tissue is more like an active organ, secreting hormones and responding to metabolic conditions in the body. At the moment there seems to be a lot of activity in finding associations between all sorts of conditions and high levels of adipose tissue (or more accurately, a large waist measurement). The causes aren't so easy to determine, but unfortunately the associations do seem to be valid, even if they don't seem to make sense.
When I was researching metabolic syndrome, I found some interesting material from the government "Foresight" analysis, which essentially says that reducing obesity requires a societal approach (like tackling climate change), rather than expecting individuals to succeed on their own in something so difficult. It made me feel slightly better about my own lack of success.
Post a Comment