Bad science: Paleo diet edition

A while ago, I wrote a post praising the scientific approach that advocates of the paleo diet appeared to be adopting in arguing their case. In particular, the language used by people like Gary Taubes in discussing dietary health seemed to show a keen appreciation for the key principles underpinning the scientific method. This includes separating causation from correlation, controlling for placebo effects and selection bias, etc, etc.

However, apparently not all paleo advocates are such sticklers for good scientific practice. For example, see this blog post by Jacques Rousseau, which skewers a new "occasional study" by Tim Noakes.[*] There is a lengthy follow-up post that also well worth reading when you have time.

The short version is that Noakes is a very prominent sports scientist in South Africa. He also happens to be an extremely vocal proponent of the low-carbohydrate-high-fat (LCHF) paleo diet, having undergone a Damascene conversion in recent years. The occasional study in question was published in the South African Medical Journal and details 127 unsolicited responses that Noakes received from people who have followed his advice in switching over to LCHF. These correspondences tell of all manner of dietary miracles and health wonders that have followed as a result, from substantial weight loss to curing "incurable" diseases like type II diabetes.

The problem with this study should be all-to-obvious to anyone who understands anything about scientific practice -- more on that in a minute.  Furthermore, a lot of people are (rightly) up in arms about how it managed to get through the peer-review process and into the country's flagship medical journal. Cynical observers have not been shy in suggesting that this is almost entirely down to Noakes' status within the local research community and very little to do with the scientific merit of the study itself. (To be fair, I'm not sure that a double blind submission would have been possible in this case.)

Now, Jacques does a very good job in explaining the manifold problems of the study. He also points out that Noakes' position on the necessity of such anecdotal evidence is very inconsistent. (If we had proper, scientifically validated evidence about the benefits of LCHF then we wouldn't require anecdotal evidence on top of that. To argue otherwise is to suggest that the scientific evidence in favour of LFHC is not actually particularly strong.) However, I think some of the commentators actually do a better job of pinpointing exactly why this study does not belong anywhere near a reputable scientific journal. For instance, "Chris" writes:

[...]You could prescribe or promote absolutely anything, and you would see some people benefit. The key point is that the sample you have is self selected from those who benefited enough that they felt the need to contact you. That is likely to be a small number of the total number of people who did indeed benefit. And we have no idea what proportion of the total number of people to have tried LCHF those people are. The fact that there are 127 people who've shown a benefit is evidence of one thing, and one thing only: those people's ability to write you an email. Can you tell me exactly what their dietary regimes were, down to the last macronutrient? Can you assure me that the change in their diet was not simply a catalyst for them to become more active, thus they expended more energy? Can you tell me that there were no other outside influences that could potentially act as a confounding variable? You can't, and you say so yourself in the article. Which begs the question of why did it get published? If we can't say anything other than these people got amazing results and said they were on LCHF then what exactly can we say?[...]

Emphasis mine. A follow-up contribution by another commentator (who was actually involved in one of the cases that Noakes cites) is equally worth reading here.

To underscore something that Jacques and many of his commentators try to make abundantly clear; criticism of this particular study does not amount to criticisms of LCHF in of itself. The outcry is entirely about sloppy scientific reasoning and misuse (absence?) of the scientific method. Proponents of LCHF and other paleo-style diets may well be correct in identifying the causes of our modern dietary ills. I personally know more than a few people who credit it with helping them to shed weight and improve their overall sense of well-being. On the other hand, I can say exactly the same thing about friends who have converted to veganism. (You see the problem with anecdotal evidence!)

To conclude, if paleo advocates want to maintain scientific credibility, they need to distance themselves from this type of research. At the very least, they should not try to defend it.
___
[*] You may recall that I actually mentioned Prof. Noakes at the beginning of my previous post. Rousseau is a senior lecturer at the University of Cape Town, whom it should be said took me for an introductory philosophy and business ethics course during my undergrad.