Bad science: Paleo diet edition

A while ago, I wrote a post praising the scientific approach that advocates of the paleo diet appeared to be adopting in arguing their case. In particular, the language used by people like Gary Taubes in discussing dietary health seemed to show a keen appreciation for the key principles underpinning the scientific method. This includes separating causation from correlation, controlling for placebo effects and selection bias, etc, etc.

However, apparently not all paleo advocates are such sticklers for good scientific practice. For example, see this blog post by Jacques Rousseau, which skewers a new "occasional study" by Tim Noakes.[*] There is a lengthy follow-up post that also well worth reading when you have time.

The short version is that Noakes is a very prominent sports scientist in South Africa. He also happens to be an extremely vocal proponent of the low-carbohydrate-high-fat (LCHF) paleo diet, having undergone a Damascene conversion in recent years. The occasional study in question was published in the South African Medical Journal and details 127 unsolicited responses that Noakes received from people who have followed his advice in switching over to LCHF. These correspondences tell of all manner of dietary miracles and health wonders that have followed as a result, from substantial weight loss to curing "incurable" diseases like type II diabetes.

The problem with this study should be all-to-obvious to anyone who understands anything about scientific practice -- more on that in a minute.  Furthermore, a lot of people are (rightly) up in arms about how it managed to get through the peer-review process and into the country's flagship medical journal. Cynical observers have not been shy in suggesting that this is almost entirely down to Noakes' status within the local research community and very little to do with the scientific merit of the study itself. (To be fair, I'm not sure that a double blind submission would have been possible in this case.)

Now, Jacques does a very good job in explaining the manifold problems of the study. He also points out that Noakes' position on the necessity of such anecdotal evidence is very inconsistent. (If we had proper, scientifically validated evidence about the benefits of LCHF then we wouldn't require anecdotal evidence on top of that. To argue otherwise is to suggest that the scientific evidence in favour of LFHC is not actually particularly strong.) However, I think some of the commentators actually do a better job of pinpointing exactly why this study does not belong anywhere near a reputable scientific journal. For instance, "Chris" writes:

[...]You could prescribe or promote absolutely anything, and you would see some people benefit. The key point is that the sample you have is self selected from those who benefited enough that they felt the need to contact you. That is likely to be a small number of the total number of people who did indeed benefit. And we have no idea what proportion of the total number of people to have tried LCHF those people are. The fact that there are 127 people who've shown a benefit is evidence of one thing, and one thing only: those people's ability to write you an email. Can you tell me exactly what their dietary regimes were, down to the last macronutrient? Can you assure me that the change in their diet was not simply a catalyst for them to become more active, thus they expended more energy? Can you tell me that there were no other outside influences that could potentially act as a confounding variable? You can't, and you say so yourself in the article. Which begs the question of why did it get published? If we can't say anything other than these people got amazing results and said they were on LCHF then what exactly can we say?[...]

Emphasis mine. A follow-up contribution by another commentator (who was actually involved in one of the cases that Noakes cites) is equally worth reading here.

To underscore something that Jacques and many of his commentators try to make abundantly clear; criticism of this particular study does not amount to criticisms of LCHF in of itself. The outcry is entirely about sloppy scientific reasoning and misuse (absence?) of the scientific method. Proponents of LCHF and other paleo-style diets may well be correct in identifying the causes of our modern dietary ills. I personally know more than a few people who credit it with helping them to shed weight and improve their overall sense of well-being. On the other hand, I can say exactly the same thing about friends who have converted to veganism. (You see the problem with anecdotal evidence!)

To conclude, if paleo advocates want to maintain scientific credibility, they need to distance themselves from this type of research. At the very least, they should not try to defend it.
___
[*] You may recall that I actually mentioned Prof. Noakes at the beginning of my previous post. Rousseau is a senior lecturer at the University of Cape Town, whom it should be said took me for an introductory philosophy and business ethics course during my undergrad.

Quote(s) of the Day - Marxism

Just to prove my even-handedness on this issue of a priorism and empirical validation, here is the father of the falsification doctrine himself, Karl Popper, on Marxism:

[O]ne might say that Marxism was once a science, but one which was refuted by some of the facts which happened to clash with its predictions[...] 

However, Marxism is no longer a science; for it broke the methodological rule that we must accept falsification, and it immunized itself against the most blatant refutations of its predictions. Ever since then, it can be described only as nonscience—as a metaphysical dream, if you like, married to a cruel reality.

HT: Jonathan Portes

F.A. Hayek, who was greatly influenced by Popper and actually parted company from the extreme a prioristic approach of his mentor, Mises, made a very similar point here (see from 04:30):

Both the Marxists and Freudians had the dreadful habit of insisting that their theories were irrefutable; they [were] logically and absolutely cogent. And that led me to see that a theory that cannot be refuted is not scientific.

UPDATE: Jon Catalan has some thoughts here. I left a comment underneath too.

Another post on empirics (and a priorism)

I was away last week in Amsterdam taking part in The Econometric Game, a competition involving 30 universities from around the world. Despite the... hmmm... dour reputation of econometric get-togethers, this ended being a very enjoyable and social event. I can certainly recommend Amsterdam to any would-be travellers too. It's relatively small, but built around tourism and there's a great deal more to the city than stoner coffeehouses and sex shops.[*]

Unfortunately, my team didn't make the finals and I think that some inexperience caught us out here, since our university decided to send an all new side this year after reaching the finals in two of the last three events.[**] The time pressure of putting together a complete academic paper in one day -- using some technical routines that you aren't necessarily familiar with -- is something that's hard to prepare for. I do think that our analysis was pretty respectable and we certainly ticked all the boxes highlighted by the case makers after the submission. However, we probably let ourselves down by not "selling" our results well enough in the conclusion and discussion parts of the paper. Nonetheless, definitely a good experience overall and congratulations to the top three teams: 1) University of Copenhagen, 2) Aarhus University, and 3) Harvard University. Danes ruling the roost!

The case topic itself was to investigate "the effect that maternal smoking during pregnancy has on birthweight". Of course, maternal smoking is associated with a range of afflictions in addition to low birthweight and premature birth; from clefts to intrauterine hypoxia. However, the long-term economic implications of low birthweight (and, thus, the causal impact of smoking) are far more important than many people realise. All other things being equal, low birthweight babies will on average suffer higher mortality rates, be more likely to have cognition and attention problems, and be more prone to unemployment and lower wage earnings in later life. (For example, see Black, Devereux and Salvanes; 2005.) 

Some of you may remember that I have actually discussed the issue of maternal smoking on this blog before. Of course, that post had very little to do with empirics and was instead aimed at exploring the philosophical ramifications of allowing pregnant mothers to smoke. In essence, it was a thought-experiment on whether it would be morally permissible to ban mothers from smoking (if this were somehow enforceable). The ensuing comments thread became quite excitable, so take a look if you want to see some divided opinions.

___

[*] Mind you, these are freely on display as well. Like any good boy from Cape Town, I am strongly in favour of dope legalization. Holland's drug policy is more complex than simple soundbites, but far superior to what one finds elsewhere.

[**] The coffeeshops had nothing to with it ;)

...

Having returned from Amsterdam, I saw that my mate Russell had left a comment under an older post concerning the paleo diet's strong popularity within libertarian circles. As a follower of the praxeological method, Russ suggests that I am "asserting a false choice" by claiming that that there is a marked inconsistency in the way that (some) libertarians invoke the scientific method and empirical evidence in finding support for their preferred worldview. You can see my reply here, which concludes: My broader point is that the praxeological fixation among its proponents has created a hermetic seal; a complete aversion to empirical methods that far surpasses the limits of what praxeology could (conceivably) claim to hold sovereignty over.

The language is perhaps a bit dramatic, but still accurate I think. A more specific point that I wanted to make is that if you are going to claim that econometrics and other empirical methods in economics hold no validity because of reasons X, Y and Z... Then it it behoves you to be equally dismissive of their applications to medical studies of the type that Gary Taubes (go-to-guy for the paleo crowd) advocates. For these too seek to identify causal effects in a world characterised by complex interactions between people and their changing environments. 

Libertarianism and the paleo diet - Natural or strange bedfellows?

I want follow up on the previous post by making a side observation on the paleo diet's marked popularity within libertarian circles -- especially anarcho-capitalists. Certainly, prominent libertarians like Lew Rockwell, Tom Woods and even Russ Roberts have gone "primal" in a very public sense.

Now, sure, this makes sense at the superficial level: It probably comes very naturally to many libertarians that they reject government-endorsed advice, take pride in swimming against the tide of mainstream thought, and so forth. Indeed, here is an article from LRC that openly flaunts the "ideological parallels" that underpin the libertarian-paleo connection. (Thanks Google!)

However, doing something simply because it feels "familiar" is likely to produce counterproductive results sooner rather than later. This is particularly likely to be true when it comes to endorsing heterodox thinking across unrelated topics.[*] After all, there's a reason why certain ideas tend to emerge as dominant over time: They explain the world around us better than the competing paradigms. (Obviously, this can change as we gain more knowledge. Science is a process after all and not an end result.)

More fundamentally, it seems to me that the underlying scientific approach of the paleo movement -- as outlined by Gary Taubes at least -- is at odds with much of what characterizes libertarian economic thought of the Mises.org variety. I've had my say about the dangers of extreme a priorism before and still consider the rejection of empirical testing by some Austrians to be a completely nonsensical dead end. Indeed, the a prioristic approach is antithetical to the empirically-based scientific approach that Taubes so strongly advocates.

Given his popularity among these groups, I wonder what Taubes makes of this?

[*] Rockwell, in particular, seems prone to indulging just about every crank,conspiratorial, anti-science theory around. (They're all there, by the way. HIV/AIDS, evolution versus creationism, vaccines and autism, climate change... Even David Icke, alien technology and unlimited energy!)

===

Speaking of libertarian-based economic research, a good example of why experimental design is so important can be found by looking at Selgin, Lastrape and White's (2010) much-circulated paper, Has the Fed has been a failure?. The study itself is a great piece of historical writing that makes an important contribution to the literature. It also uses some fairly advanced empirical techniques such as GARCH time-series modelling. However, SLW don't provide a compelling counterfactual and, as such, their work can at best be described as a "pre-post" evaluation. Despite some overzealous reviews in libertarian circles then, this study was never meant to provide a definitive answer to the question that it asked. To their credit, SLW actually emphasize right at the outset (p. 1):

These findings do not prove that any particular alternative to the Fed would in fact have delivered superior outcomes: to reach such a conclusion would require a counterfactual exercise too ambitious to fall within the scope of what is intended as a preliminary survey. The findings do, however, suggest that the need for a systematic exploration of alternatives to the established monetary system, involving the necessary counterfactual exercises, is no less pressing today than it was a century ago. 

UPDATE: Dan Kuehn weighs in with some thoughts here.  He's not completely convinced that SLW heed their own warning...