–The inevitable weaknesses of observational and diet studies
“Science is the great antidote to the poison of enthusiasm and superstition,” Adam Smith wrote more than 200 years ago. Unfortunately, it often seems as if the science of nutrition has itself been poisoned.
Two recently published papers illustrate this problem.
Nutrition and Mortality
A good example of enthusiasm and superstition is a new paper in the Journal of the American Medical Association that guesstimates the number of deaths from heart disease, stroke, and type 2 diabetes caused by 10 separate dietary factors. The authors developed extremely sophisticated models based on data from the National Health and Nutrition Examination Surveys and the National Center for Health Statistics and estimated that 45% of cardiometabolic deaths in 2012 (318,000 out of a total of 702,000) were “associated with” consuming either too little or too much of 10 dietary factors: fruits, vegetables, nuts/seeds, whole grains, unprocessed red meats, processed meats, sugar-sweetened beverages (SSBs), polyunsaturated fats, seafood omega-3 fats, and sodium.
They calculated that excess sodium caused the most damage, resulting in 66,000 deaths, or 9.5% of all cardiometabolic deaths. This was followed by low consumption of nuts and seeds (59,374 deaths, 8.5%), excess processed meats (57,766, 8.2%), low seafood omega-3 fats (54,626, 7.8%), low vegetables (53,410, 7.6%), low fruits (52,547, 7.5%), and high SSBs (51,694, 7.4%).
The study authors wrote that sodium is a “key target” but didn’t even acknowledge, as I have reported recently, that other, highly respected scientists now question the low-sodium mantra of the nutrition establishment. These scientists have even generated evidence raising the possibility that taking sodium too low may result in harmful outcomes. The authors also treated as settled fact the dangers of processed meats and low omega-3 fats but don’t acknowledge the controversies and weak evidence that bedevil the field.
‘Strong Assumption of Causality’
To be fair to the authors, of course they fully acknowledged that they had only established associations and that their study provided no evidence of causality. But, clearly, there’s no point in publishing a study without an underlying belief that these food factors really matter. As the authors of an accompanying editorial wrote, there is a “strong assumption of causality” in the study— and, indeed, in all similar studies looking at these sorts of associations.
But the authors and the editorialists shrugged off the inevitable limits of observational studies. Instead, they endorsed the wholesale application of these findings to public policy. The study, the editorialists wrote, has “the potential to guide policy makers in planning public health nutrition interventions” that could help prevent cardiometabolic deaths. The study authors adopted a similar line: “These results should help identify priorities, guide public health planning, and inform strategies to alter dietary habits and improve health.”
Here’s how fast the scientific rigor dissolves in the face of this urge to achieve significance and importance. A few hours after publication of the study the senior author, Dariush Mozaffarian, one of the top leaders in the nutrition field, put out a series of tweets which left no room for subtle distinctions between association and causation:
- “Our nation’s nutrition crisis: nearly 1,000 cardiovascular & diabetes deaths each day (!) due to poor diet.”
- “Soda and sugary drinks causing more than 50,000 deaths per year”
- “[T]oo many many sweet drinks and processed meats are the biggest dietary killers.”
This strikes me as irresponsible overreach and hubris (even for Twitter). The nutrition establishment has been playing this wink wink, nudge nudge game for a very long time. This is a perfect example of science being poisoned by enthusiasm and superstition.
Is it any wonder then that, just a few hours after the study was published, there were plenty of news reports that took their cue from the authors and editorialists and ignored or downplayed the association/causation problem. The headlines tell the story:
- NBC news: “These 10 Foods Affect Your Risk of Heart Disease the Most”
- AP: Bacon, soda and too few nuts tied to big portion of U.S. deaths
But What’s the Point?
Let’s pretend for a moment that the nutrition establishment has gotten it right and that all the dietary factors they’ve identified really do have the harmful and beneficial effects that they’ve observed. Would it even matter?
The problem is that it is nearly impossible to get people to change their diet on a large scale over a long period of time. The field of nutrition is littered with the corpses of failed diet studies. A recent study from Canada provides a good example. It illustrates the extreme difficulty of translating nutrition dogma to people in the real world.
In a paper published in the Journal of the American College of Cardiology, researchers randomized more than 900 overweight people to a control group, who received the Health Canada food guide, or one of three interventions. The first intervention group received dietary advice and counselling for 6 months; the second group did not have dietary advice but received a weekly food basket containing items recommended by the dietary advice; the third group received both the dietary advice and the weekly food basket.
The bad news: After 6 months, there was little or no difference in any of the groups in body weight, waist circumference, or blood pressure. After 6 months, participants across the groups had lost about 1 kg in weight. The authors concluded: “Increasing the intake of not only fruit, vegetables, and whole grain cereals but also functional foods proved difficult, even when these foods were provided weekly and despite the known effectiveness of such foods in managing CHD risk factors. Healthy shifts in diet among generally well populations is likely to require a range of sustained approaches and multiple forms of communication in a process measured in decades rather than months.”
In other words, even if you give people free counselling and advice and a free basket of “healthy” food every week, it won’t make a difference. That’s really a remarkable finding. Though in the context of other diet studies, we shouldn’t find it surprising, which suggests that the field could use a whole lot more humility and self-questioning.
It’s also worth pointing out that this paper received very little attention from the media, which is very likely a reflection of the bias against negative studies. I am sure that a smaller and weaker study would have received much more attention if it had been positive.
One of the most interesting moments in a recent talk by Salim Yusuf was when he reported that, surprisingly, there was no effect, negative or positive, linked to vegetable consumption. Given the central role of vegetables in the pantheon of the food gods I imagine many people may find this shocking and will resist this finding. But there’s also something liberating here. Yusuf made an important comment. He dismissed the idea that foods need to be judged based on their effect on health. “But I have to tell you, when it’s regarding diet, neutral is good. You have to eat something. If you like it eat it. Not every food has to be good or bad.”
In any case, if nutrition is going to move forward as a field, it will need to ruthlessly suppress the enthusiasms and superstitions with which it is riddled.
‘there is a “strong assumption of causality”’: so much that’s wrong with the medical trades is usefully summarised in that one blissfully foolish expression.
My old English teacher would have encouraged me to tear it to shreds. Whereas I, with the maturity of my years, simply fart in its general direction.
A Monty Python reference is perfect for this subject! And then there’s “Spam, spam, spam, spam…”
This paper has many other potential problems as well, which tend to be obscured by the fact that they use 10 separate dietary factors, leading to a profusion of tables and figures that is somewhat overwhelming. I think if it were to be limited to just one factor, with more information on that factor, the problems would be more obvious. To list just some of them, it’s unclear what relative risks they used and where they got them from, as some of them are from unpublished meta-analyses that have not been fully evaluated. Also, they ‘energy adjust’ the dietary intakes, which I think means, more or less, that if you eat two strips of bacon a day and 2000 calories you will appear to eat more bacon than someone who eats two strips of bacon a day and 3000 calories. This raises the question in part of whether the metrics that they are using for intake are actually the same metrics as used in the data for the meta-analyses or whether the relative risks are really valid for their data. The attributable fraction estimates do not appear to be properly adjusted for confounding factors. The entire enterprise is questionable.
I think thou dost protest too much. Epidemiology shows associations. We all know that – the authors said it so I am not sure why you are vilifying people that work hard to figure out what is scientifically true and make their best recommendations based on the data they have in hand. Often the associations are correct, too, especially when animal studies, population studies, genetic studies, and RCTs (even if limited) are in agreement. Indeed, we get it right most of the time – CHD event rates in the US have fallen dramatically, in large part due to risk factor control, some of which is due to meds and some due to nutritional changes. The problem is not the studies – it’s with over interpretation, over-generalization, over-certainty, and yes, over-promotion (Twitter is not a great place for medical knowledge but you and many of your medical friends like to Tweet oversimplifications, too). Salim Yusuf, who I respect greatly, is not giving good medical advice when he says that “If you like it eat it. Not every food has to be good or bad.” BUt it makes for a good quote. A wealth of evidence shows what kind of dietary patterns are associated with good CHD and cancer outcomes. The RCTs are weak or non-existent, but we have a pretty good idea. I have been a physician for over 25 years and can pretty much predict what will happen with a patient when they increase or decrease dietary sodium and saturated fat in individual patients. I am wrong sometimes too. But I don’t think you are doing anyone a favor pretending that we don’t know much about nutrition.
Jim, I confess to a certain amount of over-statement and poetic license in order to make a point. Nevertheless, the JAMA study does not strike me as terribly useful in any way. It is simply ammunition to defend public policy positions. And I simply don’t understand why they can’t acknowledge event the possibility that they may be wrong about sodium and sat fats. On the other hand, the JACC Canadian diet study is actually a really good and useful study, but the negative findings mean that its implications will be ignored or obscured by the nutrition establishment.
“CHD event rates in the US have fallen dramatically, in large part due to risk factor control”: indeed they have fallen. What’s the evidence that it’s anything to do with “risk factor control”, beyond, presumably, the decline in smoking?
Why don’t you guys talk about someone who does get results with their eating plan? Like @VinnieTortorich. High fat, moderate protein, and low carb. No sugar, no grains. Our bodies do not need them. The sugar and grains are what cause the obesity, heart disease and diabetes. Check out Dr. Jason Fung’s book, “The Obesity Code,” or Dr. Ron Rosedale’s “The Rosedale diet.” I have lost 50 pounds so far eating fatty meats, eggs, and green vegetables cooked in bacon grease.
So, when “association is not causation”, why refer to the PURE studies of dr. Yusuf et al., which are observational? Regarding Micha et al’s paper, they discuss this at length in the supplementary material to the paper. This also contain their justifications and evidence base.
I agree with Prof. Salim that not all food is harmful to everbody, being neutral helps. The person has to understand his body and how it reacts to each food they consume, helps them in maintaining their health- based on my personal experience.