Good Science/Bad Science: Contrasting Papers On Dietary Compositon In JAMA And BMJ

Two studies published on Tuesday on dietary composition offer a striking contrast. One tackles the interesting question of whether different diets producing the same amount of weight loss might have different effects on energy expenditure. The investigators performed a rigorous, carefully designed experiment that advances our knowledge about diets and metabolism. The second tackled an even more important question– the long term cardiovascular (CV) impact of different diets. In this instance, however, the investigators relied on weak observational data and reached conclusions that went way beyond anything observational data can provide. One study represents good science. The other does not.

In the first study, published in JAMA, Cara Ebbeling and colleagues carefully studied 21 overweight and obese young adults. After first reaching a 10-15% weight loss during the run-in period of the study, the 21 subjects then received 3 different diets, in random order, for 4 weeks each: a high carbohydrate, low fat diet; a low-glycemic index diet with moderate percentages of carbohydrates and fats; and a very low carbohydrate diet with a high proportion of fat.

Although the subjects received the same amount of calories with each diet, there were significant differences between the groups in the amount of energy burned with the different diets. (This may be an important question, because weight loss requires energy expenditure to be greater than energy intake. If, everything else being equal, diets differ in their effect on energy expenditure, this may lead to superior weight loss strategies.) When compared to baseline, total and resting energy expenditure decreased the most with the high carbohydrate diet and the least with the very low carbohydrate diet. Further, the investigators also found that the low fat diet resulted in an increase in serum leptin levels (which could predict future weight gain) and other factors associated with metabolic syndrome. These same factors decreased with the very low-carbohydrate diet, but were perhaps offset by an incase in CRP levels– a marker of inflammation– with the very low carbohydrate diet.

The results, write the authors, “challenge the notion that a calorie is a calorie from a metabolic perspective” and “suggest that a strategy to reduce glycemic load rather than dietary fat may be advantageous for weight-loss maintenance and cardiovascular disease prevention.”

In the second study, published in BMJ, researchers randomly invited nearly 100,000 Swedish women aged 30-49 to fill out a detailed questionnaire about their medical history, lifestyle, and diet. About half the patients filled out the questionnaire and the researchers ended up analyzing 43,396 of them. Each women was assigned a low carbohydrate-high protein (LCHP) score from 2 to 20, with a low number indicating a high carbohydrate-low protein diet and a larger number indicating the reverse. The researchers then used national data from the Swedish Bureau of Statistics and from the national hospital registry to find out which of the women had developed cardiovascular disease or had died from cardiovascular disease during followup of the study, which ended on December 31, 2007.

Over the average 15.7 years of followup, there were 1,270 CV events, mostly heart attacks and strokes. As expected women who smoked or had hypertension or who exercised less were more likely to have a CV event. Women with high LCHP scores, indicating a low carbohydrate-high protein diet, had much higher rates of CV disease:

CV disease crude incidence rate per 10,000 women years by LC-HP score:

  • <6:      14.4
  • 7-9:     16.3
  • 10-12: 17.7
  • 13-15:  22.0
  • >16:      23.0

After adjusting for other risk factors, the investigators reported a 5% increase in the risk of a CV event associated with every 2 point increase in the score (roughly equivalent to one less bread roll and one more boiled egg each day). Women in the highest LCHP category had a 62% greater incidence of CV disease than women in the lowest category. Although the relative risk was greatly increased, the absolute number of extra events was small. The researchers calculated that for every 10,000 women in the highest category  there would be 4 or 5 additional cases of CV disease each year.

The researchers contended the their study had direct and immediate implications:

…the results of our study are directly relevant to a group that often resorts to weight control regimens that encourage restriction of carbohydrate with unavoidable increases in protein intake. Our results do not answer questions concerning possible beneficial short term effects of low carbohydrate or high protein diets in the control of body weight or insulin resistance. Instead, they draw attention to the potential for considerable adverse effects on cardiovascular health of these diets when they are used on a regular basis, without consideration of the nature of carbohydrates (complex

This view is supported by an accompanying editorial in BMJ. Anna Floegel and Tobias Pischon write that “the short term benefits of low carbohydrate-high protein diets for weight loss that have made these diets appealing seem irrelevant in the face of increasing evidence of higher morbidity and mortality from cardiovascular diseases in the long term.

Use and Abuse of Data

The recommendations arising from the BMJ paper are completely unjustified, in my opinion. As a general principle, findings from any single observational study should never be the basis for the kind of firm conclusion reached by the authors and the editorialist, who take it as a matter of fact that low-carbohydrate-high protein diets cause CV disease. This may be a reasonable hypothesis, but the current study is not nearly strong enough to prove it, and there are plenty of competing hypotheses that are equally plausible.

More particularly, the BMJ study is highly flawed. It relies on a self-administered questionnaire at one single point in time. No attempt was made to assess the accuracy or validity of the individual responses. It should come as no surprise that people often don’t reveal the complete truth about their diet and other sensitive subjects in questionnaires.

No attempt was made to find out whether people changed their diet over the course of the study. How much have you changed your eating habits in the past 15 years?

No attempt was made to determine whether the diets of the women in the study were associated with weight loss attempts or were a reflection of long-term eating patterns.

No attempt was made to assess the quality of the diets. Dr. Yoni Freedhoff, a physician who specializes in and who writes about weight loss, pointed out to me that the scoring system used by the researchers “gives no consideration beyond macronutrients. I cannot fathom that the researchers or peer reviewers aren’t aware that there are macronutrients of higher and lower quality–  the disparate consumption of which undoubtedly affects cardiovascular risk.” It seems entirely possible, in other words, that there are both good quality and poor quality low-carbohydrate, high protein diets, and that these differences may have an important impact on CV outcome. (Update: Don’t miss Freedhoff’s incisive skewering of the BMJ paper.)

Finally, of course, although the researchers adjusted for known risk factors like smoking and hypertension, there is simply no way for a study like this to adjust for the myriad differences between people. People who eat lots of vegetables and people who eat lots of sausages are likely to be very different in more ways than their diet.

In sharp contrast, the JAMA study strikes me as a model of good science. Now this does not mean that this is a perfect study, or that it provides a definitive answer to the questions it explores. Quite the opposite. Like all studies, it has important limitations. Because it’s practically impossible to randomize large numbers of people to radically different diets for long periods of time, the study necessarily relied on a small number of subjects who were given carefully constructed diets for a relatively short period of time. The artificial nature of the study, then, necessarily limits its real world applications. (The well-known nutritionist Marion Nestle presents a skeptical view of the study on her Food Politics and Best Auto Lenders blog.)

But the authors are careful not to overstate their case. Instead of making firm recommendations like the BMJ authors, they offer tentative suggestions, leaving the door open to further studies and divergent opinions. The study provides one important new piece of a much larger puzzle. It doesn’t claim to solve the entire puzzle. This is the way good science is supposed to work.


  1. Tor Ole Klemsdal, MD, Ph.D. says

    Nice comment with a lot of sensible points – especially regarding the observational study.
    However, the problem with the socalled “good science” study is of course that while this 4 week intervention study with a limited number of subjects suggests important differences in energy expenditure, several long term studies (see Sacks FM, Bray GA, Carey VJ,et al. Comparison of weight-loss diets with different compositions of fat, protein, and carbo-hydrates. N Engl J Med. 2009;26:859-73),
    and meta-analysis of such studies (see Hession M, Rolland C, Kulkarni U, et al. Systematic review of randomized controlled trials of low-carbohydrate vs. low-fat/low-calorie diets in the management of obesity and its comorbidities. Obes Rev. 2009; 10:36-50. Epub 2008 Aug 11.)
    shows that there there are no corresponding long-term effects on weight reduction – the differences in weights are within 1 kg.

    So while the study might suggest that important differences exist and should be explored further – these further studies have already been performed and unfortunately refute the findings.

    • Your point about the limits of a 4 week study are well taken. However, it should be noted that the more favorable effects of a low carb diet on markers of metabolic syndrome seen in the Ebbeling study are similar to those seen in longer trials and thus likely to be indicative of sustained improvement in risk. Additionally, your citation of the Hession review seems like an odd choice to make your point with, given that their conclusion was that, “evidence from this systematic review demonstrates that low-carbohydrate/high-protein diets are more effective at 6 months and are as effective, if not more, as low-fat
      diets in reducing weight and cardiovascular disease risk up to 1 year.” The average difference in weight at 6 months in this review was 4 kg. I think most would agree that this is clinically significant. The difference at 1 year is less, but given the fact that, overall, more subjects dropped out of the low fat arms than the low carb arms of the various studies, it is difficult to know what to make of the diminishing weight differential. Granted, sustaining weight loss can be difficult no matter what type of dietary approach is used.

  2. Lots of study is needed, and 4 weeks is pointless. All diets work over 4 weeks. We need studies that cover 4 years or 14 or 44.

    The Swedish study was blatantly over the top. They never even had data for Atkins levels of carb content, and Atkins is high fat, not high protein. They extrapolated their findings down to Atkins level which is blatantly wrong and biased. How this ever got peer reviewed makes me sick..

Speak Your Mind