If you want to demonstrate that a low-fat diet is better than a ketogenic diet (or that ultra-processed food makes us overeat), these critical design flaws will help.
Thanks again Gary for the reporting. As a layman, this whole thing is absurd. That Hall takes such umbrage in public and resigning in protest is likewise absurd. For God's sake, if we're going to spend millions of taxpayer dollars, get the study design correct at the outset, or don't spend the money.
You write so clearly that your criticisms of Hall seem intuitively obvious, and that makes me wonder how a respected researcher like Hall could fail to detect and neutralize such obvious errors early in the planning of his research. I guess there is enough funding only for flawed research?
I have thought a lot about this. When I was researching my first book--40 years ago, on high energy physics--living at CERN, the huge European physics laboratory, I had some of the best experimentalists in the world tell me they wouldn't do an experiment before they ran the design past a guy named Dick Garwin--https://en.wikipedia.org/wiki/Richard_Garwin. (I now see Garwin passed away just two weeks ago at 97-years-old.) They thought Garwin was the smartest guy they knew--Fermi, his mentor, once called him the "only genius" Fermi knew. These very smart experimentalists figured if there was a flaw in a study design, Garwin would spot it, if anyone could, and then they would fix it or not do the experiment. If Garwin thought the experiment wasn't worth doing, they wouldn't do it. The point, though, is they knew how hard it was to design a proper experiment and they wanted the smartest person they knew looking it over for them. (There's a great youtube video in which the physicist Freeman Dyson describes running some of this theoretical ideas past Fermi and then dropping them completely because Fermi explains to him why he's wrong: a similar deference to the smartest guy in the field. Here's the link: https://www.youtube.com/watch?v=hV41QEKiMlM )
I suspect Hall works mostly in a vacuum at NIH. It's not full of people who have been doing experiments like his for decades and thinking deeply about the challenges and flaws. And so I wouldn't be surprised if he gets no meaningful feedback at all when he designs a study. He might also think so highly of himself that he doesn't think its necessary (that's a problem with all of us). So it's only after it gets published that people point out how he might have screwed it up and the only people who care enough to do that are the folks who don't like the findings: i.e., Ludwig et al, myself, a few others. But once you've published a bad experiment and over-interpreted it, you're stuck, as I put it, defending the indefensible. It's a terrible place to be. It's easier to attack your critics, as Hall does in his letters and on X, than to take their criticisms seriously and respond in a way that really benefits the science. Ludwig could be mostly wrong on this stuff (although I doubt it) but it's up to Hall to engage with him whether he thinks Ludwig means well or not.
Thank you for this memorable example of Dyson's heroic humility. It shows the power of bowing to the truth notwithstanding self-interest. I'm going to pass it on to a friend whose father was a physicist. This friend is a lawyer like me, and seems to wish he had taken up the more mathematical profession of his father.
I envy you both because I don't have much math in me, but I appreciate it when genius is called to my attention as you did here, and as Dyson does so beautifully.
Clinician here (not researcher), with an acknowledged bias in favor of LCHF diets and the CIM.
When I look at the data from the LC/LF crossover study, it aligns perfectly with what I would expect to see based on my personal and clinical experience. Two weeks of low fat (high carb) eating would definitely cause me to over consume nutrients if I was then deprived of those carbs during the subsequent two weeks of LC (high fat) diet. I tell my patients all the time that there will be a temporary increase in hunger during the initial transition to a LCHF/keto diet, as the body “learns” how to utilize fat for fuel and access fat stores.
The reverse is also true. Once established in ketosis, I (and my patients) are far less prone to over consume carbs during temporary periods with limited access to normal diet (travel, etc).
Doesn’t the CIM / fuel-partitioning predict exactly this?
The folks given high carb first will end up eating more (it turned out to be WAY WAY MORE) when they shift to high fat, because the body thinks it’s starving. If the remainder of the trial went on longer than two weeks, I would expect to see those numbers decline rapidly.
And the reverse, that those folks who establish ketosis in the first two weeks of LC will be metabolically flexible enough to avoid over consuming in the subsequent LF phase.
My immediate reaction upon seeing the data was to marvel that Hall seems to have accidentally provided evidence in favor of the CIM. But his biases preclude him from seeing it. My biases led me straight to it.
Hi Suzie, I've been going back and forth with Ludwig on this since I published and he thinks that the CIM does make a prediction about calories consumed on a keto vs. very low-fat, carb-rich diet, per the study. My response is that it makes no prediction for the conditions of Hall's experiment, which is one of the reasons why it's such a bad experiment. I'm not sure it makes any prediction for calories consumed in any experiment unless you can fix energy expenditure and energy expenditure cannot be fixed--i.e., large portions of it are out of our control.
Think of it in terms of the energy balance equation:
ΔE (change in energy or, essentially, fat mass) = Ein - Eout.
So the CIM says carbohydrates are fattening, they increase ΔE, and they do it by increasing insulin and maybe decreasing glucagon. And if you increase ΔE, then Ein has to be greater than Eout. But the CIM doesn't predict how that difference plays out. It doesn't tell you that Ein goe up or Eout goes down, or both. It just says that ΔE goes up with a carb-rich diet and leaves it up to the physiology to manage the compensatory changes in Ein and Eout?
And the CIM predicts that when you remove carbs from the diet, per Hall's ketogenic diet, then ΔE goes down--i.e., the subjects lose fat mass--but it makes no prediction about whether that manifests itself as Ein going down or Eout going up or some combination (most likely) of both.
If you're going to test this, you can use the carb content of the diet to manipulate ΔE hormonally, and then you can fix Ein and see the result on Eout. But, as I said, you can't fix Eout to see the result on Ein because Eout is largely out of our control.
And I hope that makes sense. it's pretty simple, but I'm rushing through this a bit.
This is why I’m grateful to have my job and not yours (or Ludwig’s).
I simply get to appreciate that the LC/LF crossover data is a very close representation of what I observe in practice when patients switch between different diets.
To your point, this may not be the perfect study design to test the CIM, but it certainly doesn’t disprove it!
Also, maybe it would be helpful for me to clarify that I view the data on calorie consumption (Ein) as a way to measure hunger / “food noise.” Increased Ein/food noise following a low fat diet (and vice versa, decreased Ein/food noise following a low carb diet) is in perfect alignment with what I have observed again and again.
"I'm not sure it makes any prediction for calories consumed in any experiment unless you can fix energy expenditure and energy expenditure cannot be fixed--i.e., large portions of it are out of our control."
Another aspect of energy consumption is the share of energy consumed by the gut microbiome. Fermentation in the gut produces heat. This thermogenesis is typically not attributed to gut microbiome activity. https://europepmc.org/article/med/6575005
Sort of like recent events where they tell you not to trust your own eyes. If you've tried multiple diets, you quickly understand which one works the best. Once you get rid of the sugar you get rid of the weight.
Yes, similar. One thing that got me about Hall's experiment and was a bit off subject, as it didn't relate to the Ludwig critique, is that for 60-odd years or so, folks have acknowledged that ketogenic diets work and then explained that away by saying they merely work by getting people to eat less. Now Hall publishes a 2-week trial saying people eat more on a ketogenic diet (or at least more than the crazy 10% fat diet that serves as a comparison) and that is embraced without anyone thinking about the implications to the last 60 years of rationalizations and (sort of) science. Everyone admits--even Hall--that ketogenic diets are effective weight loss diets. So if they don't work by getting people to eat less, how do they work?
Just as a follow up on your comments on GLP's, I see a nurse practitioner who is a bodybuilder. She weighs 130 pounds and is 5'4". She's meticulous to the milligram of what she eats and how she exercises
As she gets close to a competition, she has a functional doctor that will let her use a low-dose of tirzepatide. Eating exactly the same amount measured, doing the exact same amount of exercise and she can takeoff 7 pounds prior to the competition. She was absolutely convinced this last year that the drug directly burns fat. I realize it's the study with an N of 1 - but given the person my guess it's as accurate as anything else. I just don't understand how they have all this so backward. And sadly, my colleagues just don't have the time
Hall made it apparent through his Twitter likes in 2024 that he thinks the Ludwig re-analysis is what should be retracted, not his paper. He seems a bit dramatic from an outsider's perspective, as if he takes all criticism personally. I suppose it's only human nature to become obstinate and defensive when backed into a corner. I can't think of an example in any science when a leading researcher humbly bent the knee to a feuding school of thought. This is one limitation of your boxing analogy, because defeated boxers will *sometimes* humbly acknowledge error and defeat.
I agree with you about Hall, or at least he certainly sees this situation differently than I do. Again, I grew up as a journalist in the physics world, watching physistists in conferences and symposia and even group meetings doing what they were suppose to do, which is ripping their colleague's ideas to shreds. The reason you present a paper in that world or give a talk is not convince your colleagues you're right but to get their criticism so you can understand where's you're screwing up. I have NEVER seen that kind of feedback in an obesity or nutrition meeting, online or elsewhere. And I think it's absolutely necessary for a functioning science. Because it's almost totally absent in nutrition/chronic disease research it's easier for someone to take it personally when it suddenly appears and is pointed at them.
Re the boxer metaphor, the corner throws in the towel because they don't want their boxer to continue taking a beating. One of the unfortunate situations in the Hall scenario is that he is very much not taking any beating. (In physics, he'd have been ridiculed for this kind of mistake and his work would never be taken seriously again.) Rather, so long as his conclusions are in line with the conventional wisdom, he's rewarded for the work. Look at the last post and the conversation I described that I had with Marion Nestle who called Hall's recent retirement a "national tragedy." And then the journalists look to Marion as a voice of nutritional authority, and the same uncritical bizarrely-rewarding attitude permeates the entire field.
how can you combine the two seperate diets together??!? and then to average such wildly different results? it appears to be willful manipulation to me.
Kevin Hall's Diet Order study brought to mind Penelope J. Greene's 2003 12-week study involving low-carb and low-fat diets. Greene's study never got published in a journal. However, there are at least two articles available. One of them says, "Many critics said that the results of the experiment violate the laws of thermodynamics, since the test subjects in certain groups lost more weight than their caloric intake. The experiment’s results also challenge classical nutrition theory, which mandates that the intake of calories should equal the amount of calories expended." https://www.thecrimson.com/article/2003/10/15/harvard-study-backs-low-carbohydrate-diet-a/
The other says, "Participants in all three groups lost weight, Greene said, with the low-fat group losing an average of 17 pounds and the low-carbohydrate group that ate the same number of calories losing 23 pounds. The biggest surprise, however, was that the low-carbohydrate dieters eating extra calories lost more than those on the low-fat diet. Participants in that low-carbohydrate group lost an average of 20 pounds." https://news.harvard.edu/gazette/story/2003/10/low-carb-more-effective-than-low-fat/
What's really odd is that epidemiologists typically ignore adipose tissue arachidonic acid. An 'adipose tissue arachidonic acid metabolic syndrome' web search brings up an AI Overview that says, "Elevated levels of arachidonic acid (AA) in adipose tissue are associated with an increased risk of the metabolic syndrome, particularly in individuals with obesity and non-alcoholic fatty liver disease (NAFLD). AA, a polyunsaturated fatty acid, plays a role in inflammation and metabolic processes, and its accumulation in adipose tissue may contribute to the development of metabolic dysregulation. " https://pmc.ncbi.nlm.nih.gov/articles/PMC2730166/
Please explain something. Who is responsible for retracting papers? The journal, or the author? If it can be either one, how is this decided? Logic dictates that if Hall’s studies are flawed and unethical, the publishing journals should retract them. Why is this not happening?
In this case, as I understand it, the journal has said that the author is ultimately responsible. If fraud was involved, the journal might insist, but no one is talking about fraud or misconduct here. This is just bad science and so it's apparently up to Hall to decide. If he was still at NIH, NIH might insist, I imagine (speculating wildly here) but he's not.
Ernie, your last point is a good point. How do you know? This leads to the kind of discussion I've had with one of my colleagues/allies who I think is a terrific scientist. I'll say to him, "A large number of otherwise very smart people think we're just wrong, how do we know we're not?" And he'll respond, a bit in jest but not entirely, "because we're not." And then I'll say, but isn't it the defining characteristic of a quack that they are sure they're right? And now we're back to where we started.
Thanks again Gary for the reporting. As a layman, this whole thing is absurd. That Hall takes such umbrage in public and resigning in protest is likewise absurd. For God's sake, if we're going to spend millions of taxpayer dollars, get the study design correct at the outset, or don't spend the money.
I agree. I hadn't gone out to write a piece focusing so much on Hall's research, but the more I read, the more outrageous it all became.
You write so clearly that your criticisms of Hall seem intuitively obvious, and that makes me wonder how a respected researcher like Hall could fail to detect and neutralize such obvious errors early in the planning of his research. I guess there is enough funding only for flawed research?
I have thought a lot about this. When I was researching my first book--40 years ago, on high energy physics--living at CERN, the huge European physics laboratory, I had some of the best experimentalists in the world tell me they wouldn't do an experiment before they ran the design past a guy named Dick Garwin--https://en.wikipedia.org/wiki/Richard_Garwin. (I now see Garwin passed away just two weeks ago at 97-years-old.) They thought Garwin was the smartest guy they knew--Fermi, his mentor, once called him the "only genius" Fermi knew. These very smart experimentalists figured if there was a flaw in a study design, Garwin would spot it, if anyone could, and then they would fix it or not do the experiment. If Garwin thought the experiment wasn't worth doing, they wouldn't do it. The point, though, is they knew how hard it was to design a proper experiment and they wanted the smartest person they knew looking it over for them. (There's a great youtube video in which the physicist Freeman Dyson describes running some of this theoretical ideas past Fermi and then dropping them completely because Fermi explains to him why he's wrong: a similar deference to the smartest guy in the field. Here's the link: https://www.youtube.com/watch?v=hV41QEKiMlM )
I suspect Hall works mostly in a vacuum at NIH. It's not full of people who have been doing experiments like his for decades and thinking deeply about the challenges and flaws. And so I wouldn't be surprised if he gets no meaningful feedback at all when he designs a study. He might also think so highly of himself that he doesn't think its necessary (that's a problem with all of us). So it's only after it gets published that people point out how he might have screwed it up and the only people who care enough to do that are the folks who don't like the findings: i.e., Ludwig et al, myself, a few others. But once you've published a bad experiment and over-interpreted it, you're stuck, as I put it, defending the indefensible. It's a terrible place to be. It's easier to attack your critics, as Hall does in his letters and on X, than to take their criticisms seriously and respond in a way that really benefits the science. Ludwig could be mostly wrong on this stuff (although I doubt it) but it's up to Hall to engage with him whether he thinks Ludwig means well or not.
Thank you for this memorable example of Dyson's heroic humility. It shows the power of bowing to the truth notwithstanding self-interest. I'm going to pass it on to a friend whose father was a physicist. This friend is a lawyer like me, and seems to wish he had taken up the more mathematical profession of his father.
I envy you both because I don't have much math in me, but I appreciate it when genius is called to my attention as you did here, and as Dyson does so beautifully.
Clinician here (not researcher), with an acknowledged bias in favor of LCHF diets and the CIM.
When I look at the data from the LC/LF crossover study, it aligns perfectly with what I would expect to see based on my personal and clinical experience. Two weeks of low fat (high carb) eating would definitely cause me to over consume nutrients if I was then deprived of those carbs during the subsequent two weeks of LC (high fat) diet. I tell my patients all the time that there will be a temporary increase in hunger during the initial transition to a LCHF/keto diet, as the body “learns” how to utilize fat for fuel and access fat stores.
The reverse is also true. Once established in ketosis, I (and my patients) are far less prone to over consume carbs during temporary periods with limited access to normal diet (travel, etc).
Doesn’t the CIM / fuel-partitioning predict exactly this?
The folks given high carb first will end up eating more (it turned out to be WAY WAY MORE) when they shift to high fat, because the body thinks it’s starving. If the remainder of the trial went on longer than two weeks, I would expect to see those numbers decline rapidly.
And the reverse, that those folks who establish ketosis in the first two weeks of LC will be metabolically flexible enough to avoid over consuming in the subsequent LF phase.
My immediate reaction upon seeing the data was to marvel that Hall seems to have accidentally provided evidence in favor of the CIM. But his biases preclude him from seeing it. My biases led me straight to it.
In other words, why wasn’t the headline: Low Carb Diet Protects From Overeating During Times of High Carbohydrate Availability
Hi Suzie, I've been going back and forth with Ludwig on this since I published and he thinks that the CIM does make a prediction about calories consumed on a keto vs. very low-fat, carb-rich diet, per the study. My response is that it makes no prediction for the conditions of Hall's experiment, which is one of the reasons why it's such a bad experiment. I'm not sure it makes any prediction for calories consumed in any experiment unless you can fix energy expenditure and energy expenditure cannot be fixed--i.e., large portions of it are out of our control.
Think of it in terms of the energy balance equation:
ΔE (change in energy or, essentially, fat mass) = Ein - Eout.
So the CIM says carbohydrates are fattening, they increase ΔE, and they do it by increasing insulin and maybe decreasing glucagon. And if you increase ΔE, then Ein has to be greater than Eout. But the CIM doesn't predict how that difference plays out. It doesn't tell you that Ein goe up or Eout goes down, or both. It just says that ΔE goes up with a carb-rich diet and leaves it up to the physiology to manage the compensatory changes in Ein and Eout?
And the CIM predicts that when you remove carbs from the diet, per Hall's ketogenic diet, then ΔE goes down--i.e., the subjects lose fat mass--but it makes no prediction about whether that manifests itself as Ein going down or Eout going up or some combination (most likely) of both.
If you're going to test this, you can use the carb content of the diet to manipulate ΔE hormonally, and then you can fix Ein and see the result on Eout. But, as I said, you can't fix Eout to see the result on Ein because Eout is largely out of our control.
And I hope that makes sense. it's pretty simple, but I'm rushing through this a bit.
This is why I’m grateful to have my job and not yours (or Ludwig’s).
I simply get to appreciate that the LC/LF crossover data is a very close representation of what I observe in practice when patients switch between different diets.
To your point, this may not be the perfect study design to test the CIM, but it certainly doesn’t disprove it!
Also, maybe it would be helpful for me to clarify that I view the data on calorie consumption (Ein) as a way to measure hunger / “food noise.” Increased Ein/food noise following a low fat diet (and vice versa, decreased Ein/food noise following a low carb diet) is in perfect alignment with what I have observed again and again.
"I'm not sure it makes any prediction for calories consumed in any experiment unless you can fix energy expenditure and energy expenditure cannot be fixed--i.e., large portions of it are out of our control."
One aspect of energy expenditure that typically isn't considered is unabsorbed calories. https://www.bmj.com/rapid-response/2011/10/31/unabsorbed-calories-important-consideration
Another aspect of energy consumption is the share of energy consumed by the gut microbiome. Fermentation in the gut produces heat. This thermogenesis is typically not attributed to gut microbiome activity. https://europepmc.org/article/med/6575005
Sort of like recent events where they tell you not to trust your own eyes. If you've tried multiple diets, you quickly understand which one works the best. Once you get rid of the sugar you get rid of the weight.
Yes, similar. One thing that got me about Hall's experiment and was a bit off subject, as it didn't relate to the Ludwig critique, is that for 60-odd years or so, folks have acknowledged that ketogenic diets work and then explained that away by saying they merely work by getting people to eat less. Now Hall publishes a 2-week trial saying people eat more on a ketogenic diet (or at least more than the crazy 10% fat diet that serves as a comparison) and that is embraced without anyone thinking about the implications to the last 60 years of rationalizations and (sort of) science. Everyone admits--even Hall--that ketogenic diets are effective weight loss diets. So if they don't work by getting people to eat less, how do they work?
Just as a follow up on your comments on GLP's, I see a nurse practitioner who is a bodybuilder. She weighs 130 pounds and is 5'4". She's meticulous to the milligram of what she eats and how she exercises
As she gets close to a competition, she has a functional doctor that will let her use a low-dose of tirzepatide. Eating exactly the same amount measured, doing the exact same amount of exercise and she can takeoff 7 pounds prior to the competition. She was absolutely convinced this last year that the drug directly burns fat. I realize it's the study with an N of 1 - but given the person my guess it's as accurate as anything else. I just don't understand how they have all this so backward. And sadly, my colleagues just don't have the time
Hall made it apparent through his Twitter likes in 2024 that he thinks the Ludwig re-analysis is what should be retracted, not his paper. He seems a bit dramatic from an outsider's perspective, as if he takes all criticism personally. I suppose it's only human nature to become obstinate and defensive when backed into a corner. I can't think of an example in any science when a leading researcher humbly bent the knee to a feuding school of thought. This is one limitation of your boxing analogy, because defeated boxers will *sometimes* humbly acknowledge error and defeat.
I agree with you about Hall, or at least he certainly sees this situation differently than I do. Again, I grew up as a journalist in the physics world, watching physistists in conferences and symposia and even group meetings doing what they were suppose to do, which is ripping their colleague's ideas to shreds. The reason you present a paper in that world or give a talk is not convince your colleagues you're right but to get their criticism so you can understand where's you're screwing up. I have NEVER seen that kind of feedback in an obesity or nutrition meeting, online or elsewhere. And I think it's absolutely necessary for a functioning science. Because it's almost totally absent in nutrition/chronic disease research it's easier for someone to take it personally when it suddenly appears and is pointed at them.
Re the boxer metaphor, the corner throws in the towel because they don't want their boxer to continue taking a beating. One of the unfortunate situations in the Hall scenario is that he is very much not taking any beating. (In physics, he'd have been ridiculed for this kind of mistake and his work would never be taken seriously again.) Rather, so long as his conclusions are in line with the conventional wisdom, he's rewarded for the work. Look at the last post and the conversation I described that I had with Marion Nestle who called Hall's recent retirement a "national tragedy." And then the journalists look to Marion as a voice of nutritional authority, and the same uncritical bizarrely-rewarding attitude permeates the entire field.
how can you combine the two seperate diets together??!? and then to average such wildly different results? it appears to be willful manipulation to me.
Kevin Hall's Diet Order study brought to mind Penelope J. Greene's 2003 12-week study involving low-carb and low-fat diets. Greene's study never got published in a journal. However, there are at least two articles available. One of them says, "Many critics said that the results of the experiment violate the laws of thermodynamics, since the test subjects in certain groups lost more weight than their caloric intake. The experiment’s results also challenge classical nutrition theory, which mandates that the intake of calories should equal the amount of calories expended." https://www.thecrimson.com/article/2003/10/15/harvard-study-backs-low-carbohydrate-diet-a/
The other says, "Participants in all three groups lost weight, Greene said, with the low-fat group losing an average of 17 pounds and the low-carbohydrate group that ate the same number of calories losing 23 pounds. The biggest surprise, however, was that the low-carbohydrate dieters eating extra calories lost more than those on the low-fat diet. Participants in that low-carbohydrate group lost an average of 20 pounds." https://news.harvard.edu/gazette/story/2003/10/low-carb-more-effective-than-low-fat/
What's really odd is that epidemiologists typically ignore adipose tissue arachidonic acid. An 'adipose tissue arachidonic acid metabolic syndrome' web search brings up an AI Overview that says, "Elevated levels of arachidonic acid (AA) in adipose tissue are associated with an increased risk of the metabolic syndrome, particularly in individuals with obesity and non-alcoholic fatty liver disease (NAFLD). AA, a polyunsaturated fatty acid, plays a role in inflammation and metabolic processes, and its accumulation in adipose tissue may contribute to the development of metabolic dysregulation. " https://pmc.ncbi.nlm.nih.gov/articles/PMC2730166/
More on Greene's research. https://www.cbsnews.com/news/low-carb-dieters-can-eat-more/
Please explain something. Who is responsible for retracting papers? The journal, or the author? If it can be either one, how is this decided? Logic dictates that if Hall’s studies are flawed and unethical, the publishing journals should retract them. Why is this not happening?
In this case, as I understand it, the journal has said that the author is ultimately responsible. If fraud was involved, the journal might insist, but no one is talking about fraud or misconduct here. This is just bad science and so it's apparently up to Hall to decide. If he was still at NIH, NIH might insist, I imagine (speculating wildly here) but he's not.
Ernie, your last point is a good point. How do you know? This leads to the kind of discussion I've had with one of my colleagues/allies who I think is a terrific scientist. I'll say to him, "A large number of otherwise very smart people think we're just wrong, how do we know we're not?" And he'll respond, a bit in jest but not entirely, "because we're not." And then I'll say, but isn't it the defining characteristic of a quack that they are sure they're right? And now we're back to where we started.