What's Wrong With NHANES' Data?

NHANES data is the best available - and it's totally unreliable. So what can the Cell Metabolism study teach us about diet and health?

Remember, the study's authors called NHANES 'the only nationally representative dietary survey in the United States.' They used NHANES because there was nothing else.

Well, it's not what one critic calls 'physiologically plausible.'

What that means is that NHANES' data is so far wrong - there's so much 'underreporting' going on -that if most people who reported their eating habits to NHANES were telling the truth, they'd have starved to death long ago, in about a third of cases throughout the history of NHANES.

That's sufficient caloric deficit to lose about half a pound or a pound of body weight a week. If you kept that up for eighteen years, the duration of the survey, you'd lose somewhere around 450 pounds of body weight in that period: equivalent to about three good-sized people.

Respondents who were quite large to begin with were reporting even higher discrepancies: obese respondents typically underreported caloric intake by about 716 calories a day for men and 816 calories a day for women. If they'd kept that up for 18 years they'd have lost about 900 pounds of body weight. Obviously that's not happening.

There are more limitations to this study, such as dietary intake assessment, which in large population studies can be challenging. People may not accurately report their food intake, leading to potential inaccuracies in the data.

Also, NHANES study struggle to account for all confounding variables. These are factors other than the one being studied that might influence the outcome. For example, if high-protein diets are associated with early death, it might be due to other factors common in people who choose such diets, like smoking or lack of exercise.

And what about how much protein they were eating?

The Cell Metabolism authors described a 'high-protein diet' - but what did they mean by that? 'In this study, on average people ate 51% of their calories in the form of carbohydrates, 33% as fat and 16% as protein (11% animal protein). This is likely to be higher in fat and lower in carbohydrates than that recommended on the "Eatwell Plate" which shows the relative proportions of food that we should aim to eat,' commented NHS Choices in the UK. A diet containing 11% animal protein by calories - about 55g a day for a 2000-calorie diet - isn't really a 'high protein diet.' It's a high carbohydrate, protein-starvation diet. A high-protein diet would see consumption top 25% of calories - well over 1g per pound of bodyweight a day, more than double the figures shown here.

Professor Winkler sums up: 'you've got a correlation between food intake and disease, but the food intake data is so poor that any correlation is suspect. Academics sit in front of a computer screen and do data analysis. But they never question the quality of the primary data. I'm not being rude, but my attitude towards that study is contempt.'

But the mistakes and deceits we find can tell us things too. We know what we can't learn from the Cell Metabolism study: we can't learn that protein causes early death, because in reality we don't know for sure what these people were eating.

What can we learn from it?

First, not to trust surveys and headlines at face value. Exactly how many cigarettes a day a high protein diet is equal to is open to question now, I think. Second, we can learn things about what's really going on in Americans' diets by looking at the known rates of underreportage.

The design of a study can impact its reliability. Some NHANES studies are observational, which means they gather data on a population without intervening. Observational studies can be valuable for generating hypotheses but are not as strong in establishing cause and effect as controlled clinical trials.

If obese people underreport caloric intake more than people of a health weight, where do those extra, unreported calories come from? It's unlikely to be from protein, because protein is regarded as a healthy food. Whether that's right or wrong (it's right, relax), that's the perception. So if people are 'lying' to 'put themselves in the best possible light,' which foods will they underreport - ones they consider healthy or ones they consider bad? Probably we would find that consumption of sugar, junk food and saturated fat had been underreported significantly. If anything we would expect to find lean protein intake and consumption of fruits and vegetables to be significantly overreported as people 'respond normatively,' giving the answers they think ought to be true instead of what actually happened.

See Also: Are You Getting Enough Protein?

In other words, the takeaway from this study is that it's not protein that kills you: it's sugar, which really does cause diabetes, and not having an accurate idea of what you eat. We're not dying of protein overdose; we're dying of self-deception.

If you agree, or you think I'm horribly wrong, or you have a story to tell, get in touch in the comments section: I'd love to hear from you.