Saturday, 10 November 2012

Why Nutrition Science is so Bad

When I first started looking into nutritional science, I was shocked at how badly it was done.    To be fair, nutrition science if very difficult.  There are so many lifestyle and nutrition choices that can have a positive or negative influence on health, that it is hard to isolate the effect of any one food or group of foods.  There are two main types of studies; Observational (Epidemiological) studies and Controlled Experiments.  Tom Naughton has a really good video called “Science for smart People”  that explains the difference in an informative and humorous way.

The gold standard of science is the controlled experiment.  In a controlled experiment, you hold all variables constant except for the one variable that you are studying.  For example, if you wanted to study the effect of a fertilizer on growing plants, you would plant two groups of the same seeds.  You would use the same soil, same size pot; put the pots in the same area where the temperature and light exposure are the same, and give them the same amount of water.  You would put fertilizer in one but not the other.  If the plants in the fertilized pot grow faster and larger, then this would support your hypothesis that fertilizer helps plants grow.  

It is very hard to do this in a nutritional study.  The only way to be sure of the quality and quantity of food consumed, would be to lock people in a metabolic ward where food could be controlled.  You would have to weigh and measure all the food served, as well as all the food that wasn’t eaten.  Since most people are interested in being healthy for the rest of their life, and not just for the next 6 months, these studies would have to last 10 to 20 years to be really meaningful.  Not too many people would volunteer to be in a study that would lock them in a metabolic ward for 20 years!  (well, maybe Al Bundy would volunteer).  Even if you could do a study like this, it might not be relevant to the real world.  In a metabolic ward you have to stick to the diet; there is no other food choices.  In the real world, there are food choices on almost every street corner.  People have to be able to stay on a nutritional lifestyle long term for it to be useful. 

Very few controlled studies are done in nutritional science.  Almost all the studies we hear about are observational studies.  In these studies information is gathered on what people eat, usually with food frequency questionnaires.  These people are followed for a number of years and their health outcomes are observed.  Correlations are made between the food people ate and their health outcomes.  These types of studies have many limitations, the biggest being that they cannot provide any information on cause and effect.  Observational studies are useful for coming up with a new hypothesis, but then the hypothesis has to be tested in a Controlled Experiment.  A couple hypothetical studies from Tom Naughton's video might explain why this is.

If we were to do a study of the BMI of marathon runners compared to the BMI of the average person, you might find that running marathons is correlated with a lower BMI.  The conclusion of the study could say that running in marathons was linked, or correlated, or associated with a lower BMI.  They could not say that running in marathons causes you to become lean and have a lower BMI, even though most people hearing about this study would think that running marathons does cause people to have a lower BMI.  It conforms with our preconceived notions about exercise and weight, so we assume that causation is proved by the study.   If we look at another hypothetical study, it will illustrate why this is not the case.  If we did a similar study but used professional basketball players and height, we would find that playing basketball is correlated with being taller than the average person.  Does that mean that playing basketball causes you to grow taller?  If you are 5’6” and want to be 6’ tall, can you play basketball for a few years and expect to  grow?  Of course not!  Playing basketball doesn’t cause you to grow taller; it’s just that if you are tall you are more likely to play professional basketball.  The same logic can be used in the other study: Running marathons might not make you lean, it’s just that lean people are more likely to run marathons.  

When something in a study (A) is correlated with something else (B) it is easy to jump to the false conclusion that A is causing B.  A may or may not be causing B, as there is no way to know just from an observational study.  A may be causing B.  B may be causing A.  A third variable, C may be causing A and B.  This third variable C is what is called a confounding variable.  An example of this is that ice cream sales in Florida are correlated with shark attacks.  Does this mean that eating ice cream causes shark attacks?  Maybe the sharks like the ice cream dripping down your chin so they are more likely to attack.  Of course this is silly.  When it is hot, people eat more ice cream and they also go swimming more, which leads to more shark attacks.    Observational studies are full of confounding variables.  There are two main groups of people that influence the outcomes of nutritional studies.  There are people who are very health conscious and do whatever they can to be healthy, and those that do not care about their health and make food and lifestyle choices that are purely based on giving them pleasure.  People who are health conscious tend to have better health than people who are not health conscious.  They follow the health advice that has been generally given the last 40 years; they smoke less, drink less alcohol, exercise more, take vitamins, eat less calories, eat less sugar, eat less refined processed foods, and eat more vegetables.  Any one of these variables could be contributing to their good health, but from an observational study you can’t tell which one.  You would need a controlled experiment to do that.   

Another problem with observational nutritional studies is that the data collected is not very reliable.  It is usually collected using food frequency questionnaires.  In these studies people are asked to recall what they ate in the last day, month, year, or even four years.  You can find an example of a questionnaire here.  Most people can’t remember what they ate last Tuesday, let alone what they ate three years ago.  People who see themselves as healthy tend to overestimate foods they consider healthy and underestimate foods they consider unhealthy. Who wants to admit that their breakfast consisted of twinkies and oreos?  If the data that the study is based on is not accurate, then how useful is the study?

Some scientists can have such a strong belief in what the outcome of their study will be that they become biased.  Since there are so many confounding variables, you can make the outcome of a study say pretty much whatever you want it to say.  Most studies will try and account for these variables, but it is almost impossible to know exactly how much of a part each one played in someone’s health.  Some researchers will use a third variable to link two items when there is no direct link.  A good example of this is saturated fat, cholesterol and heart disease.  Many studies (such as Dr. Jolliffe's Anti-Coronary Club experiment mentioned here) will claim to show that saturated fat causes heart disease even if the data in their data shows that people who ate more saturated fat had a lower incidence of heart disease.  They do this by saying that saturated fat intake was associated with higher cholesterol, which is claimed to be a marker for heart disease. (Even though it has never been proven that high cholesterol causes heart disease)

This is why we get so many mixed messages from the so called “nutrition experts”.  The way Observational studies are reported, it gives the idea that they determine cause and effect when they really only show correlation. In one group of people a certain food may be correlated with high cancer rates, in another group they may be correlated with low cancer rates.  The truth may be that the food has NO impact on cancer rates.  As long as we remember the limitations of the study we won’t get sucked into these false assumptions.  So the next time you hear about the latest study telling you to stay away from a certain food, or that another food is a miracle cure, remember that 99% of these studies don’t actually prove anything.  

It is time to stop funding these types of observational studies.  How many studies do we need that give sensational headlines but do not add to our knowledge.  We have enough hypotheses about health and nutrition.  We need to start doing controlled experiments to find out which hypotheses about diet and nutrition result in healthy outcomes.  With obesity related diseases and health care costs skyrocketing, we need to find these answers now. 


No comments:

Post a Comment