Reader Comments
Post a new comment on this article
Post Your Discussion Comment
Please follow our guidelines for comments and review our competing interests policy. Comments that do not conform to our guidelines will be promptly removed and the user account disabled. The following must be avoided:
- Remarks that could be interpreted as allegations of misconduct
- Unsupported assertions or statements
- Inflammatory or insulting language
Thank You!
Thank you for taking the time to flag this posting; we review flagged postings on a regular basis.
closeFormal comment on Validity of U.S. Nutritional surveillance: national health and nutrition examination survey caloric energy intake data, 1971-2010
Posted by wmurphyrd on 14 Nov 2013 at 20:13 GMT
Dear Dr. Johannsen,
We would like to congratulate Drs. Archer, Hand, and Blair on their recent publication, “Validity of U.S. Nutritional surveillance: national health and nutrition examination survey caloric energy intake data, 1971-2010.” The re-evaluation of data sets that serve as the foundation for research and policy is an important pursuit, and reminders that familiarity and convenience do not equate to validity and reliability are often needed. We would also like to thank the authors for bringing to light a similar and no less important consideration of the tools that are used in the field of nutrition research.
In the present study, established regression equations were used to estimate basal metabolic rate (BMR) and maintenance total energy expenditure (TEE) from anthropometric data gathered in the National Health and Nutrition Examination Survey (NHANES) studies and compared these results to energy intake data from the same studies with the intent of evaluating the accuracy of the 24-hour recall method used to assess dietary intake. When using these measures, one must consider both their origins and their limitations, as inappropriate applications would have significant bearing on the validity of any conclusions drawn.
As BMR is difficult to measure directly[1], this term is used instead to describe the estimation of such via indirect calorimetry, the application of biochemical stoichiometry to measurements of respiratory gas exchange and nitrogen balance[2]. This application relies upon assumptions regarding the metabolic state of the subject[1] that, in practice, require that individuals be in a state of caloric equilibrium without changes in lean or fatty tissue mass. TEE is also used to describe an approximation thereof, in this case via the doubly labeled water (DLW) method which estimates carbon dioxide (CO2) production in free-living subjects with the aid of stable isotope markers. The stoichiometric equations used in BMR are then applied with this measure of CO2 production and an estimation of oxygen consumption derived from dietary intake[3]. As it builds from the previously described indirect calorimetry method, TEE carries the assumptions of the previous method as well as the additional assumptions of stable hydration status and accurate assessment of dietary intake, as wells as a further reliance upon caloric equilibrium to interpret the dietary intake[4].
In the creation of the estimating equations used in the present study, the included data sets were selected in both cases to ensure that the above discussed assumptions were met, including the essential requirement for caloric equilibrium[5,6]. By using these equations, the present article implies the above assumptions about the individuals whose data is analyzed, as well as additional assumptions made directly by the analyses performed: the caloric equilibrium assumption is made once again when equating dietary intake to TEE, the implausibility of BMR:TEE ratios lesser than 1.35 or greater than 2.40, and an unsubstantiated mean physical activity level. Here it would seem that the reminder is also necessary in relation to the use and application of energy expenditure estimations: familiarity and convenience do not equate to validity and reliability.
The caloric equilibrium assumption has implications on multiple aspects of the study, as any individuals restricting caloric intake or losing weight would not only have dietary intakes that would appear to suggest underreporting or implausible intakes but also BMR and TEE that cannot be accurately estimated by the methods used. We respectfully acknowledge that the implausible BMR:TEE ratio designations were not defined by the present authors yet find their use is alongside TEE equations based upon data sets that cast doubt upon those designations, with greater than 10% of the subjects included having an observed ratio of indirect calorimetry BMR to DLW TEE that falls outside the defined plausible range[7], to be paradoxical. Finally the assumption that the average physical activity level was “low active” (equivalent to walking 2.2 mi/day at 3-4 mi/hour for a 70kg individual[6]) leaves the possibility that TEE was significantly overestimated, particularly when considering that evidence is available regarding the physical activity levels of NHANES participants and it indicates that as many as 95% of adult participants may belong instead to the sedentary category[8].
Therefore, when discrepancies appear between reported intake and estimated expenditure, it is not necessarily correct to assume that the fault lies with the dietary recall methodology without first considering of the applicability of the energy estimation methods used and the potential that any of the several implied and direct assumptions were not met. For the present study, potential assumption violations that could plausibly explain the observed discrepancies do exist, and the unequivocal attribution of fault to dietary intake methodology is inappropriate.
With respect to the accuracy of dietary recalls, the present study aptly describes a body of evidence that consistently finds discrepancies between reported intake and TEE as estimated via the DLW method. Even within the confines of stable weight status, this comparison is dubious since, as described above, the accuracy of the DLW method depends upon the accuracy of dietary assessment to determine where, in its 30% range[3,4], the energy yield of the measured CO2 lies. When a study uses this comparison to conclude that the dietary recall is inaccurate, the TEE used must then have been inaccurate as well, and the comparison loses meaning.
Conversely, when dietary recalls are compared to what they actually purport to measure, the quantities of foods consumed, the results are different. A sampling of studies comparing dietary recalls with observed intakes reveals overall mean differences ranging from 3-8%[9–11], and these differences were not large enough to warrant rejection of the null hypothesis, although significant differences were observed in some sub-group analyses. Furthermore, criticisms of the accuracy of a single 24-hour recall from NHANES, such as in the present study, are attacks upon a straw man, as daily variations in intake are well known [6,12], analytical methodology to estimate usual intake from two separate recalls has been developed[13], the National Center for Health Statistics provides instruction on the use of these methods[14], and recent analyses from NHANES data utilize this methodology[15]. Considering the concerns regarding the comparative standards used and the dissimilarity between the data used and actual applications of NHANES data, any conclusions drawn by the present study about the validity of U.S. national surveillance are tenuous at best.
William J Murphy, MS, RDN
Rosa K Hand, MS, RDN
Alison L Steiber, PhD, RDN
1. Ferrannini E (1988) The theoretical bases of indirect calorimetry: a review. Metabolism 37: 287–301.
2. WEIR JB (1949) New methods for calculating metabolic rate with special reference to protein metabolism. J Physiol 109: 1–9.
3. Schoeller DA (1988) Measurement of energy expenditure in free-living humans by using doubly labeled water. J Nutr 118: 1278–1289.
4. Schoeller DA (1983) Energy expenditure from doubly labeled water: some fundamental considerations in humans. Am J Clin Nutr 38: 999–1005.
5. Schofield C (1985) An annotated bibliography of source material for basal metabolic rate data. Hum Nutr Clin Nutr 39 Suppl 1: 42–91.
6. Panel on Macronutrients, Subcommittees on Upper Reference Levels of Nutrients and Interpretation and Uses of Dietary Reference Intakes, Standing Committee on the Scientific Evaluation of Dietary Reference Intakes (2005) Chapter 5: Energy. Dietary Reference Intakes for Energy, Carbohydrate, Fiber, Fat, Fatty Acids, Cholesterol, Protein, and Amino Acids (Macronutrients). Washington, DC: The National Academies Press. pp. 107–264. Available: http://www.nap.edu/openbo....
7. Panel on Macronutrients, Subcommittees on Upper Reference Levels of Nutrients and Interpretation and Uses of Dietary Reference Intakes, Standing Committee on the Scientific Evaluation of Dietary Reference Intakes (2005) Appendix I: Doubly Labeled Water Data Used to Predict Energy Expenditure. Dietary Reference Intakes for Energy, Carbohydrate, Fiber, Fat, Fatty Acids, Cholesterol, Protein, and Amino Acids (Macronutrients). Washington, DC: The National Academies Press. pp. 1104–1202. Available: http://www.nap.edu/openbo....
8. Troiano RP, Berrigan D, Dodd KW, Mâsse LC, Tilert T, et al. (2008) Physical activity in the United States measured by accelerometer. Med Sci Sports Exerc 40: 181–188. doi:10.1249/mss.0b013e31815a51b3.
9. Karvetti RL, Knuts LR (1985) Validity of the 24-hour dietary recall. J Am Diet Assoc 85: 1437–1442.
10. Jonnalagadda SS, Mitchell DC, Smiciklas-Wright H, Meaker KB, Van Heel N, et al. (2000) Accuracy of energy intake data estimated by a multiple-pass, 24-hour dietary recall technique. J Am Diet Assoc 100: 303–308.
11. Conway JM, Ingwersen L a, Moshfegh AJ (2004) Accuracy of dietary recall using the USDA five-step multiple-pass method in men: an observational validation study. J Am Diet Assoc 104: 595–603. doi:10.1016/j.jada.2004.01.007.
12. Beaton GH (1994) Approaches to analysis of dietary data: relationship between planned analyses and choice of methodology. Am J Clin Nutr 59: 253S–261S.
13. Beaton GH, Burema J, Ritenbaugh C (1997) Errors in the interpretation of dietary assessments. Am J Clin Nutr 65: 1100S–1107S.
14. CDC/National Center for Health Statistics (2011) Modeling Usual Intake Using Dietary Recall Data. NHANES Dietary Web Tutorial. Available: http://www.cdc.gov/nchs/t.... Accessed 11 August 2013.
15. Yeung LF, Cogswell ME, Carriquiry AL, Bailey LB, Pfeiffer CM, et al. (2011) Contributions of enriched cereal-grain products, ready-to-eat cereals, and supplements to folic acid and vitamin B-12 usual intake and folate and vitamin B-12 status in US children: National Health and Nutrition Examination Survey (NHANES), 2003-2006. Am J Clin Nutr 93: 172–185. doi:10.3945/ajcn.2010.30127.
RE: Formal comment on Validity of U.S. Nutritional surveillance: national health and nutrition examination survey caloric energy intake data, 1971-2010
edwardarcher replied to wmurphyrd on 21 Nov 2013 at 00:07 GMT
Dear Dr. Johannsen,
We would like to thank the staff members of the Research, International, and Scientific Affairs department of the Academy of Nutrition and Dietetics (A.N.D.) for their comments and critique, and for offering us the opportunity to continue the scientific social discourse that we hope our current and future work generate.
Scientific Measurement vs. Data Collection
I begin by addressing what the A.N.D. staff members state dietary recalls “actually purport to measure” because it emphasizes the profound dissonance between basic scientific measurement principles and the methods employed in nutritional epidemiology.
Dietary recalls (e.g., 24HR, FFQ) are not a measurement protocol. The data derived from dietary surveys are the a priori numeric values assigned by the researcher to the respondent’s retrospective perceptions of eating behavior. In other words, nutrition researchers designate numeric caloric values from unvalidated and incomplete nutrient databases [1-6] to be applied to what the survey respondents are willing and able to report in relation to what they think (or want the researcher to think) [7] they consumed during the study period.
Methodological Critique
The major critique by the A.N.D. staff members was the validity of our use of BMR predictive equations to arrive at rEI/BMR values. First, the method we employed was validated specifically for the evaluation of reporting bias in dietary surveys [13,14]. Second, the Journal of the Academy of Nutrition and Dietetics and its predecessor (i.e., the Journal of The American Dietetic Association) have published numerous studies on the validation and comparisons of predictive equations [15,16], numerous studies that use predictive equations [17,18], and even reviews that recommend the use of the method we employed in our study [19]. As such, this critique lacks substance.
With respect to the requirement of accurately measuring food consumption to arrive at a valid RQ for use with doubly labeled water-derived TEE comparisons with rEI, the A.N.D. staff members state “When a study uses this comparison to conclude that the dietary recall is inaccurate, the TEE used must then have been inaccurate as well, and the comparison loses meaning.” This statement was demonstrated to be false decades ago. The error from the Weir equation for calculation of metabolic rate in the estimation of DLW-derived TEE when the RQ varies is ~1%. As Mansell & Macdonald (1990) unequivocally state “[Metabolic rate] may therefore be estimated…without reference to the proportions of carbohydrate, fat, and protein consumed” [21]. As such, this critique lacks empirical support.
The critique of our use of the “low-active” physical activity (PA) value rather than the “sedentary” PA value for use in the Institute of Medicine equations is potentially valid. Nevertheless, the average American takes >6,000 steps per day [22] [23], the equivalent of >3 miles of walking each day. As the A.N.D. staff acknowledge, the ‘low-active” is “equivalent to walking 2.2 mi/day at 3-4 mi/hour for a 70kg individual.” As such, the average American may be considered "low-active." Nonetheless, as a ‘sensitivity analysis’ we performed our original examination using both the “sedentary” and “low-active” PA values. While we did not present results from both protocols, each indicates substantial under-reporting. For example, the mean disparity values for obese women and men using the “sedentary” PA values were -387 and -353 kcals per day respectively, while over-reporting increased significantly in men. The consistency of our results despite substantial changes in the PA value in the IOM equations clearly demonstrates the robustness of our method and conclusion (i.e., the NHANES dietary data are invalid due to fatally flawed data collection).
The final critique revolves around the “caloric equilibrium assumption.” We addressed this by conducting our original analyses with and without individuals reporting the various diets and intakes categorized by the NHANES. The inclusion and exclusion of these respondents had no significant effect on the results or conclusions. Additionally, in the NHANES surveys that included data from a second dietary recall, the level of underreporting on the second day of recall increased significantly relative to the first day. Therefore, the proposition that the inclusion of multiple dietary recalls would improve the estimates is patently false. More importantly, the range of rEI/BMR values that were considered plausible in our study (i.e., >1.35 & < 2.40) was substantially greater than the “study specific cutoffs” recommended by the validation research of the method we employed [13,14]. When the study specific cutoff was used, misreporting increased significantly.
Given these facts, our method is consistent with a large body of literature [20,24-26], (inclusive of work published by the official journal of the Academy [19]), has been empirically validated [13,14], is demonstrably robust, and our results and conclusions are conservative.
It has been over two decades since Schoeller unequivocally demonstrated the lack of validity of dietary self-reports [27,28]. Given that these pseudo-quantitative data serve as the foundation for research and public health policy, one significant question remains to be answered: When will the field of nutrition epidemiology begin to value valid scientific measurement protocols over familiarity and expedience?
Edward Archer, PhD, MS
References
1. Phillips KM, Patterson KY, Rasor AS, Exler J, Haytowitz DB, et al. (2006) Quality-control materials in the USDA National Food and Nutrient Analysis Program (NFNAP). Anal Bioanal Chem 384: 1341-1355.
2. Weihrauch JL. Problems in Energy Calculations. In: USDA, editor; 1992; Baltimore, Maryland. Human Nutrition Informatuion Service, HNIS.
3. Ahuja JKC, Goldman JD, Perloff BP (2006) The effect of improved food composition data on intake estimates in the United States of America. Journal of Food Composition and Analysis 19, Supplement: S7-S13.
4. Ahuja JKC, Perloff BP (2008) Quality control procedures for the USDA Food and Nutrient Database for Dietary Studies nutrient values. Journal of Food Composition and Analysis 21, Supplement: S119-S124.
5. Guenther PM, Perloff BP, Vizioli TL, Jr. (1994) Separating fact from artifact in changes in nutrient intake over time. J Am Diet Assoc 94: 270-275.
6. Perloff B, Ahuja JK (1998) Development and maintenance of nutrient databases for national dietary surveys. Public Health Rev 26: 43-47.
7. Hebert JR, Clemow L, Pbert L, Ockene IS, Ockene JK (1995) Social desirability bias in dietary self-report may compromise the validity of dietary intake measures. Int J Epidemiol 24: 389-398.
8. Rimm EB, Stampfer MJ, Ascherio A, Giovannucci E, Colditz GA, et al. (1993) Vitamin E consumption and the risk of coronary heart disease in men. N Engl J Med 328: 1450-1456.
9. Stampfer MJ, Hennekens CH, Manson JE, Colditz GA, Rosner B, et al. (1993) Vitamin E consumption and the risk of coronary disease in women. N Engl J Med 328: 1444-1449.
10. Yusuf S, Dagenais G, Pogue J, Bosch J, Sleight P (2000) Vitamin E supplementation and cardiovascular events in high-risk patients. The Heart Outcomes Prevention Evaluation Study Investigators. N Engl J Med 342: 154-160.
11. Tatsioni A, Bonitsis NG, Ioannidis JP (2007) Persistence of contradicted claims in the literature. JAMA 298: 2517-2526.
12. Schoeller DA, Thomas D, Archer E, Heymsfield SB, Blair SN, et al. (2013) Self-report-based estimates of energy intake offer an inadequate basis for scientific conclusions. Am J Clin Nutr 97: 1413-1415.
13. Black AE (2000) Critical evaluation of energy intake using the Goldberg cut-off for energy intake:basal metabolic rate. A practical guide to its calculation, use and limitations. Int J Obes Relat Metab Disord 24: 1119-1130.
14. Black AE (2000) The sensitivity and specificity of the Goldberg cut-off for EI:BMR for identifying diet reports of poor validity. Eur J Clin Nutr 54: 395-404.
15. Frankenfield D, Roth-Yousey L, Compher C (2005) Comparison of predictive equations for resting metabolic rate in healthy nonobese and obese adults: a systematic review. J Am Diet Assoc 105: 775-789.
16. Frankenfield DC, Rowe WA, Smith JS, Cooney RN (2003) Validation of several established equations for resting metabolic rate in obese and nonobese people. J Am Diet Assoc 103: 1152-1159.
17. Anderson EJ, Sylvia LG, Lynch M, Sonnenberg L, Lee H, et al. (2013) Comparison of Energy Assessment Methods in Overweight Individuals. J Acad Nutr Diet.
18. Kyle UG, Jaimon N, Coss-Bu JA (2012) Nutrition Support in Critically Ill Children: Underdelivery of Energy and Protein Compared with Current Recommendations. Journal of the Academy of Nutrition and Dietetics 112: 1987-1992.
19. Black AE, Cole TJ (2001) Biased Over- Or Under-Reporting is Characteristic of Individuals Whether Over Time or by Different Assessment Methods. Journal of the American Dietetic Association 101: 70-80.
20. Livingstone MBE, Black AE (2003) Markers of the Validity of Reported Energy Intake. pp. 895S-920S.
21. Mansell PI, Macdonald IA (1990) Reappraisal of the Weir equation for calculation of metabolic rate. Am J Physiol 258: R1347-1354.
22. Tudor-Locke C, Ham SA, Macera CA, Ainsworth BE, Kirtland KA, et al. (2004) Descriptive epidemiology of pedometer-determined physical activity. Med Sci Sports Exerc 36: 1567-1573.
23. Tudor-Locke C, Johnson WD, Katzmarzyk PT (2009) Accelerometer-determined steps per day in US adults. Med Sci Sports Exerc 41: 1384-1391.
24. McGowan MJ, Harrington KE, Kiely M, Robson PJ, Livingstone MB, et al. (2001) An evaluation of energy intakes and the ratio of energy intake to estimated basal metabolic rate (EI/BMRest) in the North/South Ireland Food Consumption Survey. Public Health Nutr 4: 1043-1050.
25. Johansson L, Solvoll K, Bjorneboe GE, Drevon CA (1998) Under- and overreporting of energy intake related to weight status and lifestyle in a nationwide sample. Am J Clin Nutr 68: 266-274.
26. Briefel RR, Sempos CT, McDowell MA, Chien S, Alaimo K (1997) Dietary methods research in the third National Health and Nutrition Examination Survey: underreporting of energy intake. Am J Clin Nutr 65: 1203S-1209S.
27. Schoeller DA (1990) How accurate is self-reported dietary energy intake? Nutr Rev 48: 373-379.
28. Schoeller DA (1995) Limitations in the assessment of dietary energy intake by self-report. Metabolism 44: 18-22.