Artificial intelligence is a promising tool in disseminating nutrition information, study finds

A recently published study JAMA Network Open investigated the accuracy and reliability of nutritional information provided by two versions of Chat Generative Pre-trained Transformer (ChatGPT) chatbots.

The findings suggest that although chatbots cannot replace nutritionists, they can improve communication between healthcare professionals and patients if further developed and strengthened.

To work: Consistency and Accuracy of AI in Providing Nutrition Information. Image Credit: Iryna Imago/Shutterstock.com

Background

Today, many people depend on the internet to access health, medicine, food and nutrition information. But research shows that almost half of online nutrition information is low quality or inaccurate.

Artificial intelligence (AI) chatbots have the potential to streamline the way users navigate the vast array of publicly available scientific information by providing conversational, easy-to-understand explanations of complex topics.

Previous studies have evaluated how well chatbots can disseminate medical information, but their reliability in providing nutritional information remains relatively unexplored.

About the study

In this cross-sectional study, researchers followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline. They evaluated the accuracy of information provided by ChatGPT-3.5 and ChatGPT-4 in two languages ​​(Traditional Chinese and English) on macronutrients (proteins, carbohydrates, and fats) and energy content of 222 foods.

They provided a command that asked the chatbot to create a table containing the nutritional profile of each food in its uncooked form. This search was conducted in September-October 2023.

Each search was performed five times to assess consistency; The coefficient of variation (CV) was calculated for each food over these five measurements.

The accuracy of the chatbot’s responses was evaluated by cross-referencing its responses with recommendations from nutritionists, according to the food composition database maintained by Taiwan’s Food and Drug Administration.

The answer was considered correct if the chatbot’s energy (in kilocalories) or macronutrient (in grams) estimate was within 10% to 20% of those provided by nutritionists.

The researchers also calculated whether the chatbots’ responses were significantly different from nutritionists’ recommendations and between the two versions of ChatGPT.

Results

There were no significant differences between the estimates provided by chatbots and nutritionists regarding the fat, carbohydrate and energy levels of eight adult menus. But the researchers found that protein predictions differed significantly. Chatbot responses were considered accurate in terms of energy content in 35-48% of the 222 foods included, with a CV below 10%. The newer version, ChatGPT-4, outperformed ChatGPT-3.5 overall but tended to overestimate protein levels.

Results

The study shows that chatbot responses compare well to nutritionists’ advice in certain respects, but can overestimate protein levels and also show high levels of inaccuracy.

As they become widely available, they have the potential to be a convenient tool for people who want to look up macronutrient and energy information on common foods and don’t know which sources to turn to.

However, the authors emphasize that chatbots cannot replace nutritionists; They can improve communication between patients and public health professionals by providing additional resources and simplifying complex medical language into conversational, easy-to-follow terms.

They also note that the foods they included in the study may not be consumed frequently, which may affect the validity of their findings.

AI chatbots cannot provide users with personalized dietary recommendations or precise portion sizes, or create specific diet and nutrition-related guidelines. Moreover, chatbots may not be able to adapt their responses to the user’s region.

Portion sizes and consumption units vary greatly from country to country, as well as by the type of food and the way it is prepared. Chatbots cannot account for significant cultural and geographic differences or provide relevant household units for each consumer.

Arguably the most important limitation is that ChatGPT is a general-purpose chatbot, not specifically trained in dietetics and nutrition.

Since the training dataset is due in September 2021, more recent studies will not be included. Users should not confuse chatbots with search engines, as their responses are a product of the training datasets and the wording of the prompts.

However, given the immense popularity of chatbots and other forms of generative AI, future products will overcome these limitations and provide increasingly accurate, updated, relevant and practical information on diet and nutrition.