Language allows people to communicate their thoughts to each other because each person’s brain responds to the meaning of words in similar ways. In our newly published research, my colleagues and I developed a framework to model the brain activity of speakers as they engage in face-to-face conversations.
We recorded the electrical activity of two people’s brains while they had unscripted conversations. Previous research has shown that when two people converse, their brain activity matches, or aligns, and the degree of neural coupling is associated with better understanding of the speaker’s message.
A neural code refers to specific patterns of brain activity associated with different words in their context. We found that the speakers’ brains aligned on a shared neural code. Importantly, the brain’s neural code resembled the artificial neural code of large language models, or LLMs.
Neural patterns of words
A large language model is a machine learning program that can generate text by predicting which words are more likely to follow others. Large language models are excellent at learning the structure of language, generating human-like text, and carrying on conversations. They can even pass the Turing test, making it difficult for someone to tell whether they are interacting with a machine or a human. Like humans, LLMs learn to speak by reading or listening to text produced by other humans.
By giving LLM a transcript of the speech, we were able to infer its “neural activations,” or how it translated words into numbers as it “read” the text. Then, we correlated the speaker’s brain activity with both LLM’s activations and the listener’s brain activity. We found that LLM’s activations could predict the combined brain activity of the speaker and the listener.
In order to understand each other, people have a common agreement on the rules of grammar and the meanings of words in context. For example, we know to use the past tense of a verb to talk about past actions, such as: “He visited the museum yesterday.” We also intuitively understand that the same word can have different meanings in different situations. For example, the word cold in the sentence “you are as cold as ice” can refer to someone’s body temperature or a personality trait, depending on the context. Because of the complexity and richness of natural language, until the recent success of large language models, we lacked a precise mathematical model to describe it.
Our study found that large language models can predict how linguistic information is encoded in the human brain, providing a new tool for interpreting human brain activity. The similarity between the human brain and the large language model’s linguistic code has allowed us, for the first time, to track how information in the speaker’s brain is encoded into words and transferred word by word to the listener’s brain during face-to-face conversations. For example, we found that brain activity associated with the meaning of a word occurs in the speaker’s brain before the word is pronounced, and that the same activity quickly reappears in the listener’s brain after the word is heard.
Powerful new tool
Our work has provided insights into the neural code for language processing in the human brain and how both humans and machines can use this code to communicate. We found that large language models are better able to predict shared brain activity when compared to different features of language, such as syntax or the order in which words are connected to form phrases and sentences. This is partly due to the LLM’s ability to incorporate the contextual meaning of words and integrate multiple levels of the linguistic hierarchy into a single model: from words to sentences to conceptual understanding. This suggests important similarities between the brain and artificial neural networks.
A key aspect of our research is using daily recordings of natural conversations to ensure that our findings capture real-life brain processing. This is called ecological validity. Unlike experiments where participants are told what to say, we relinquish control of the study and allow participants to speak as naturally as possible. This loss of control makes the data difficult to analyze because each conversation involves two interacting individuals speaking uniquely and spontaneously. Our ability to model neural activity as people engage in everyday conversations testifies to the power of large language models.
Other dimensions
Now that we have developed a framework for assessing the shared neural code between brains during everyday conversations, we are interested in what factors drive or inhibit this coupling. For example, would linguistic coupling increase if a listener better understood the speaker’s intentions? Or perhaps complex language, such as jargon, would reduce neural coupling.
Another factor that might affect linguistic matching might be the relationship between the speakers. For example, you might be able to convey a lot of information in a few words to a good friend but not to a stranger. Or you might be neurally better matched to your political allies than to your opponents. This is because differences in how we use words across groups might make it easier for us to fit in and match with people inside our social groups rather than outside them.
This article is republished from The Conversation, a nonprofit, independent news organization that brings you facts and trusted analysis to help you understand our complex world. By Zaid Zada Princeton University
Read more:
Zaid Zada does not work for, consult, own shares in, or receive funding from any company or organization that would benefit from this article, and has disclosed no affiliations beyond his academic appointment.