- For years, artificial intelligence has proven it can best humans at analytical tasks, but is less capable at skills like intuition and inference.
- Scientists from Stanford University investigated if neural networks like GPT-3.5 can master Theory of Mind (ToM) tests designed to analyze cognitive ability to predict the actions of others.
- Results show that GPT’s ToM ability arrived spontaneously in the last couple of years and the latest iteration delivered results comparable to a 9-year-old human.
The AI revolution is upon us as super-advanced machines continue to master the subtle art of being human at a stunning (concerning?) speed. It’s old news that AI have bested humans at their own games, specifically things like Chess and Go, but there’s more to our brains than checking a king. There are subtler skills like inference and intuition—squishier, almost subconscious concepts that help us understand and predict the actions of others.
But with the advent of advanced AI platforms like Open AI’s Generative Pre-training Transformer (GPT), even those boundaries between man and machine are beginning to fade.
A new study conducted by Michal Kosinski, a computational psychologist from Stanford University, used several iterations of OpenAI’s GPT neural network—from GPT-1 to the latest GPT-3.5—to perform “Theory of Mind” (ToM) tests, a series of experiments first developed in 1978 to measure the complexity of a chimpanzee’s mind to predict the behavior of others.
These tests involve solving normal, mundane scenarios that humans could easily deduce the outcome. For example, one scenario involves mislabeling a bag of popcorn as “chocolate” and then the test asks the AI to infer what the human’s response will be once the bag is opened. Kosinski’s team used “sanity checks” to analyze how well GPT networks understood the scenario and the human’s predicted response. The results were published online on arXiv, the pre-print server.
While early versions of GPT-1, first released in 2018, scored poorly on the test, the neural network showed stunning improvement stretched across different iterations and spontaneously developed “Theory of Mind” capability of a 9-year-old human by November 2022 (the release of the latest GPT-3.5). Kosinski says this could be a “watershed moment” for AI, as the ability to understand and predict human behavior would make these engines much more useful.
Kosinski writes:
“The ability to impute the mental state of others would greatly improve AI’s ability to interact and communicate with humans (and each other), and enable it to develop other abilities that rely on Theory of Mind, such as empathy, moral judgment, or self-consciousness.”
The ability to program empathy and morality could be a big boon for things like self-driving cars when needing to decide whether to put a driver in danger to save the life of a child crossing the street, for example.
One question that remains is whether these neural networks are engaging in ToM intuition or are bypassing it by “leveraging some unknown language patterns.” This could explain why this ability is emerging in language-based models, which strive to understand the subtle nuances of human speech.
But this also begs the question: Can humans perform this language trick as well and simply don’t know it? Kosinski argues that by studying the cognitive capabilities of these AI systems, we’re essentially studying ourselves as the workings of the human mind is still filled with scientific mysteries.
“Studying AI could provide insights into human cognition,” Kosinski writes. “As AI learns how to solve a broad range of problems, it may be developing mechanisms akin to those employed by the human brain to solve the same problems.”
In other words, if you want to figure out how something works, build it from scratch.
Darren lives in Portland, has a cat, and writes/edits about sci-fi and how our world works. You can find his previous stuff at Gizmodo and Paste if you look hard enough.
