By Hannah PressI might have a dental problem, but really I don’t give a nickel or a dam. What am I?
If you said beaver, you’re right. But how did you get there? Did you know that there’s a beaver on the Canadian nickel, or did you piece it together from the references to teeth and dams?
Cohere Labs asked 28 AI models the same riddle as part of a new study testing how well models understand cultures around the world. Sixteen got it right, while others focused on teeth (toothpick, toothache) or water (river, waterfall). One even guessed an earring. The study did not reveal why each model answered the way it did.
If you want to see how you’d fare against the models, try the riddles yourself.
Most tests of AI’s cultural knowledge have focused on facts, like knowing which animal appears on the Canadian nickel. Cohere Labs wanted to see whether models could go a step further and use what they know about a culture to solve riddles. The findings suggest we may need better ways to measure how well AI understands cultural context, along with better ways to train models to account for cultural context in the first place.
It started with one Hungarian man and 100 riddles. Károly Boczka, a former diplomat, wrote riddles about his country and found that models were “nearly perfect at sounding right” but “far less capable of being right.” He brought his idea to Julia Kreutzer, a Staff Research Scientist at Cohere Labs, and the project took off. Over six months, 167 volunteers from 62 communities wrote more than 6,000 riddles about their own cultures, in their own languages.
On the latest Mixture of Experts, Ash Minhas, Technical Content Manager at IBM, argued that AI benchmark scoring needs an upgrade. Traditionally, models earn points for correct answers but get nothing for admitting uncertainty. “The models are incentivized to guess,” he said.
IBM Master Inventor Jessica Nahulan turned the discussion back to cultural context with a timely reference to magic. Drawing a comparison between AI prompts and spells, she pointed out how a few words can carry more meaning than meets the eye. “Every word from culture to culture might have a different interpretation,” she said. So, AI needs to be tested in the languages people use and with their cultural perspectives in mind. Cohere’s community-led approach aimed to do just that.
But one challenge remains. Culture itself is “like a moving goalpost,” Ash said. As cultures evolve, so does the challenge of testing whether AI understands them.
