Sponsored by Evolution AI: [ Ссылка ]
Abstract: Abstract reasoning is a key ability for an intelligent system. Large language models achieve above-chance performance on abstract reasoning tasks, but exhibit many imperfections. However, human abstract reasoning is also imperfect, and depends on our knowledge and beliefs about the content of the reasoning problem. For example, humans reason much more reliably about logical rules that are grounded in everyday situations than arbitrary rules about abstract attributes.
The training experiences of language models similarly endow them with prior expectations that reflect human knowledge and beliefs. We therefore hypothesized that language models would show human-like content effects on abstract reasoning problems.
We will describe experiments testing this hypothesis across three logical reasoning tasks: natural language inference, judging the logical validity of syllogisms, and the Wason selection task (Wason, 1968). We find that state of the art large language models (with 7 or 70 billion parameters; Hoffman et al., 2022) reflect many of the same patterns observed in humans across these tasks -- like humans, models reason more effectively about believable situations than unrealistic or abstract ones. We will use these findings to reflect on some broader questions about reasoning processes, and comparing language models to humans.
Bio: Andrew Lampinen is a Research Scientist at DeepMind. He focuses on applying ideas from cognitive science to improve or analyze AI systems, especially in reinforcement learning and language. He completed his PhD at Stanford University, and his BA at UC Berkeley. In his spare time, he enjoys rock climbing.
