Is Reasoning Without Logic Viable? An Evaluation of Large Language Models using the NeuBAROCO dataset

Conférence de Koji Mineshima (Keio University)

"Is Reasoning Without Logic Viable? An Evaluation of Large Language Models using the NeuBAROCO dataset"

Résumé :
We examine the logical reasoning abilities of current large language models (LLMs) in natural language, focusing on their similarity to human reasoning biases. We employ syllogisms, a key form of deductive reasoning studied in cognitive science, and introduce the NeuBAROCO dataset, which was originally developed for psychological experiments to evaluate human reasoning and contains syllogistic reasoning problems in English and Japanese. Our experiments show that LLMs exhibit similar biases to humans, especially in reasoning tasks where the premises neither entail nor contradict the conclusion. We also report on the efficacy of Chain-of-Thought prompting, which involves translating syllogisms into logical expressions and explaining the reasoning process, to highlight areas where LLMs can improve. (This is joint work with Hirohiko Abe, Risako Ando, Takanobu Morishita, Mitsuhiro Okada and Kentaro Ozeki)