SokoBench: Evaluating Long-Horizon Planning and Reasoning in Large Language Models

SokoBench Benchmark Evaluates Long-Horizon Planning in Large Language Models

Recent research has revealed significant insights into the long-horizon planning capabilities of state-of-the-art Large Reasoning Models (LRMs). A new benchmark, named SokoBench, assesses these models through a series of simplified Sokoban puzzles designed to isolate the complexities of long-term planning.

This evaluation highlights a marked decline in the performance of LRMs when tasked with solving problems that require more than 25 moves, indicating a fundamental limitation in the models' capacity for forward planning.

Benchmark Findings

The SokoBench was developed to focus on planning challenges, allowing researchers to examine the inherent capabilities of LRMs in long-horizon reasoning tasks. Results indicate a consistent degradation in performance as the length of the required solution path increases.

• Models demonstrated significant difficulty with puzzles necessitating over 25 moves.
• The performance decline suggests limitations in how LRMs can project future states.

The research team incorporated Planning Domain Definition Language (PDDL) parsing and solving tools into the LRMs, resulting in modest performance improvements, but not fully compensating for the observed deficiencies.