This paper explores whether modern AI models like OpenAI’s o1 can handle complex tasks that require planning—crucial for advanced problem-solving.
Previous large language models (LLMs), such as GPT-4o or Claude Sonnet 3.5, excel at language understanding but struggle with multi-step reasoning.
The study shows that OpenAI's o1 represents a significant leap forward, solving nearly all tasks in the Blocksworld test with 97.8% accuracy, compared to 62.6% for LLaMA 3.1 405B (zero-shot) and 57.6% for Claude 3.5 Sonnet (one-shot).
In the more challenging Mystery Blocksworld, where tasks are obfuscated to make them harder to solve, o1-preview achieved 52.8% in zero-shot, far surpassing previous models like GPT-4o, which managed only 4.3% in one-shot, and LLaMA 3.1 405B, which managed only 0.8% in zero-shot.
These results indicate that o1-preview brings a substantial (10-40 fold) improvement in reasoning capabilities.
This was only just the preview version of o1 - we can only imagine the capabilities of the actual o1 model, let alone when it is multimodal with tools.
*Paper summary partially created in collaboration with GPT-4o
Ville Johannes Pajala
@VJPajala_1
Building Intelligent Automation | Expanding expertise in AI | ChatGPT Ninja