On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability

Kevin Wang * · Junbo Li * · Neel P. Bhatt * · Yihan Xi
. Qiang Liu . Ufuk Topcu . Atlas Wang .

*Equal contribution and co-first authors

arXiv | Project Page(coming soon) | more examples + evaluations(coming soon)

We evaluated the GPT4 and o1 on planning tasks, highlighting their strength in problem understanding and identifying challenges in spatial reasoning and generalization.

TODO List

We will update the detailed information and share access to more files soon.

• Release detailed experiments evaluation
• Project page
• Release automoation evaluation script (This would take a while)

The File Hierarchy:

OpenAI's o1 Models
  └─results
     └─barman (the domains)
     ...
     └─tyreworld
        └─p_.pddl.prompt (the prompt we used for experiments, including the domain and problem in natural language)
        └─p_.pddl.gpt4 (GPT4 results to the prompt)
        └─p_.pddl.o1-mini (O1-mini results to the prompt)
        └─p_.pddl.o1-preivew(o1-preview results to the prompt)
        └─random.py(only in randomized example, this encode the problem with random symbol)
  └─visual (this would include more visual examples and graphic)
  └─scripts (scripts used to generate files, and update in the future)

The detailed experiment results

Citation

If you find our paper useful or interesting, please consider giving a star ⭐ and citing the following paper 📝.

@misc{wang2024planningabilitiesopenaiso1,
      title={On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability}, 
      author={Kevin Wang and Junbo Li and Neel P. Bhatt and Yihan Xi and Qiang Liu and Ufuk Topcu and Zhangyang Wang},
      year={2024},
      eprint={2409.19924},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2409.19924}, 
}

Acknowledgements

The basic prompts are from llm+p available at this GitHub repository. We thank all the authors for their great work and repos.

There are also some concurrent works that were released recently or will be released soon:

• LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench
• Embodied Agent Interface: A Single Line to Evaluate LLMs for Embodied Decision Making