Awesome AI Agents › Sensor Fusion Agents

OSU-NLP-Group/TravelPlanner

⭐ 546 Python added to this list on 2025-03-10 repository created 2024-01-01

TravelPlanner is a benchmark designed to evaluate the capabilities of language agents in real-world planning tasks that require complex tool use and adherence to multiple constraints. The project focuses on generating comprehensive travel plans based on user queries, which include transportation, daily meals, attractions, and accommodation for each day of the trip. The benchmark incorporates three types of constraints to simulate realistic scenarios: Environment Constraints, Commonsense Constraints, and Hard Constraints. These constraints ensure that the plans generated by language agents are practical, sensible, and feasible in real-world contexts. The project provides two main modes for evaluation: the two-stage mode and the sole-planning mode. In the two-stage mode, language agents use various search tools to gather necessary information before formulating a plan that meets user needs and commonsense constraints. The sole-planning mode simplifies the task by focusing solely on the planning aspect, assuming all crucial information is already available, allowing agents to concentrate on generating effective plans. TravelPlanner includes a comprehensive setup process involving environment creation, dependency installation, and database download. It supports multiple advanced language models such as GPT-3.5, GPT-4, Gemini, Mistral, and Mixtral. The project also offers tools for postprocessing, including converting natural language plans into JSON format using GPT-4, parsing, element extraction, and combining plan files for evaluation. Evaluation is supported through offline validation scripts and an official leaderboard to ensure fair and reliable benchmarking. The project strictly prohibits cheating methods such as reverse engineering, hard coding evaluation cues, or any human interference that compromises generalizability. TravelPlanner also provides fine-tuned models like Llama3.1-8B-Instruct and Qwen2-7B-Instruct for the sole-planning mode. Overall, TravelPlanner serves as a valuable resource for researchers and developers aiming to advance the capabilities of language agents in complex, real-world planning tasks with practical constraints and tool use.

https://github.com/OSU-NLP-Group/TravelPlanner

accommodationattractionsautonomous-agentsbenchmarkcommonsense-constraintcomplex-planningconstraintsenvironment-constraintevaluationfine-tuned-modelsgeminigpt-3.5gpt-4hard-constraintjson-parsinglanguage-agentlanguage-agentslarge-language-modelsleaderboardllama3.1-8b-instructmealsmistralmixtralplanningqwen2-7b-instructreal-world-planningsole-planning-modetool-usetransportationtravel-planningtravelplannertwo-stage-mode

Also in Sensor Fusion Agents

xlang-ai/OSWorld

OSWorld is a benchmarking platform for evaluating multimodal AI agents performing open-ended tasks in real computer environments using virtual machines and various virtualization technologies.

IBM/AssetOpsBench

Open IBM framework and benchmark for building, orchestrating and evaluating domain-specific AI agents in industrial asset operations, with MCP servers over sensors, failure modes, time series and work orders.

web-arena-x/webarena

WebArena is a self-hostable web environment designed for building and evaluating autonomous agents capable of realistic web navigation and interaction tasks.

google-deepmind/android_env

AndroidEnv is a Python library by DeepMind that provides a Reinforcement Learning platform by simulating Android devices, enabling agents to interact with real-world Android applications through touchscreen gestures for diverse RL tasks.

google-research/android_world

AndroidWorld is a comprehensive environment and benchmark for autonomous agents to interact with and control Android devices through a live emulator, featuring diverse tasks and integration with web-based benchmarks.

karanchawla/GPS_IMU_Kalman_Filter

An Extended Kalman Filter implementation that fuses GPS, IMU, and encoder sensor data to accurately estimate the pose of a ground robot in a navigation frame.

StonyBrookNLP/appworld

AppWorld is a high-fidelity execution environment simulating a world of apps and people to benchmark and evaluate interactive coding agents through diverse and challenging autonomous tasks.

nottelabs/open-operator-evals

Open-operator-evals is an open-source benchmarking framework that evaluates the performance of web operators and agents using multiple metrics and a reproducible dataset to provide transparent and statistically sound comparisons.