Awesome AI Agents › Multi-Agent Frameworks

RobotecAI/rai

⭐ 588 Python repository created 2024-06-04

RAI is a framework from Robotec.ai for adding large language model agents to physical and simulated robots. It targets ROS 2, supports the Humble and Jazzy distributions on Ubuntu 22.04 and 24.04, and is split into packages that can be adopted separately. The core package provides the multi-agent runtime, human-robot interaction and multimodal message handling, so an agent can consume camera images, sensor readings and speech in the same conversation and issue commands back to the robot stack. A component called rai whoami builds a description of a specific robot embodiment by synthesising documentation, images and URDF models from a structured directory, which gives the agent grounded knowledge about the machine it controls. Speech is covered by separate packages for speech-to-text and text-to-speech, perception adds object detection using open-set vision models, rai_sim connects agents to simulators, rai_nomad integrates the NoMaD navigation policy, and rai_bench is a benchmarking suite for evaluating agents, models, tools and simulators; a fine-tuning package for training on embodied data is listed as planned. Documented demonstrations run in simulation and include obstacle and mission reasoning for an autonomous tractor in a virtual orchard, natural language manipulation with a Franka Panda arm combined with Grounded SAM 2, navigation with a Husarion ROSbot XL, and an on-board agentic mobile manipulator. The project is published under Apache 2.0, has an accompanying arXiv paper describing the architecture, and is one of the main projects of the ROS Embodied AI Community Group. It suits robotics engineers who already have a working ROS 2 stack and want natural language interaction, flexible task reasoning and reusable AI capabilities layered on top of it.

https://github.com/RobotecAI/rai

ai-agents-frameworkembodied-agentsembodied-airoboticsros2llmmultimodalvlmsimulationbenchmarking