Awesome AI Agents › Stateful Serverless Frameworks
LlamaDeploy, formerly known as llama-agents, is an asynchronous-first framework designed to deploy, scale, and productionize agentic multi-service systems based on workflows created with llama_index. It enables developers to build multiple workflows in llama_index and run them as services accessible via HTTP APIs, which can be integrated with user interfaces or other system services. The primary goal of LlamaDeploy is to facilitate the transition from development environments, such as notebooks, to cloud-based production systems with minimal or no changes to the original code. This is achieved through two main interaction methods: the llamactl command-line interface (CLI) and the LlamaDeploy SDK, both included in the llama_deploy Python package. LlamaDeploy offers several key advantages. It provides seamless deployment by bridging the gap between development and production, allowing workflows to be deployed with minimal code modifications. Its microservices architecture supports easy scalability of individual components as system demands grow. The framework employs a hub-and-spoke architecture, offering flexibility to swap components like message queues or add new services without disrupting the entire system. It also includes fault tolerance features such as built-in retry mechanisms and failure handling to enhance robustness in production environments. State management is handled by a control plane that manages state across services, simplifying complex multi-step processes. Being async-first, LlamaDeploy is optimized for high-concurrency scenarios, making it suitable for real-time and high-throughput applications. The project provides practical examples to help users get started quickly, including quick start guides, web-based user interface deployments, and message queue integrations. Comprehensive documentation is available to explain the architecture and components in detail. LlamaDeploy is installed via pip and is designed to make deploying agentic workflows to production straightforward and efficient.
https://github.com/run-llama/llama_deploy
LiteLLM is a Python SDK and proxy server that provides a unified OpenAI-compatible interface to call over 100 large language model APIs from multiple providers, featuring consistent output, retry logic, budget controls, and extensive observability integrations.
Flowise is an open-source drag-and-drop platform that enables users to easily build and deploy customized Large Language Model (LLM) application workflows with a user-friendly interface and flexible deployment options.
Parlant is a Conversation Modeling engine that enables precise, consistent, and reliable control over GenAI-driven conversational agents by enforcing structured behavioral guidelines and adapting dynamically to user interactions.
ToolBench is an open platform for training, serving, and evaluating large language models with advanced tool-use capabilities using a large-scale, richly annotated dataset of real-world APIs.
RivetKit is a stateful serverless framework that enables building scalable, real-time, and collaborative applications deployable across multiple platforms including Rivet, Cloudflare Workers, Bun, and Node.js.
Ragbits is a modular framework providing building blocks for rapid development, deployment, and monitoring of scalable and reliable Generative AI applications with flexible LLM integration and advanced document processing capabilities.
Mirascope is a flexible and user-friendly library that provides a unified interface to work with multiple large language model providers, simplifying AI-driven text generation and information extraction tasks.
Strands Agents Tools is a versatile toolkit that equips AI agents with powerful capabilities for file operations, system interaction, API communication, multimedia processing, memory management, and advanced reasoning to build intelligent applications efficiently.