Awesome AI Agents › Stateful Serverless Frameworks
The llama-cpp-agent framework is a comprehensive tool designed to facilitate easy and efficient interaction with Large Language Models (LLMs). It provides a versatile interface that allows users to chat with LLMs, execute structured function calls, and obtain structured outputs, even from models that are not fine-tuned for JSON output or function calls. The framework supports various LLM providers including llama.cpp server, llama-cpp-python, TGI, and vllm servers, making it highly adaptable to different environments. Key features of the llama-cpp-agent include a simple chat interface for seamless conversations with LLMs, the ability to generate structured output objects, and support for both single and parallel function calling. It also incorporates Retrieval Augmented Generation (RAG) with colbert reranking to enhance information retrieval capabilities. The framework supports agentic chains with tools, enabling conversational, sequential, and mapping chains for complex text processing tasks. A notable aspect of the framework is its use of guided sampling, which constrains model output to user-defined structures using grammars and JSON schema generation. This allows even smaller 7B parameter LLMs to perform function calling and produce structured outputs effectively. The framework is compatible with various tools and schemas including python functions, pydantic tools, llama-index tools, and OpenAI tool schemas, providing flexibility for diverse applications ranging from casual chatting to specific function executions. The project offers extensive documentation and a variety of usage examples demonstrating its capabilities, such as simple chat, parallel function calling, structured output generation, RAG, llama-index tool integration, sequential and mapping chains, and knowledge graph creation. It also provides predefined message formatters and supports the creation of custom message formatters to tailor interactions with LLMs. Overall, llama-cpp-agent is a powerful and flexible framework that simplifies the integration and utilization of LLMs for developers and researchers, enhancing the ability to build sophisticated applications leveraging large language models.
https://github.com/Maximilian-Winter/llama-cpp-agent
LiteLLM is a Python SDK and proxy server that provides a unified OpenAI-compatible interface to call over 100 large language model APIs from multiple providers, featuring consistent output, retry logic, budget controls, and extensive observability integrations.
Flowise is an open-source drag-and-drop platform that enables users to easily build and deploy customized Large Language Model (LLM) application workflows with a user-friendly interface and flexible deployment options.
Parlant is a Conversation Modeling engine that enables precise, consistent, and reliable control over GenAI-driven conversational agents by enforcing structured behavioral guidelines and adapting dynamically to user interactions.
ToolBench is an open platform for training, serving, and evaluating large language models with advanced tool-use capabilities using a large-scale, richly annotated dataset of real-world APIs.
RivetKit is a stateful serverless framework that enables building scalable, real-time, and collaborative applications deployable across multiple platforms including Rivet, Cloudflare Workers, Bun, and Node.js.
LlamaDeploy is an async-first framework that enables seamless deployment, scaling, and productionization of agentic multi-service workflows built with llama_index, facilitating easy transition from development to cloud-based production environments.
Ragbits is a modular framework providing building blocks for rapid development, deployment, and monitoring of scalable and reliable Generative AI applications with flexible LLM integration and advanced document processing capabilities.
Mirascope is a flexible and user-friendly library that provides a unified interface to work with multiple large language model providers, simplifying AI-driven text generation and information extraction tasks.