Awesome AI Agents › Stateful Serverless Frameworks
Graphsignal is an AI optimization platform designed to help developers enhance the performance and efficiency of their machine learning models and applications. The platform provides tools to identify and optimize the main contributors to latency, analyze model API costs across various dimensions such as deployments, models, sessions, or custom tags, and ensure optimal inference performance and model configuration for hosted models. Additionally, it offers capabilities to track errors and monitor APIs, compute resources, and GPU utilization, making it a comprehensive solution for managing AI workloads. The Graphsignal Python tracer library allows seamless integration with Python applications. It can be installed easily via pip and configured using an API key, which can be set directly in the code or through environment variables. The tracer supports auto-instrumentation for popular libraries and frameworks like OpenAI and LangChain, automatically recording traces, errors, and relevant data such as prompts and completions. This data is then available for detailed analysis through the Graphsignal web application. The platform emphasizes low overhead, with tracing adding less than 100 microseconds per trace, ensuring minimal impact on application performance. Security and privacy are also prioritized; the tracer only makes outbound connections to the Graphsignal API endpoint and does not accept inbound connections or commands. Users can control data recording settings, such as disabling payload recording if desired. Graphsignal provides extensive documentation, including guides on manual tracing, inference profiling, session tracking, cost and usage monitoring, and feedback scoring. Integration examples are available in a separate repository to help developers get started quickly. Debugging support is available through configurable debug logging, and the platform encourages users to report issues via their support channels. Overall, Graphsignal offers a robust and user-friendly solution for monitoring, optimizing, and analyzing AI model performance and resource usage in Python applications.
https://github.com/graphsignal/graphsignal-python
LiteLLM is a Python SDK and proxy server that provides a unified OpenAI-compatible interface to call over 100 large language model APIs from multiple providers, featuring consistent output, retry logic, budget controls, and extensive observability integrations.
Flowise is an open-source drag-and-drop platform that enables users to easily build and deploy customized Large Language Model (LLM) application workflows with a user-friendly interface and flexible deployment options.
Parlant is a Conversation Modeling engine that enables precise, consistent, and reliable control over GenAI-driven conversational agents by enforcing structured behavioral guidelines and adapting dynamically to user interactions.
ToolBench is an open platform for training, serving, and evaluating large language models with advanced tool-use capabilities using a large-scale, richly annotated dataset of real-world APIs.
RivetKit is a stateful serverless framework that enables building scalable, real-time, and collaborative applications deployable across multiple platforms including Rivet, Cloudflare Workers, Bun, and Node.js.
LlamaDeploy is an async-first framework that enables seamless deployment, scaling, and productionization of agentic multi-service workflows built with llama_index, facilitating easy transition from development to cloud-based production environments.
Ragbits is a modular framework providing building blocks for rapid development, deployment, and monitoring of scalable and reliable Generative AI applications with flexible LLM integration and advanced document processing capabilities.
Mirascope is a flexible and user-friendly library that provides a unified interface to work with multiple large language model providers, simplifying AI-driven text generation and information extraction tasks.