raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK platform for observability, monitoring, and evaluation of AI agents and LLM projects, featuring tracing, debugging, advanced analytics, and project management tools.
Awesome AI Agents › Agent Monitoring Tools
JudgmentLabs/judgeval
Judgeval is an open-source Python SDK for observing and evaluating LLM-powered agents in development and in production. Instrumentation is decorator-based: a call to Tracer.init names the project, the wrap helper patches an LLM client, and the Tracer.observe decorator marks any function as a span with a type such as tool or agent. Inputs, outputs and token usage are captured automatically. The tracing layer is built on OpenTelemetry, so traces can flow into existing observability stacks alongside the vendor backend. On top of tracing the SDK provides agent judges: prompt-based scorers that grade agent behavior at scale and emit structured, labeled results rather than a single numeric score. Those results accumulate into a searchable record, so recurring failure modes can be found across many runs. Judges run against live production traffic server-side, which the project states adds no latency to the application path, and the same judges can be replayed over historical traces to check whether a code change actually fixed a regression before it ships. Detected behaviors can be routed to Slack alerts. A query language, JQL, is exposed through a Judgeval client for filtering traces by session or other fields from Python. Auto-instrumentation covers OpenAI, Anthropic, Google GenAI and Together AI clients, with framework-level support for LangGraph, OpenLit and the Claude Agent SDK. A companion CLI manages agents, traces, judges, behaviors and evaluations from the terminal, and an MCP server exposes the same data to MCP-compatible assistants and IDEs so failures can be inspected without leaving the editor. The SDK is installed from PyPI and requires an API key and organization id. It targets engineers running agents in production who need failure detection, root-cause triage and regression testing backed by real usage data.
https://github.com/JudgmentLabs/judgeval
Also in Agent Monitoring Tools
MervinPraison/PraisonAI
PraisonAI is a production-ready Multi AI Agents framework that enables the creation and management of intelligent AI agents to automate and solve complex problems through collaboration and customization.
FailproofAI/failproofai
Local observability and policy-enforcement layer that hooks into 12 AI agent harnesses, including Claude Code, Codex and Hermes, to log every run and block dangerous tool calls before they execute.
apache/hertzbeat
Apache HertzBeat is an AI-powered open source real-time observability system that unifies metrics and logs collection, alerting, and notification with high performance and customizable monitoring capabilities.
junhoyeo/tokscale
Tokscale is a CLI utility and dashboard for tracking and visualizing token usage and costs across various AI coding agents.
matt1398/claude-devtools
Desktop and Docker application that reads local Claude Code session logs and reconstructs tool calls, thinking steps, subagent trees and per-turn token attribution in a visual interface.
eumemic/ai-legion
AI Legion is a platform for autonomous agents powered by large language models that collaborate to accomplish tasks with minimal human intervention.
traceroot-ai/traceroot
TraceRoot is open-source observability for AI agents that traces runs, detects failures with LLM judges, root-causes them against source code and GitHub history, opens fix PRs and evaluates each fix.