raga-ai-hub/RagaAI-Catalyst
RagaAI Catalyst is a Python SDK platform for observability, monitoring, and evaluation of AI agents and LLM projects, featuring tracing, debugging, advanced analytics, and project management tools.
Awesome AI Agents › Agent Monitoring Tools
traceroot-ai/traceroot
TraceRoot is an open-source observability and self-improvement layer for AI agent systems. It captures LLM calls, agent actions and tool usage through an OpenTelemetry-compatible SDK, then filters that stream so the traces that matter are surfaced instead of every span. On top of tracing sit detectors: an LLM-as-judge evaluator screens incoming traces for hallucinations, tool and logic failures, safety violations and intent drift, raises findings, sends email and Slack alerts, and can automatically trigger a root cause analysis. The agentic debugging component reads the traces, connects to a sandbox holding the production source code, points at the exact failing line, and correlates the failure with the repository's commits, pull requests and open issues, then opens a pull request with a proposed fix. Any model provider can be plugged in through bring-your-own-key configuration, with OpenAI, Anthropic, Gemini, xAI, DeepSeek, OpenRouter, Kimi and GLM listed. The evaluation side closes the loop: confirmed production findings can be turned into golden datasets in one click, and offline evaluations run from the TraceRoot CLI or SDK, including from inside coding agents such as Claude Code, Codex and Cursor, so every fix is verified against the recorded failures and regressions are caught release over release. Deployment options are a hosted cloud service, a local developer mode that runs infrastructure in Docker with the application on the host, a full local Docker mode, and an experimental Terraform plus Helm path for Kubernetes on AWS. Native SDKs are published for Python and TypeScript. Both the observability platform and the AI debugging layer are open source, which the project positions against vendor lock-in. It targets engineering teams running LLM agents in production.
https://github.com/traceroot-ai/traceroot
Also in Agent Monitoring Tools
superset-sh/superset
Superset is a desktop application (macOS) designed to orchestrate and manage multiple CLI-based AI coding agents in parallel using isolated Git worktrees, featuring built-in diff viewing and IDE in...
MervinPraison/PraisonAI
PraisonAI is a production-ready Multi AI Agents framework that enables the creation and management of intelligent AI agents to automate and solve complex problems through collaboration and customization.
apache/hertzbeat
Apache HertzBeat is an AI-powered open source real-time observability system that unifies metrics and logs collection, alerting, and notification with high performance and customizable monitoring capabilities.
junhoyeo/tokscale
Tokscale is a CLI utility and dashboard for tracking and visualizing token usage and costs across various AI coding agents.
matt1398/claude-devtools
Desktop and Docker application that reads local Claude Code session logs and reconstructs tool calls, thinking steps, subagent trees and per-turn token attribution in a visual interface.
eumemic/ai-legion
AI Legion is a platform for autonomous agents powered by large language models that collaborate to accomplish tasks with minimal human intervention.
chiphuyen/sniffly
Sniffly is a local analytics dashboard that parses Claude Code logs to show usage statistics, error breakdowns and full message history, with optional shareable dashboards.