Awesome AI Agents › Agent Monitoring Tools

agentscope-ai/OpenJudge

⭐ 831 Python repository created 2025-07-08

OpenJudge, from the AgentScope project, is an open-source evaluation framework for LLM applications such as agents and chatbots. It targets the full evaluation loop: collect test data, define graders, run evaluation at scale, analyse weaknesses and iterate. The library ships more than fifty ready-to-use graders organised into a taxonomy. General graders cover semantic relevance, text similarity, code syntax validation and JSON structure compliance. Agent graders assess the agent lifecycle rather than only the final answer, scoring tool selection accuracy, memory and context preservation, plan feasibility and trajectory quality, along with reflection. Multimodal graders judge image-text coherence, text-to-image generation quality and how much an image contributes to an answer. A separate set of skill graders evaluates agent skill packages for threat exposure under an attack taxonomy, alignment between declaration and behaviour, completeness, relevance and design quality. Beyond the built-in set, OpenJudge can generate scenario-specific rubrics that act as graders for a particular application, and it can convert grading output into reward signals usable for fine-tuning and reinforcement learning. Each grader is shipped with benchmark datasets and pytest integration so its reliability can be re-validated, and validation artefacts are published as Hugging Face datasets. The ecosystem around it includes PawBench, a model-by-harness co-evaluation benchmark for agentic AI with 150 tasks across nine models and three harnesses with a public leaderboard, a reference hallucination arena for measuring fabricated academic citations, and an automated paper review application. A Streamlit user interface provides grader testing and an automatic arena, available both locally and as a hosted playground where graders can be tried without installation. Installation is from PyPI as py-openjudge and requires Python 3.10 or newer. Documentation is published as a site with English and Chinese versions. Licensed Apache-2.0.

https://github.com/agentscope-ai/OpenJudge

evaluationllm-as-judgegradersbenchmarkingagent-evaluationrewardsrlhfpython