Awesome AI Agents › AI Agent Gateway

Arize-ai/phoenix

⭐ 11444 Jupyter Notebook added to this list on 2025-04-19 repository created 2022-11-09

Phoenix is an open-source AI observability platform designed to facilitate experimentation, evaluation, and troubleshooting of AI applications, particularly those involving large language models (LLMs). It offers a comprehensive suite of tools and features that enable developers and data scientists to trace, evaluate, and optimize their AI models and prompts effectively. Key functionalities include tracing LLM application runtime using OpenTelemetry-based instrumentation, which helps in monitoring and diagnosing model behavior during execution. The platform supports evaluation by leveraging LLMs themselves to benchmark application performance through response and retrieval evaluations, providing insights into model accuracy and effectiveness. Phoenix also allows users to create versioned datasets for experimentation, evaluation, and fine-tuning, ensuring reproducibility and systematic testing of AI models. Experiments can be tracked and evaluated to assess changes in prompts, LLMs, and retrieval methods, facilitating iterative improvement. The playground feature enables prompt optimization, model comparison, parameter adjustment, and replaying of traced LLM calls, making it a versatile environment for prompt engineering. Additionally, prompt management capabilities include version control, tagging, and systematic testing of prompt changes, which are crucial for maintaining and improving prompt quality over time. The platform is vendor and language agnostic, supporting popular frameworks such as LlamaIndex, LangChain, Haystack, DSPy, and smolagents, as well as LLM providers like OpenAI, Bedrock, MistralAI, VertexAI, and LiteLLM. This broad compatibility ensures that Phoenix can be integrated into diverse AI development workflows. It is designed to run in various environments, including local machines, Jupyter notebooks, containerized deployments, and cloud infrastructures, offering flexibility in deployment. Installation is straightforward via pip or conda, and container images are available on Docker Hub for easy deployment using Docker or Kubernetes. The platform includes the full Phoenix package and lightweight sub-packages for Python and TypeScript to complement the main platform deployment. Overall, Phoenix provides a robust, flexible, and comprehensive solution for AI observability and evaluation, empowering users to enhance the performance and reliability of their AI systems.

https://github.com/Arize-ai/phoenix

agentsai-monitoringai-observabilityaiengineeringanthropicbedrockbenchmarkingcloud-deploymentconda-installationcontainerized-deploymentdatasetsdockerdspyevalsevaluationexperimentationexperimentsfine-tuninghaystackjupyter-notebookkuberneteslangchainlanguage-agnosticlarge-language-modelslitellmllamaindexllmllm-evalllm-evaluationllmopsllmslocal-deploymentmistralaiopenaiopentelemetrypip-installationprompt-engineeringprompt-managementprompt-optimizationsmolagentstaggingtracingtroubleshootingvendor-agnosticversion-controlvertexai