Awesome AI Agents › Video Processing Agents

microsoft/DeepVideoDiscovery

⭐ 427 Python repository created 2025-06-04

Deep Video Discovery is the official implementation of a NeurIPS 2025 paper from Microsoft on agentic search for understanding very long videos. Rather than feeding a whole video to a multimodal model at once, the system segments the video into clips and treats that segmented collection as an environment the agent explores. A large language model plans autonomously, chooses among tools that work at different granularities, from a global browse tool that reasons over textual descriptions of many clips down to detailed inspection of a single segment, and iterates: it extracts evidence, summarises and reflects on what it has observed, then decides what to look at next until it can answer the user question. This deep research style loop lets the agent answer complex questions about hours of footage without processing every frame at full detail. The repository provides the agent code, configuration for both the OpenAI API and Azure OpenAI, a local run script that takes a YouTube URL and a question, and a Gradio demo with automatic subtitle support. A lite mode uses subtitles only, which suits podcast style recordings and lowers cost. Reported results place the method well ahead of prior work on several long video benchmarks, including LVBench, Video-MME, LongVideoBench and EgoSchema, and the authors publish precomputed captions for those benchmarks together with a reproduction guide so the numbers can be checked. Compared with the original paper code, the released version was refactored for readability and the global browse tool was changed to use textual clip descriptions instead of raw pixels for efficiency. It targets researchers and engineers building video understanding systems and anyone studying tool-using agents over non-text environments.

https://github.com/microsoft/DeepVideoDiscovery

agentvideo-understandingdeep-researchlong-form-videotool-usevideo-processingbenchmarks