Awesome AI Agents › AI Developer Toolkit

openinterpreter/aifs

⭐ 463 Python repository created 2024-01-16

aifs, short for AI Filesystem, is a minimal Python package that provides local semantic search over a directory tree. A single call to its search function takes a natural-language query and a path, and returns the passages from that folder that match the query in meaning rather than by keyword. On the first run over a folder, aifs uses the unstructured library to parse and chunk every supported file, covering plain text, Python and shell sources, Word, PowerPoint and PDF documents, HTML, email messages and images. It then uses Chroma to embed those chunks locally and writes the resulting index into a hidden index file inside the same folder. Subsequent searches reuse that stored index, so only the first query pays the parsing and embedding cost and later searches are fast. When a file changes or a new file appears, only the affected chunks are re-indexed; deletions are acknowledged as an unfinished part of the implementation. Everything runs on the local machine, with no external service required for either embedding or retrieval, which makes it usable over private files. The project deliberately keeps its scope narrow and defers to upstream components for the hard parts: parsing quality is delegated to unstructured, and embedding quality to Chroma, on the stated assumption that both will keep tracking current practice. The authors note an open question about chunking, arguing that different file types deserve different chunking strategies. aifs was built by the Open Interpreter team to let their agent search a user file system quickly, and that is its main audience: developers building LLM agents or assistants that need a lightweight retrieval layer over local folders without standing up a vector database service. Installation is a single pip command, with an optional extra pulling in the full document-parsing dependency set.

https://github.com/openinterpreter/aifs

semantic-searchembeddingschromaretrievallocal-firstopen-interpreteragent-toolingpython