Sourcerer

Enables AI agents to find relevant code chunks across a codebase using natural language queries rather than regex or file browsing. The system converts user queries into embeddings using OpenAI's embedding API, then performs vector similarity search against a chromem-go vector database containing embeddings of all parsed code chunks. This approach dramatically reduces token consumption by returning only semantically relevant code segments instead of entire files.

Parses source code using Tree-sitter language parsers to build Abstract Syntax Trees (ASTs), then extracts semantic chunks at the granularity of functions, classes, methods, and interfaces. Each chunk receives a stable ID following the pattern file.ext::Type::method, enabling precise code retrieval and reference. The system supports Go, JavaScript, Python, TypeScript, and Markdown with language-specific extraction rules that respect syntactic boundaries.

Continuously monitors the workspace directory for file changes using file system watchers, detects modifications to source files, and triggers re-indexing of affected chunks with debouncing to avoid redundant parsing during rapid edits. The system respects .gitignore rules to exclude non-source files and maintains a queue of pending files awaiting indexing. This enables the semantic search index to stay synchronized with the codebase without manual refresh commands.

Exposes semantic code search and navigation capabilities through the Model Context Protocol (MCP) as callable tools that AI agents can invoke. The system implements five primary MCP tools: semantic_search (natural language queries), get_chunk_code (retrieve by ID), find_similar_chunks (discover related code), index_workspace (manual re-indexing), and get_index_status (progress tracking). This integration allows Claude, other LLMs, and AI agents to treat code discovery as a native capability without custom API integration.

Retrieves specific code chunks by their stable IDs (format: file.ext::Type::method) without requiring file path knowledge or line number tracking. The system maintains a mapping from chunk IDs to their source locations and content, enabling precise code access that survives file edits and refactoring. This capability supports both direct ID-based retrieval and discovery of similar chunks through semantic comparison.

Builds and maintains a chromem-go in-memory vector database containing embeddings of all parsed code chunks. For each semantic chunk extracted by the parser, the system generates an embedding using OpenAI's embedding API, stores it in the vector database with the chunk ID and metadata, and enables fast similarity search. The database is rebuilt incrementally as files change, with new chunks added and deleted chunks removed from the index.

Analyzes source code across five programming languages (Go, JavaScript, Python, TypeScript, Markdown) using language-specific Tree-sitter parsers and extraction rules. Each language parser understands language-specific constructs: Go extracts functions/methods/types/interfaces, JavaScript extracts functions/classes/variables, Python extracts functions/classes/decorators, TypeScript extracts functions/interfaces/enums/classes, and Markdown extracts sections/headings. This enables semantically accurate code chunking that respects language idioms and structure.

Provides visibility into the indexing state of the workspace through a get_index_status MCP tool that reports current progress, lists files pending indexing, and indicates whether the index is fully synchronized with the file system. The system tracks which files have been parsed, which are queued for processing, and provides status updates without blocking ongoing searches. This enables agents and users to understand index freshness and plan queries accordingly.

Provides an index_workspace MCP tool that allows agents or users to manually trigger a full re-indexing of the workspace, bypassing the automatic file watcher and debouncing logic. This is useful after large code changes, when the file watcher may have missed changes, or when the index becomes corrupted. The re-indexing process parses all source files, generates new embeddings, and rebuilds the vector database from scratch.