OSS AI agent that indexes and searches the Epstein files

Ingests unstructured document collections (the Epstein files) and builds a dual-index combining traditional full-text search with vector embeddings for semantic similarity. The system likely uses an embedding model (e.g., OpenAI, Hugging Face) to vectorize document chunks, stores them in a vector database (FAISS, Pinecone, or Weaviate), and maintains a parallel inverted index for keyword matching. This enables hybrid search where queries can match both exact terms and semantically similar content across thousands of documents.

Wraps the indexed documents in an agentic Q&A loop where user queries are converted to embeddings, matched against the index, and the top-K retrieved chunks are passed as context to an LLM (likely GPT-4 or Claude) to generate grounded answers. The agent maintains conversation history to enable follow-up questions and likely implements retrieval-augmented generation (RAG) with prompt engineering to cite sources and avoid hallucination. The system probably includes a feedback loop where users can rate answer quality, which informs retrieval ranking.

Extends basic search with structured filtering on document metadata (dates, entities, document types) and likely uses named entity recognition (NER) to extract people, organizations, and locations from documents for faceted search. The system probably parses document metadata (creation date, author, classification) and builds a filter layer that allows queries like 'find documents mentioning John Doe between 2010-2015'. Entity extraction may use spaCy, BERT-based NER, or LLM-based extraction to populate a knowledge graph of relationships.

Uses embedding-based similarity to group related documents and identify patterns across the corpus. The system likely computes pairwise similarities between document embeddings, applies clustering algorithms (k-means, DBSCAN, or hierarchical clustering) to group semantically similar documents, and surfaces clusters to users as 'related documents' or 'document groups'. This enables discovery of thematic patterns, duplicate or near-duplicate documents, and document families without explicit user queries.

Implements an agent loop where the LLM can iteratively refine searches, retrieve additional context, and reason over retrieved documents to answer complex questions. The agent likely uses a tool-calling interface (OpenAI function calling or Anthropic tool_use) to invoke search, retrieve specific documents, and extract information, maintaining state across multiple reasoning steps. This enables complex workflows like 'find all meetings between X and Y, extract attendees, then find other meetings with those attendees' without explicit user guidance.

Enables users to export search results, answer chains, and evidence compilations into structured formats (PDF, JSON, CSV) with formatting, citations, and metadata preservation. The system likely uses a template engine (Jinja2, Handlebars) to format results, a PDF library (ReportLab, WeasyPrint) to generate PDFs with proper styling, and includes options for batch export of multiple documents or search results. This supports investigative workflows where findings must be compiled into shareable reports.

Implements role-based access control (RBAC) and detailed audit logging for document access, searches, and exports. The system likely uses a permission model (document-level or collection-level) to restrict who can view/search documents, logs all access with timestamps and user identity, and provides audit reports for compliance. This is critical for sensitive document collections where access must be tracked and restricted.