swirl-search

Executes a single user query across 100+ heterogeneous data sources simultaneously using Celery workers and asynchronous task distribution, without copying or indexing data. The Search Orchestrator (swirl/models.py Search class) decomposes queries into source-specific formats, dispatches parallel tasks to Celery workers, and aggregates results as they complete. Uses Django ORM to manage Search objects with state tracking (RUNNING, COMPLETED, FAILED) and WebSocket communication for real-time progress updates to the Galaxy UI.

Provides extensible connector framework (swirl/connectors/connector.py base class) that abstracts 100+ data sources (HTTP APIs, databases, search engines, Microsoft Graph) into a unified interface. Each connector translates SWIRL's normalized query format into source-native syntax (SQL WHERE clauses, Elasticsearch queries, REST API parameters, GraphQL), executes the query, and normalizes results back to SWIRL's unified schema. Supports HTTP connectors for REST/GraphQL APIs, database connectors for SQL/NoSQL, and specialized connectors for Salesforce, Jira, Microsoft 365, Slack, BigQuery, and others.

Provides Galaxy web-based user interface (Django templates, static files, JavaScript) accessible at port 8000 for searching and visualizing results. Implements real-time search progress tracking via WebSocket, progressive result display as sources complete, and result filtering/sorting. Supports both simple keyword search and advanced search with filters, date ranges, and field-specific queries. Includes result preview, source attribution, and relevance scoring visualization. Built with Django templates and vanilla JavaScript for minimal dependencies.

Implements asynchronous search execution using Celery task queue (swirl/tasks.py) with configurable worker pool for parallel query execution across sources. Each source query is dispatched as separate Celery task, allowing independent execution and failure handling. Results are cached in Redis (configurable TTL) to avoid redundant queries for identical search parameters. Celery workers can be scaled horizontally to handle increased query load. Supports task monitoring, retry logic, and dead-letter queue for failed tasks.

Implements per-source authentication handling (swirl/connectors/) supporting multiple authentication methods: API keys, OAuth 2.0, basic auth, database credentials, and custom authentication schemes. Each connector manages its own authentication logic, allowing sources to use different authentication methods simultaneously. Credentials are stored in Django settings or environment variables (not in code). Supports OAuth token refresh for long-lived sessions. No centralized credential vault; requires external integration for enterprise credential management.

Provides Django admin interface for configuring data sources, managing searches, and monitoring system health. Allows admins to add/edit/delete data sources, configure connector parameters, set authentication credentials, and manage search history. Includes admin guide (docs/Admin-Guide.md) for production deployment and troubleshooting. Supports bulk operations for managing multiple sources. Provides search analytics (query volume, source performance, result quality metrics).

Implements result processing pipeline (swirl/processors/) that normalizes results from different sources into unified schema, applies relevance re-ranking algorithms, and deduplicates results. The Mixer component (swirl/mixers/mixer.py) combines results from multiple sources using configurable ranking strategies (BM25, TF-IDF, LLM-based relevance scoring). Processors transform raw connector output into normalized Result objects with standardized fields, handle PII removal (swirl/processors/remove_pii.py), and apply source-specific post-processing. Results are re-ranked based on relevance scores, source credibility, and recency.

Implements RAG pipeline (swirl/processors/rag.py) that uses LLM APIs (OpenAI, Anthropic, Ollama, Azure OpenAI) to synthesize answers from search results without moving data. The RAG processor takes normalized search results, constructs a prompt with result snippets as context, and calls the configured LLM to generate a natural language answer. Supports streaming responses via WebSocket to Galaxy UI for real-time answer generation. Integrates with search result ranking to prioritize high-relevance results in LLM context window.

Implements query processing layer (swirl/search.py, swirl/models.py Query class) that parses user natural language queries and transforms them into source-specific syntax. Each connector implements query_to_native_syntax() method to translate SWIRL's normalized query format (query string, filters, date ranges, field-specific queries) into source-native formats: SQL WHERE clauses for databases, Elasticsearch DSL for search engines, REST API parameters for HTTP APIs, GraphQL queries for GraphQL endpoints, Microsoft Graph query syntax for Microsoft 365. Supports query expansion, synonym replacement, and field mapping per source.

Implements WebSocket-based real-time communication (swirl/views.py, Galaxy UI) that streams search progress and results to clients as they complete. The Search Orchestrator updates Search object state (RUNNING, COMPLETED, FAILED) as Celery workers complete queries on individual sources. WebSocket connection pushes progress updates (source completion, result count, execution time) and result chunks to Galaxy UI in real-time, enabling users to see results from fast sources immediately without waiting for slow sources. Supports both polling (REST API) and streaming (WebSocket) interfaces.

Provides LLM provider abstraction layer (swirl/processors/rag.py) that supports multiple LLM APIs (OpenAI, Anthropic, Ollama, Azure OpenAI) through unified interface. Each provider implementation handles authentication, request formatting, streaming response parsing, and error handling. Supports streaming responses where LLM output is returned token-by-token via WebSocket, enabling real-time answer generation in Galaxy UI. Allows switching between LLM providers through configuration without code changes.

Provides specialized connectors for Microsoft 365 ecosystem (swirl/connectors/) including Microsoft Graph API connector for querying Teams, SharePoint, OneDrive, Outlook, and other Microsoft 365 services. Implements OAuth 2.0 authentication flow for secure credential management without storing passwords. Supports Microsoft Graph query syntax translation, pagination, and result normalization. Includes admin guide (docs/M365-Guide.md) for configuring Microsoft 365 integration in enterprise environments.

Provides Docker containerization (Dockerfile, docker-compose.yml) and Kubernetes deployment manifests for production deployment. Includes Nginx reverse proxy configuration for load balancing and SSL termination. Supports environment-based configuration management through .env files and Django settings, enabling deployment across development, staging, and production environments without code changes. Includes CI/CD pipeline (GitHub Actions) for automated testing, building, and deployment.

Implements pluggable processor pipeline architecture (swirl/processors/processor.py base class) where each processor transforms search results independently. Processors are composed in sequence, allowing flexible result transformation workflows: normalization, ranking, PII removal, RAG synthesis, custom filtering. Each processor implements process() method that takes results and returns transformed results. Processors can be enabled/disabled through configuration, and custom processors can be added by subclassing Processor base class.