TurboPilot

Runs quantized code generation models (6B+ parameters) entirely on-device using GGML tensor library from llama.cpp, enabling CPU/GPU inference without cloud API calls. The architecture abstracts model implementations through a TurbopilotModel base class with predict_impl() virtual methods, allowing multiple model architectures (GPT-J, GPT-NeoX, Starcoder) to share common inference plumbing while delegating architecture-specific forward passes to concrete subclasses.

Provides a polymorphic TurbopilotModel base class with load_model() and predict_impl() virtual methods that allows swapping between GPT-J, GPT-NeoX, and Starcoder architectures without changing client code. Each concrete model implementation handles architecture-specific tokenization, attention patterns, and forward pass logic while inheriting common synchronization and error handling from the base class.

Uses Crow C++ web framework to implement HTTP server with request routing to different handlers (OpenAI-compatible, HF-compatible, health check, auth). Crow handles HTTP parsing, routing, JSON serialization, and response formatting, allowing TurboPilot to expose multiple API formats from a single server process. Request handlers are registered as route callbacks that parse incoming requests, call model inference, and serialize responses.

Implements synchronization primitives (mutexes, locks) in the TurbopilotModel base class to ensure thread-safe model inference when multiple requests arrive concurrently. The predict() method acquires a lock before calling predict_impl(), serializing inference across threads and preventing race conditions in model state. This allows the HTTP server to accept concurrent requests while ensuring model inference is atomic and consistent.

Provides Dockerfile and Docker Compose configuration for containerized TurboPilot deployment, enabling consistent environment across development, testing, and production. Docker image includes C++ build tools, CUDA runtime (optional), model weights, and TurboPilot binary, allowing single-command deployment without manual setup. Docker Compose enables multi-container deployments with volume mounts for model persistence and port mapping for API access.

Implements GitHub Actions CI/CD pipeline that automatically builds TurboPilot on push, runs unit tests, validates model loading, and publishes Docker images to registry. Pipeline ensures code quality, catches regressions early, and enables automated deployment. Tests verify model inference correctness, API endpoint functionality, and performance benchmarks across different model architectures.

Exposes OpenAI-compatible REST API endpoints (POST /v1/completions, POST /v1/engines/codegen/completions) that translate incoming OpenAI format requests into internal TurboPilot model calls, then map responses back to OpenAI schema. This allows drop-in replacement of OpenAI API calls with local TurboPilot endpoints without client code changes, implemented via Crow C++ HTTP server request handlers that parse JSON, validate parameters, and serialize responses.

Exposes POST /api/generate endpoint compatible with Hugging Face Inference API schema, translating HF-format requests (inputs, parameters) into TurboPilot model calls and returning HF-compatible response format. Enables integration with HF ecosystem tools and allows testing models against HF benchmarks without code changes, implemented as a separate request handler in the Crow HTTP server.

Integrates with VS Code through the FauxPilot extension, which sends code context (file content, cursor position, surrounding lines) to TurboPilot server and displays completions as inline suggestions. The extension handles editor state management, context extraction, and UI rendering while TurboPilot handles inference, allowing seamless code completion experience within the editor without leaving the IDE.

Supports both CPU-only and GPU-accelerated inference via CUDA 11.0+, with automatic fallback to CPU if CUDA unavailable. The GGML library handles tensor operations and memory management for both backends, allowing the same model binary to run on CPU (slower but portable) or GPU (faster, requires NVIDIA hardware). Users specify inference device at startup via command-line flags, and TurboPilot automatically routes computations to the selected backend.

Loads pre-quantized model weights in GGML format (.bin, .gguf) directly into memory without conversion, enabling efficient storage and fast loading of large models. The GGML library handles deserialization, memory mapping, and format validation, allowing 6B parameter models to fit in 4GB RAM through 4-bit or 8-bit quantization. Users download quantized weights from HuggingFace or convert their own models using provided Python scripts.

Provides Python conversion scripts that transform PyTorch model weights to GGML quantized format, handling architecture-specific weight mapping, quantization, and format serialization. Users run conversion scripts on their fine-tuned models or custom architectures, producing .bin/.gguf files compatible with TurboPilot inference engine. Conversion handles tokenizer export, metadata embedding, and validation to ensure converted models are inference-ready.

Exposes GET / endpoint that returns server health status and basic metadata (version, loaded model, inference capabilities). Enables monitoring tools and clients to verify TurboPilot availability, detect server crashes, and confirm model loading before sending inference requests. Health check is lightweight (no inference) and returns immediately, suitable for load balancer health probes and automated monitoring.

Exposes GET /copilot_internal/v2/token endpoint that mimics GitHub Copilot's authentication flow, returning mock authentication tokens for clients expecting Copilot-compatible auth. Enables GitHub Copilot-compatible clients (like Copilot CLI) to authenticate against TurboPilot without modification, though actual authentication is not enforced (tokens are mock values). This allows testing Copilot-compatible tools against local TurboPilot.