Mixtral 8x7B

Routes each input token through exactly 2 of 8 expert networks per transformer layer using a learned router network, activating only 12.9B of 46.7B total parameters per forward pass. The router makes independent routing decisions per token per layer, with expert outputs combined additively. This sparse activation pattern enables inference throughput equivalent to a 12.9B dense model while maintaining GPT-3.5-level performance across benchmarks.

Generates coherent, contextually-aware text across general-purpose language tasks by applying transformer decoder architecture with 32K token context window. The model was trained on open web data and achieves performance parity with GPT-3.5 on standard benchmarks (MMLU, HellaSwag, TruthfulQA, Winogrande, GSM8K, MATH, HumanEval) while maintaining lower computational cost through sparse routing. Supports both base and instruction-tuned variants, with the Instruct variant fine-tuned via supervised fine-tuning (SFT) and Direct Preference Optimization (DPO).

Evaluated across standard language model benchmarks including MMLU (knowledge), HellaSwag (common sense reasoning), TruthfulQA (factuality), Winogrande (coreference resolution), GSM8K (math), MATH (advanced math), and HumanEval (code generation). Results demonstrate performance parity with GPT-3.5 on most benchmarks, with specific scores provided for MT-Bench (8.30 for Instruct variant). Benchmark evaluation enables quantitative comparison with other models and verification of capability claims.

Base model (non-Instruct variant) has no built-in safety guardrails and will follow any instruction without refusal or content filtering. Safety behavior is not enforced through training or architecture; instead, the model relies on explicit prompting or preference optimization (as in the Instruct variant) to learn refusal behavior. This design choice prioritizes capability and flexibility over safety by default, requiring users to implement safety measures explicitly.

Generates and completes code across multiple programming languages by applying transformer decoder architecture trained on code-inclusive datasets. The model demonstrates strong performance on HumanEval benchmark and supports code generation for tasks ranging from single-function completion to multi-file refactoring. Instruction-tuned variant (Mixtral 8x7B Instruct) provides improved code understanding and explanation capabilities through supervised fine-tuning and preference optimization.

Generates coherent text in English, French, German, Spanish, and Italian through transformer decoder architecture trained on multilingual open web data. The model maintains language-specific performance across supported languages while using the same sparse routing mechanism as English generation. Multilingual performance is documented with benchmark results for each language, though specific scores are not detailed in available documentation.

Follows natural language instructions and engages in multi-turn conversation through the Mixtral 8x7B Instruct variant, which is fine-tuned via supervised fine-tuning (SFT) and Direct Preference Optimization (DPO). The instruction-tuned variant achieves MT-Bench score of 8.30, positioning it as the best open-source model on this benchmark at release. The model learns to refuse harmful requests and provide helpful, harmless, and honest responses through preference optimization, though safety guardrails are not guaranteed without explicit prompting.

Enables efficient inference through integration with vLLM framework and Megablocks CUDA kernels, which are specifically optimized for sparse mixture-of-experts computation. The sparse activation pattern (2 of 8 experts per token) is implemented via custom CUDA kernels that avoid computing inactive expert parameters, reducing memory bandwidth and compute requirements. Inference throughput is equivalent to a 12.9B dense model despite 46.7B total parameters, achieving 6x speedup over Llama 2 70B while maintaining equivalent performance.

Enables cloud deployment of Mixtral via SkyPilot framework, which abstracts cloud infrastructure provisioning and management. SkyPilot handles GPU selection, instance provisioning, and vLLM endpoint deployment across cloud providers, reducing operational complexity for teams deploying Mixtral at scale. The framework integrates with vLLM for optimized inference, enabling rapid deployment of Mixtral endpoints without manual infrastructure configuration.

Demonstrates reduced bias compared to Llama 2 70B on BBQ (Bias Benchmark for QA) evaluation, indicating improved fairness across demographic groups. The model was evaluated on BBQ benchmark which measures bias in question-answering tasks across protected attributes (gender, race, religion, etc.). Additionally evaluated on BOLD (Bias in Open-Ended Language Generation) benchmark, showing more positive sentiment than Llama 2 but with similar variance, suggesting different bias characteristics rather than elimination of bias.

Distributed under Apache 2.0 open-source license, enabling unrestricted commercial use, modification, and redistribution with minimal attribution requirements. The open weights are available for download and self-hosting, providing full control over model deployment, fine-tuning, and integration without reliance on proprietary APIs or vendor lock-in. Apache 2.0 licensing permits both commercial and non-commercial use cases with explicit patent protection.

Supports 32,768 token context window, enabling processing of long documents, multi-turn conversations, and complex prompts without chunking or context truncation. The extended context window is implemented through standard transformer architecture without explicit long-context techniques (e.g., ALiBi, RoPE modifications), relying on training data that includes long sequences. Context window enables coherent reasoning across longer documents and more complex multi-step tasks compared to models with smaller context windows.