Gemma 2

Gemma 2 implements a hybrid attention mechanism that alternates between local (sliding window) and global (full sequence) attention layers throughout the transformer stack. Local attention reduces computational complexity from O(n²) to O(n·w) where w is window size, while global attention layers maintain long-range dependencies. This architecture enables efficient processing of contexts up to 8K tokens without the quadratic memory scaling of standard dense attention, using a pattern similar to Longformer but optimized for inference speed on consumer hardware.

Gemma 2 is trained using knowledge distillation from larger Gemini models, where the 27B variant learns to replicate reasoning patterns and factual knowledge from Gemini's 70B+ scale models. This involves training on synthetic data generated by Gemini, response ranking using Gemini outputs as ground truth, and fine-tuning on instruction-following tasks where Gemini demonstrates superior performance. The distillation process preserves reasoning capabilities while reducing model size by ~60%, enabling the 27B model to match 70B Llama 3 performance on benchmarks like MMLU and GSM8K.

Achieves strong performance on standard ML benchmarks (MMLU, HumanEval, GSM8K, etc.) with the 27B variant matching or exceeding Llama 3 70B on many tasks despite being 2.6x smaller. Performance comes from combination of base training on diverse data, instruction-tuning for task-specific formats, and knowledge distillation from Gemini models. Benchmark results are publicly available and reproducible, enabling informed model selection for specific use cases.

Gemma 2 provides three model sizes (2B, 9B, 27B) with identical tokenizer, architecture, and API interface, enabling seamless scaling from edge devices to high-performance inference. All variants use the same vocabulary, attention patterns, and instruction format, allowing developers to prototype on 2B, validate on 9B, and deploy on 27B without code changes. This consistency is achieved through careful architectural design where layer counts and hidden dimensions scale proportionally while maintaining the same transformer block structure and attention mechanism.

Gemma 2 is fine-tuned on instruction-following tasks using a specific prompt format that enables reliable structured output generation (JSON, code, markdown tables) through prompt engineering rather than constrained decoding. The model learns to follow format specifications in system prompts and examples, using patterns like 'Output as JSON: {"key": "value"}' to guide generation. This approach leverages the model's reasoning capabilities to understand and respect output constraints without requiring specialized decoding logic, making it compatible with any inference framework.

Gemma 2 is optimized for inference through native support for 8-bit and 4-bit quantization (via bitsandbytes, GPTQ, AWQ) and Flash Attention v2 integration, reducing memory footprint by 75-87% and improving throughput by 2-4x compared to full-precision inference. The model architecture is designed to maintain quality under aggressive quantization through careful layer normalization and activation scaling during training. Inference frameworks like vLLM, Ollama, and llama.cpp provide optimized kernels for Gemma 2 specifically, enabling sub-100ms latency on consumer GPUs.

Gemma 2 is trained with constitutional AI and safety fine-tuning to reduce generation of harmful, illegal, or unethical content while maintaining instruction-following capability. The model uses a combination of RLHF (reinforcement learning from human feedback) with safety-focused reward models and instruction-following data to balance helpfulness and safety. This is implemented through a two-stage training process: first instruction-following on benign tasks, then safety fine-tuning on adversarial examples to reduce harmful outputs without catastrophic forgetting of useful capabilities.

Gemma 2 is trained on multilingual instruction-following data, enabling the model to follow instructions and generate coherent responses in 10+ languages including English, Spanish, French, German, Italian, Portuguese, Dutch, Russian, Chinese, and Japanese. The model achieves this through cross-lingual transfer during training, where instruction-following patterns learned in English transfer to other languages through shared vocabulary and transformer representations. Performance varies by language, with European languages performing near-English quality while Asian languages show 10-20% quality degradation due to tokenization and training data imbalance.

Gemma 2 is trained on code from multiple programming languages (Python, JavaScript, Java, C++, Go, Rust, etc.) and can generate syntactically correct code snippets, complete functions, and debug code based on natural language descriptions or partial code context. The model uses instruction-following to understand code-specific requests like 'write a function that sorts an array' or 'fix the bug in this code', and maintains language-specific syntax through learned patterns. Performance is strongest in Python and JavaScript (languages with abundant training data) and weaker in niche languages like Rust or Kotlin.

Gemma 2 can adapt to new tasks by including examples in the prompt (few-shot learning), where 2-5 examples of input-output pairs teach the model to perform classification, extraction, or generation tasks without fine-tuning. This works through the model's instruction-following capability and learned ability to recognize patterns from examples. The mechanism relies on the transformer's ability to attend to example patterns and apply them to new inputs, with performance improving as more examples are provided (up to a point where context becomes too long).

Provides fully open-source model weights, training code, and documentation enabling researchers and developers to understand the model architecture, reproduce training procedures, and fine-tune for custom tasks. The model uses standard transformer architecture with published modifications (interleaved attention), allowing integration into existing ML frameworks and research pipelines. Open weights enable local deployment without API dependencies and support for custom quantization, pruning, and fine-tuning.