WildGuard

Classifies incoming prompts across multiple harm categories (e.g., violence, illegal activity, sexual content, hate speech, self-harm) using a fine-tuned language model trained on diverse adversarial examples. The model learns to recognize harmful intent patterns across different risk domains simultaneously, enabling nuanced detection beyond binary safe/unsafe classification. Outputs confidence scores per harm category to support downstream risk-based routing decisions.

Analyzes LLM-generated responses to identify harmful content that slipped past prompt filtering, classifying violations across the same harm taxonomy as prompt detection. Uses a separate classification head trained on model outputs paired with human safety judgments, enabling detection of harmful content generation even when the initial prompt appeared benign. Supports both full-response analysis and streaming token-level detection for real-time filtering.

Identifies when an LLM refuses to answer a prompt and classifies the refusal reason (safety concern, capability limitation, policy violation, etc.) using a specialized classifier trained on refusal patterns. This enables distinguishing between legitimate refusals (model correctly declining harmful requests) and false refusals (model unnecessarily blocking benign requests), supporting both safety auditing and user experience optimization. Outputs refusal confidence and category to enable downstream handling (e.g., rephrasing suggestions, escalation).

Provides a structured dataset of 10K+ adversarial prompts spanning 13 harm categories, each annotated by human raters for prompt harmfulness, response harmfulness, and refusal appropriateness. The dataset includes diverse attack patterns (jailbreaks, prompt injections, social engineering) and edge cases, enabling researchers and builders to train, evaluate, and benchmark safety classifiers. Supports both supervised fine-tuning of safety models and evaluation of existing LLM safety mechanisms.

Provides a fine-tuned language model (based on Llama 2 or similar backbone) trained via multi-task learning to simultaneously predict prompt harmfulness, response harmfulness, and refusal appropriateness. The model uses shared representations for all three tasks, enabling efficient inference and transfer learning across safety dimensions. Available in multiple sizes (7B, 13B parameters) to support different latency/accuracy trade-offs in production deployments.

Defines a structured taxonomy of 13 harm categories (violence, illegal activity, sexual content, hate speech, self-harm, etc.) with clear definitions and annotation guidelines for consistent human labeling. The schema supports multi-label annotation (a single prompt can belong to multiple categories) and confidence scoring, enabling nuanced safety classification beyond binary safe/unsafe decisions. Includes inter-rater agreement metrics and quality control procedures for maintaining annotation consistency.

Provides standardized evaluation metrics and benchmark results for safety classifiers, including precision, recall, F1-score, and ROC-AUC across all 13 harm categories. Enables comparison of different safety approaches (API-based, fine-tuned models, rule-based systems) on a common test set with consistent evaluation methodology. Includes ablation studies showing the contribution of different training techniques (multi-task learning, data augmentation, etc.) to overall performance.

Provides training scripts, loss functions, and hyperparameter configurations for fine-tuning the WildGuard base model on domain-specific safety concerns with minimal labeled data. Implements techniques like low-rank adaptation (LoRA), data augmentation, and curriculum learning to improve sample efficiency and reduce overfitting. Includes evaluation utilities for monitoring validation performance and early stopping to prevent degradation on the original safety tasks.

Provides pre-built integrations with popular LLM inference frameworks (vLLM, TGI, Ollama) and API providers (OpenAI, Anthropic, Together AI) to enable seamless safety classification in production pipelines. Supports both synchronous and asynchronous inference, batching for throughput optimization, and caching to reduce redundant classifications. Includes middleware for intercepting prompts/responses and applying safety filters before delivery to users.