Prompt Guard

Prompt Guard implements a specialized binary classification model that analyzes raw user input text to detect prompt injection attacks and jailbreak attempts before they reach the target LLM. The classifier operates as a preprocessing filter, examining input tokens against learned patterns of adversarial prompt structures without requiring full prompt context or conversation history. It uses a compact model architecture optimized for low-latency inference suitable for real-time API gateway deployment.

Prompt Guard leverages CyberSecEval's multilingual prompt injection benchmark dataset, which includes machine-translated versions of attack prompts across multiple languages. The model learns to recognize injection patterns that persist across language boundaries, enabling detection of non-English jailbreak attempts without requiring separate language-specific classifiers. This approach uses a single unified model that generalizes adversarial prompt structures across linguistic variations.

Prompt Guard operates as a pluggable scanner component within LlamaFirewall's modular security architecture. LlamaFirewall coordinates multiple safeguard models (Prompt Guard for input filtering, Llama Guard for output moderation, CodeShield for code safety) through a unified configuration and execution pipeline. Prompt Guard receives input tokens from the framework's preprocessing stage, executes classification, and returns verdicts that feed into LlamaFirewall's decision logic for accepting, blocking, or quarantining requests.

Prompt Guard's performance is measured using CyberSecEval v2's comprehensive prompt injection test suite, which includes MITRE-based attack patterns, textual injection techniques, and false refusal rate (FRR) measurements. The benchmark framework executes Prompt Guard against curated adversarial prompt datasets, measuring detection accuracy, false positive rates, and performance across attack categories. This enables quantitative comparison of Prompt Guard's robustness against known injection techniques and assessment of its real-world effectiveness.

Prompt Guard is architected as a compact binary classifier optimized for low-latency inference suitable for deployment in API gateway environments. The model uses efficient neural network architectures (likely transformer-based with reduced layer depth or width) and supports multiple inference backends (PyTorch, ONNX, vLLM) to minimize computational overhead. Inference latency is designed to be sub-50ms on CPU, enabling synchronous preprocessing of user inputs without blocking LLM request handling.

Prompt Guard is designed to work in tandem with Llama Guard, Meta's output moderation model, creating a bidirectional security architecture. Prompt Guard filters malicious inputs before they reach the LLM, while Llama Guard filters unsafe outputs before they reach users. Both models are integrated into the Purple Llama safeguard ecosystem and can be orchestrated together through LlamaFirewall, enabling comprehensive coverage of both input and output attack surfaces. The two models use complementary detection approaches optimized for their respective positions in the request/response pipeline.

Prompt Guard's binary classification architecture supports fine-tuning on custom datasets to adapt detection to domain-specific prompt injection patterns. Organizations can augment the base model with examples of attacks relevant to their specific LLM application (e.g., financial fraud prompts for banking, medical misinformation for healthcare). Fine-tuning leverages transfer learning from the base model's pre-trained weights, requiring significantly less data than training from scratch while maintaining performance on general injection patterns.

Prompt Guard outputs a confidence score (0.0-1.0) alongside its binary safe/unsafe classification, enabling risk-based decision logic beyond simple accept/reject. Applications can use confidence scores to implement tiered security responses: high-confidence unsafe inputs are blocked immediately, low-confidence borderline inputs are quarantined for human review, and high-confidence safe inputs proceed normally. This approach reduces false positives by allowing human-in-the-loop review for ambiguous cases rather than blocking all uncertain inputs.