Defense Benchmark

This page measures Defender's detector: on its own, with the model out of the loop, how well it separates attacks from ordinary text and what it costs to run; then back in the loop, as the attack-success reduction it produces per model. For the undefended attack picture, see the Attack Benchmark. Every figure names its benchmark and metric. Read the method.

Detector comparisons

0.000.250.500.751.002.55102040latency per call, milliseconds (log)F1 Meta PromptGuard v1: F1 0.203, 43 ms, 1064 MB, T4 GPU Meta PromptGuard v1 Meta PromptGuard v2: F1 0.041, 43 ms, 1064 MB, T4 GPU Meta PromptGuard v2 ProtectAI DeBERTa v3: F1 0.789, 43 ms, 704 MB, T4 GPU ProtectAI DeBERTa v3 NVIDIA NemoGuard JailbreakDetect: F1 0.311, 5.9 ms, 563 MB, A10G GPU NVIDIA NemoGuard JailbreakDetect StackOne Defender, Tier 2: F1 0.944, 2.48 ms, 22 MB, CPU (ONNX int8) StackOne Defender, Tier 2
Tier 2 against every external classifier we could run on InjecAgent, 1,071 tool results. Up is better, left is cheaper, and the circle is the model on disk.

Ours is 0.16 F1 above the next best, at 17 times less latency and 32 times smaller, on a CPU rather than a GPU.

0.000.250.500.751.000%10%20%30%40%50%60%70%false positives, share of benign samplesF1 ProtectAI DeBERTa v3: F1 0.736, 61.75% false positives ( 20,452 of 33,121) ProtectAI DeBERTa v3 Meta PromptGuard v2: F1 0.625, 2.96% false positives ( 980 of 33,121) Meta PromptGuard v2 NVIDIA NemoGuard JailbreakDetect: F1 0.582, 44.23% false positives ( 14,649 of 33,121) NVIDIA NemoGuard JailbreakDetect Meta PromptGuard v1: F1 0.547, 66.76% false positives ( 22,112 of 33,121) Meta PromptGuard v1 StackOne Defender, Tier 2: F1 0.943, 4.76% false positives ( 1,577 of 33,121) StackOne Defender, Tier 2
Tier 2 at a threshold of 0.8 against the same classifiers on jayavibhav/prompt-injection, 32,294 attacks against 33,121 pieces of ordinary text. Up is better, left is better.

The best of them on quality gives up 62% of ordinary text to get there. The one with a false-positive rate near ours catches 34% less.

Every rate here divides by the same 33,121 benign samples. Competitors are each at their own operating point, which is the only one they expose; ours is one of four in the table below. Measured on 0.7.0.

Tier 2: the bundled classifier

The classifier that scores every tool result, against other detectors small enough to run inline.

Footprint

What it costs to run on every tool result. Hardware is listed because a CPU figure and a GPU figure are not comparable without it.

DetectorSizeLatencyHardware
StackOne Defender, Tier 222 MB2.48 msCPU (ONNX int8)
NVIDIA NemoGuard JailbreakDetect563 MB5.9 msA10G GPU
ProtectAI DeBERTa v3704 MB43 msT4 GPU
Meta PromptGuard v11.0 GB43 msT4 GPU
Meta PromptGuard v21.0 GB43 msT4 GPU

jayavibhav/prompt-injection65,000 samples, 32,294 attack, 33,121 benign

00.250.50.751F1 (higher is better)StackOne Defender: F1 0.944, 5.6% false positivesStackOne Defender0.945.6% FPProtectAI DeBERTa v3: F1 0.736, 61.8% false positivesProtectAI DeBERTa v30.7461.8% FPMeta PromptGuard v2: F1 0.625, 3.0% false positivesMeta PromptGuard v20.633.0% FPNVIDIA NemoGuard JailbreakDetect: F1 0.582, 44.2% false positivesNVIDIA NemoGuard JailbreakDetect0.5844.2% FPMeta PromptGuard v1: F1 0.547, 66.8% false positivesMeta PromptGuard v10.5566.8% FP
F1 for each detector, with its false-positive rate on the right. Longer is better; ours is highlighted.
Every operating point
DetectorThresholdF1False positives
ProtectAI DeBERTa v30.73620452 of 33121 (61.8%)
Meta PromptGuard v20.625980 of 33121 (3.0%)
NVIDIA NemoGuard JailbreakDetect0.58214649 of 33121 (44.2%)
Meta PromptGuard v10.54722112 of 33121 (66.8%)

Ours is configurable, so it is not one point but a curve. Raise the threshold and both recall and false positives fall; read off the false-positive rate you would accept.

0%25%50%75%100%0.300.500.700.80block thresholdRecall at threshold 0.3: 97%Recall at threshold 0.5: 96%Recall at threshold 0.7: 95%Recall at threshold 0.8: 94%RecallFalse positives at threshold 0.3: 10%False positives at threshold 0.5: 7%False positives at threshold 0.7: 6%False positives at threshold 0.8: 5%False positives
Recall and false-positive rate for our detector across thresholds.
Every threshold
ThresholdRecallF1False positives
0.3096.9%0.9373,162 of 33,121 (9.5%)
0.5095.7%0.9412,471 of 33,121 (7.5%)
0.7094.5%0.9441,871 of 33,121 (5.6%)
0.8093.6%0.9431,577 of 33,121 (4.8%)

xxz224/prompt-injection-attack-dataset22,500 samples, 18,735 attack, 3,747 benign

00.250.50.751F1 (higher is better)Meta PromptGuard v1: F1 0.924, 82.2% false positivesMeta PromptGuard v10.9282.2% FPStackOne Defender: F1 0.846, 15.7% false positivesStackOne Defender0.8515.7% FPNVIDIA NemoGuard JailbreakDetect: F1 0.770, 14.3% false positivesNVIDIA NemoGuard JailbreakDetect0.7714.3% FPMeta PromptGuard v2: F1 0.665, 0.1% false positivesMeta PromptGuard v20.660.1% FPProtectAI DeBERTa v3: F1 0.331, 4.5% false positivesProtectAI DeBERTa v30.334.5% FP
F1 for each detector, with its false-positive rate on the right. Longer is better; ours is highlighted.
Every operating point
DetectorThresholdF1False positives
Meta PromptGuard v10.9243079 of 3747 (82.2%)
NVIDIA NemoGuard JailbreakDetect0.770535 of 3747 (14.3%)
Meta PromptGuard v20.6655 of 3747 (0.1%)
ProtectAI DeBERTa v30.331170 of 3747 (4.5%)

Ours is configurable, so it is not one point but a curve. Raise the threshold and both recall and false positives fall; read off the false-positive rate you would accept.

0%25%50%75%100%0.300.500.700.80block thresholdRecall at threshold 0.3: 76%Recall at threshold 0.5: 72%Recall at threshold 0.7: 68%Recall at threshold 0.8: 65%RecallFalse positives at threshold 0.3: 16%False positives at threshold 0.5: 13%False positives at threshold 0.7: 10%False positives at threshold 0.8: 9%False positives
Recall and false-positive rate for our detector across thresholds.
Every threshold
ThresholdRecallF1False positives
0.3075.6%0.846587 of 3,747 (15.7%)
0.5071.7%0.823492 of 3,747 (13.1%)
0.7067.8%0.798383 of 3,747 (10.2%)
0.8065.3%0.781334 of 3,747 (8.9%)

InjecAgent1,071 samples, 1,054 attack, 17 benign

00.250.50.751F1 (higher is better)StackOne Defender: F1 0.944, 5.9% false positivesStackOne Defender0.945.9% FPProtectAI DeBERTa v3: F1 0.789, 58.8% false positivesProtectAI DeBERTa v30.7958.8% FPNVIDIA NemoGuard JailbreakDetect: F1 0.311, 0.0% false positivesNVIDIA NemoGuard JailbreakDetect0.310.0% FPMeta PromptGuard v1: F1 0.203, 0.0% false positivesMeta PromptGuard v10.200.0% FPMeta PromptGuard v2: F1 0.041, 0.0% false positivesMeta PromptGuard v20.040.0% FP
F1 for each detector, with its false-positive rate on the right. Longer is better; ours is highlighted.
Every operating point
DetectorThresholdF1False positives
StackOne Defender, Tier 20.500.9441 of 17 (5.9%)
StackOne Defender, Tier 20.640.9231 of 17 (5.9%)
StackOne Defender, Tier 20.800.8961 of 17 (5.9%)
ProtectAI DeBERTa v30.78910 of 17 (58.8%)
NVIDIA NemoGuard JailbreakDetect0.3110 of 17 (0.0%)
Meta PromptGuard v10.2030 of 17 (0.0%)
Meta PromptGuard v20.0410 of 17 (0.0%)

Tier 3: Deep Scan

A larger model reviewing what the classifier cannot settle alone. Every detector below was run on the same benchmark, so the small inline classifiers appear here too.

Footprint

What it costs to run on every tool result. Hardware is listed because a CPU figure and a GPU figure are not comparable without it.

DetectorSizeLatencyHardware
NVIDIA NemoGuard JailbreakDetect563 MB5.9 msA10G GPU
ProtectAI DeBERTa v3704 MB43 msT4 GPU
Meta PromptGuard v11.0 GB43 msT4 GPU
Meta PromptGuard v21.0 GB43 msT4 GPU
StackOne Defender, Tier 32.0 GB150 msL4 GPU
NVIDIA NemoGuard ContentSafety 8B15.0 GB448.5 msA10G GPU

Microsoft BIPIA400 samples, 200 attack, 200 benign

00.250.50.751F1 (higher is better)StackOne Defender: F1 0.920, 8.0% false positivesStackOne Defender0.928.0% FPMeta PromptGuard v1: F1 0.833, 18.5% false positivesMeta PromptGuard v10.8318.5% FPNVIDIA NemoGuard JailbreakDetect: F1 0.348, 18.5% false positivesNVIDIA NemoGuard JailbreakDetect0.3518.5% FPProtectAI DeBERTa v3: F1 0.266, 17.5% false positivesProtectAI DeBERTa v30.2717.5% FPNVIDIA NemoGuard ContentSafety 8B: F1 0.049, 0.5% false positivesNVIDIA NemoGuard ContentSafety 8B0.050.5% FPMeta PromptGuard v2: F1 0.010, 0.0% false positivesMeta PromptGuard v20.010.0% FP
F1 for each detector, with its false-positive rate on the right. Longer is better; ours is highlighted.
Every operating point
DetectorThresholdF1False positives
StackOne Defender, Tier 30.900.92016 of 200 (8.0%)
StackOne Defender, Tier 30.950.9184 of 200 (2.0%)
StackOne Defender, Tier 30.800.90730 of 200 (15.0%)
Meta PromptGuard v10.83337 of 200 (18.5%)
NVIDIA NemoGuard JailbreakDetect0.34837 of 200 (18.5%)
ProtectAI DeBERTa v30.26635 of 200 (17.5%)
NVIDIA NemoGuard ContentSafety 8B0.0491 of 200 (0.5%)
Meta PromptGuard v20.0100 of 200 (0.0%)

Per-model reduction

The charts above measure the detector on its own. This one puts it back in the loop: the same frontier models from the Attack Benchmark, run as agents, with Light and Deep in front of them. Each row is a model's attack success rate undefended, then behind each tier. Lower is better.

020406080attacks reaching the harmful action, percent Gemini 3.5 Flash: 82% undefended, 64% Light, 25% Deep Gemini 3.5 Flash64%25%82% GPT-5.4 mini: 68% undefended, 54% Light, 21% Deep GPT-5.4 mini54%21%68% GPT-5.6 (luna): 58% undefended, 46% Light, 16% Deep GPT-5.6 (luna)46%16%58% GPT-5.6 (terra): 40% undefended, 31% Light, 12% Deep GPT-5.6 (terra)31%12%40% GPT-5.5: 38% undefended, 28% Light, 14% Deep GPT-5.528%14%38% GPT-5.6 (sol): 35% undefended, 27% Light, 8% Deep GPT-5.6 (sol)27%8%35% Claude Haiku 4.5: 33% undefended, 27% Light, 12% Deep Claude Haiku 4.527%12%33% GLM 5.2: 33% undefended, 25% Light, 12% Deep GLM 5.225%12%33% Gemini 3.6 Flash: 28% undefended, 23% Light, 10% Deep Gemini 3.6 Flash23%10%28% Gemini 3.7 Flash: 23% undefended, 20% Light, 9% Deep Gemini 3.7 Flash20%9%23% Qwen3.8 Flash: 23% undefended, 19% Light, 10% Deep Qwen3.8 Flash19%10%23% GLM 5.3: 22% undefended, 18% Light, 8% Deep GLM 5.318%8%22% Gemini 3.8 Flash: 21% undefended, 16% Light, 7% Deep Gemini 3.8 Flash16%7%21% Claude Sonnet 4.6: 9% undefended, 8% Light, 3% Deep Claude Sonnet 4.68%3%9% Claude Sonnet 5: 4% undefended, 4% Light, 1% Deep Claude Sonnet 54%1%4% Claude Opus 5: 3% undefended, 2% Light, 1% Deep Claude Opus 52%1%3% Claude Opus 4.8: 3% undefended, 3% Light, 2% Deep Claude Opus 4.83%2%3%
Every model run as a tool-using agent against the attack library. Each mark is the share of attacks that reached the harmful action. Lower is better.