Method

What follows is the whole basis for every number on this site. If a figure cannot answer these questions, it does not get published.

Lanes

Each attack runs three times against the same model, changing only the defense.

• No defender. The reference: what the model does unprotected.
• Light. Tier 1 pattern detection plus the Tier 2 classifier, exactly as shipped in @stackone/defender. This is what you get by installing it.
• Deep. Deep Scan (Tier 3) on its own, reading every tool result and blocking on its own decision. It does not run behind the package: it replaces it, so the two lanes are separate measurements, not one stacked on the other. Hybrid, which sends only the results Tier 2 is unsure about to Deep Scan, is not measured here.

We never collapse these into a single "with Defender" series. Two of the three are different products, and merging them would either flatter the free package or hide what the subscription is for.

The metric

We report whether the harmful action actually executed, not whether a judge model called the attack a success.

The reason is mechanical. The harm fires the moment the model calls the tool with arguments that satisfy the action's gate. A defense that blocks the tool's result after that point hides the confirmation from the model, but the action already happened. A judge reading the conversation sees a blocked result and scores the attack as failed, which undercounts real harm. So the measurement reads the tool call, not the returned result.

Cohorts

Attacks are grouped by shape, and never pooled. Difficulty varies widely between shapes, so one combined average would move with the mix rather than with the defense.

• Single-read injection. The payload arrives in one tool result.
• Multi-step escalation. Harm requires a chain of calls.
• Split payload. The injection is fragmented across separate reads, so no single result carries a self-sufficient payload.

That last shape is where a per-result classifier reaches a structural limit: it cannot see an injection that was never assembled into one result. A model that reads the result in context is the answer to it. How much that recovers in practice is measured on the tool-attack corpus, which is not published yet.

Exclusions

• Individual scenarios. Per-scenario results, including which attacks succeed against which models, amount to a ranked target list. We publish aggregates.
• Detector-only numbers. Scoring a payload in isolation, with no model in the loop, overstates evasion. Every figure here comes from a full run.
• Anything measured on a corpus we no longer trust. A benchmark is only as good as the validity of its scenarios, and a scenario whose "harm" does not actually cause harm makes a defense look better or worse than it is.
• False-positive rates without naming the benign corpus. Synthetic and real-world benign traffic give very different answers. The corpus is stated on the axis.