Adversarial Benchmark
97/97 adversarial attack variants detected across 8 categories with zero false positives, plus a 101/101 service-hardening battery. Self-authored Garak-style probing; independent audit welcomed.
F1 Score
Perfect precision & recall
Attacks Detected
Across 8 adversarial categories
Specificity
0/25 false positives
Inline Latency
SDK-level detection speed
1.0000
Every detection was correct
1.0000
Every attack was detected
100.0%
97/97 adversarial tests
Each category tests a distinct attack vector. All categories achieved 100% detection.
30 test prompts
11 test prompts
12 test prompts
7 test prompts
12 test prompts
19 test prompts
6 test prompts
25 safe, legitimate inputs were correctly allowed without any blocking.
Beyond raw detection accuracy, every security service is hardened against end-to-end attack scenarios — agent firewall bypass, passport forgery, delegation abuse, egress exfiltration, evidence tampering, and more. The battery exits non-zero if any scenario regresses.
Scenarios passing
100% across 21 services
Services covered
Guard, agents, identity, evidence, SIEM
Failing scenarios
Run June 29, 2026 at 12:00 AM UTC
Command: npx tsx tests/comprehensive-adversarial-test-battery.ts — 101/101 passing. Measures service hardening, distinct from the F1 detection benchmark above.
Recorded API-level latency including HTTP overhead. No separate inline SDK latency was captured by this benchmark.
p50 (Median)
891ms
Adversarial probes
p95
1656ms
Adversarial probes
p99
2719ms
Adversarial probes
97 adversarial prompts across 8 categories (prompt injection, jailbreak/DAN, encoding/obfuscation, multilingual, indirect injection, PII, secrets, unsafe output) were sent to Soter's /api/guard/analyze endpoint. 25 safe inputs were included for false-positive verification.
Try the interactive playground, then protect both sides of your model call.
Full benchmark results available at /api/benchmarks (JSON). Source: View JSON results
