Instructions to use protectai/deberta-v3-base-prompt-injection-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use protectai/deberta-v3-base-prompt-injection-v2 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="protectai/deberta-v3-base-prompt-injection-v2")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("protectai/deberta-v3-base-prompt-injection-v2") model = AutoModelForSequenceClassification.from_pretrained("protectai/deberta-v3-base-prompt-injection-v2", device_map="auto") - Inference
- Notebooks
- Google Colab
- Kaggle
Benchmark on AgentDojo: flags 27/27 attacks alone, 23% once embedded in tool output
I benchmarked this model alongside 9 other open-source injection detectors on
AgentDojo: 629 indirect prompt injections embedded in realistic tool output (emails,
bills, reviews, Slack messages), plus 97 benign tool outputs.
The interesting result is context dilution:
| Setting | Attacks caught | False alarms |
|---|---|---|
| Each of the 27 distinct attack texts, scored alone | 27 / 27 | β |
| Same attacks embedded in tool output (threshold 0.5) | 145 / 629 (23%) | 4 / 97 |
| LLM Guard with this model, shipped threshold 0.92 | 124 / 629 (20%) | 2 / 97 |
| Threshold tuned to a 2% false-alarm budget, measured on unseen AgentDojo domains | 132 / 629 (21%) | 3 / 97 |
So the model recognises every attack on its own, but once the same text sits after a
normal bill or email, the surrounding benign content pulls the score down. Unlike some
other detectors, changing the threshold doesn't fix it; the embedded attacks rank
among the benign outputs.
For anyone using it to scan tool results in an agent, scoring shorter chunks or
sentences separately might help. I haven't tested that for this model yet, so treat
it as a hypothesis.
Caveat: every AgentDojo attack uses the same wrapper template, so these numbers are
for one attack style, not all injections.
Code, raw scores and a one-command rerun: https://github.com/rudratoshs/buried-injections