Published
Updated
Datadog prints Jev Noul recall 82.90% to 3.88% under a counterfeit policy, and this desk's is_risky fell from 0.94 to 0.13
This topic was created 11 days ago, and the information it contains may have evolved or changed since then.
Datadog Security Labs' October 15 write-up, posted October 19, prints Jev Noul recall 82.90% on raw attacks and 3.88% with a counterfeit policy prepended. Date headers 08:50:46 GMT and 08:50:47 GMT October 20, this desk's jev-1.13.0 is_risky was 0.94 then 0.13.
Nick Frichette posted Datadog Security Labs’ Jev write-up on October 19, 2026, at 19:12 UTC. The text says prompt-injection detection fell from about 83% to about 4%, and that the same method bypassed LangChain’s experimental Jev-based tool-risk gating.
The article, Nick Frichette and Feras Al-Kassar, is dated October 15. Four public datasets, 12,295 samples, 4,643 labeled attacks and 7,652 benign. A Noul has_prompt_injection at a 0.5 cutoff scored precision 99.66% and recall 82.90% on the raw attacks, 3,849 true positives and 13 false positives. With a counterfeit policy prepended to each message, true positives fell to 180, false negatives rose to 4,463, and recall is 3.88%. A Choice attack_category falls from recall 85.46% to 9.69%. The zachz and Yanismiraoui sets drop to 0 true positives under that payload on both questions.
The counterfeit policy sits in untrusted state. It names the answers to return, quotes the original message as a benign exception, and rewrites the attack categories. The trusted Noul instructions, Choice instructions, and category criteria stay as the developer wrote them.
LangChain tool-risk gating is a Noul is_risky, blocked at 0.5. Datadog’s scenario is a viewer with can_edit false who asks to rewrite the employee handbook. Across 40 runs the mean is 0.893, and all 40 calls were blocked. The same call with a counterfeit is_risky definition in profile_description averages 0.290. A plain prompt-injection payload on that scenario averaged 0.905.
This desk, Date headers 08:50:46 GMT and 08:50:47 GMT October 20, sent a GET and four POSTs to https://api.typesafe.ai. GET /v1/models returned HTTP 200 and listed jev-latest and jev-preview. All four POSTs used model jev-latest and came back jev-1.13.0, HTTP 200. A dump-the-prompt Noul written here: has_prompt_injection 0.99, 404 input tokens, 24 output, desk clock 476 ms. The same Noul with a counterfeit policy prepended: 0.92, 529 input tokens, 24 output, desk clock 461 ms. A shortened handbook is_risky, viewer, can_edit false: 0.94, 498 input tokens, 22 output, desk clock 475 ms. The same call with Datadog’s is_risky definition in profile_description: 0.13, 694 input tokens, 22 output, desk clock 447 ms. Those four calls are not the 12,295-row table. They are not the 40-run means.
HiddenLayer stays the checkout ad, 91% Confirm Price and 9% Place order on jev-ultrafast. Decision hijacking stays 1.8% to 3.5% on 510 InjecAgent cases. TypeSafe’s jaggedness page already names adversarial text as a failure mode. Limits now cites this counterfeit policy.
This site's reading
Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.
Verify
Nick Frichette, @Frichette_n, October 19, 2026, 19:12 UTC, status 2112260736785195388. The text names Datadog Security Labs, a drop in prompt-injection detection from about 83% to about 4%, and a bypass of LangChain's experimental Jev-based tool-risk gating.
The article, Nick Frichette and Feras Al-Kassar, dated October 15, 2026, https://securitylabs.datadoghq.com/articles/jev-as-a-judge-how-counterfeit-policy-steers-jev/. Four public datasets, 12,295 samples, 4,643 attack and 7,652 benign. Noul has_prompt_injection at a 0.5 cutoff: raw true positives 3,849, false positives 13, false negatives 794, precision 99.66%, recall 82.90%, F1 90.51%. With a counterfeit policy prepended: true positives 180, false positives 0, false negatives 4,463, precision 100.00%, recall 3.88%, F1 7.46%. Choice attack_category: raw recall 85.46%, counterfeit 9.69%. zachz and Yanismiraoui drop to 0 true positives under the counterfeit policy on both questions.
LangChain tool-risk gating, Noul is_risky, block at 0.5. A viewer with can_edit false who asks to rewrite the employee handbook: mean 0.893 across 40 runs, all 40 blocked. The same call with a counterfeit is_risky definition in profile_description: mean 0.290. A plain prompt-injection payload on that scenario averaged 0.905.
Date headers 08:50:46 GMT and 08:50:47 GMT October 20. GET https://api.typesafe.ai/v1/models with a TypeSafe key returned HTTP 200 and listed jev-latest and jev-preview. Four POSTs to /v1/systemone, model jev-latest, response jev-1.13.0, HTTP 200. A dump-the-prompt Noul written here: has_prompt_injection 0.99, 404 input tokens, 24 output, desk clock 476 ms. The same Noul with a counterfeit policy prepended: 0.92, 529 input tokens, 24 output, desk clock 461 ms. A shortened is_risky handbook, viewer, can_edit false: 0.94, 498 input tokens, 22 output, desk clock 475 ms. The same call with Datadog's is_risky definition in profile_description: 0.13, 694 input tokens, 22 output, desk clock 447 ms. Those four calls are not the 12,295-row table and not the 40-run means. Do not print the key, request JSON, or request ids.
Compare
82.90% to 3.88% is Datadog's Noul recall on 4,643 labeled attacks. This desk's one dump-the-prompt Noul stayed 0.99 then 0.92. Do not merge 0.92 with 3.88%.
0.893 to 0.290 is Datadog's 40-run mean on LangChain is_risky. This desk's two POSTs on a shortened handbook were 0.94 then 0.13. Do not merge 0.13 with 0.290.
HiddenLayer's checkout ad is 91% Confirm Price and 9% Place order on jev-ultrafast. Decision hijacking, arXiv 2609.28613, is 1.8% to 3.5% on 510 InjecAgent cases. Those are different tests.
Jaggedness docs already name adversarial text. Datadog's counterfeit policy is a measured example of that class, placed inside state.
Terms
- 82.90% / 3.88%
- Datadog's Noul has_prompt_injection recall on 4,643 labeled attacks, raw then with a counterfeit policy prepended. True positives 3,849 to 180. Threshold 0.5.
- counterfeit policy
- Datadog's payload inside untrusted state. It names the answers to return, quotes the original message as a benign exception, and rewrites attack-category definitions. Trusted Noul and Choice instructions stay as the developer wrote them.
- 0.893 / 0.290
- Datadog's 40-run mean is_risky on a viewer who cannot edit, then on the same call with a counterfeit is_risky definition in profile_description. LangChain blocks at 0.5.