Research
Results, measured on tests the detector never saw.
Ghost P is the current edition of Ulterior Safety’s detector. It reads a model’s internal activity to flag harmful requests. Here it runs on a model whose safety training had been removed, so the model rarely refuses by itself. Every figure below comes from public tests kept out of training, and every figure names its benchmark and its false-alarm bar.
01In short
What the results say.
It catches most clearly harmful requests, with few false alarms. On 1,259 clearly harmful prompts, Ghost P caught 95.4% while flagging 1% of safe prompts.
It is ahead of two leading open guard models on the same prompts: Llama Guard 3 8B caught 89.2% and ShieldGemma 9B 46.6% at the same 1% bar.
It does not rely on the model’s own safety training. The model it monitors had that training removed. Compared with the original model’s built-in refusals, it gives nearly the same protection with far fewer wrongly refused questions.
The evidence has limits. One model family, no external audit yet, and on two smaller comparisons Llama Guard 3 was ahead. These are set out below.
02Headline results
Ghost P, held out, at 1% false alarms.
One bar for everything: the detector’s threshold is set so that 1% of 3,398 pooled safe prompts are flagged. The rates below are what that single threshold gives on each test set.
95.4%
of clearly harmful requests caught
95.8%
of plain harmful requests caught on JailbreakBench
95.5%
of harmful prompts caught on XSTest
0%
of everyday prompts flagged
| Test set | Harmful requests caught | Safe prompts flagged |
|---|---|---|
| Clearly harmful set 1,259 prompts; pooled safe prompts for the false-alarm rate | 95.4% | 1% |
| JailbreakBench, plain requests Harmful behaviours; JailbreakBench benign prompts | 95.8% | 5.0% |
| XSTest Harmful prompts; XSTest safe prompts | 95.5% | 2.4% |
| Everyday prompts Ordinary questions and tasks | n/a | 0% |
The JailbreakBench benign and XSTest safe prompts are written to sound harmful while being harmless, so more of them are flagged than everyday prompts. Benchmarks: JailbreakBench and XSTest.
03Against industry guards
Same prompts, same bar.
The two guard models were run locally, off the shelf, on the same 1,259 clearly harmful prompts, with their thresholds set to the same 1% false-alarm bar on the same 3,398 pooled safe prompts.
| Detector | Caught |
|---|---|
| Ghost P Ulterior Safety | 95.4% |
| Llama Guard 3 8B Meta, open | 89.2% |
| ShieldGemma 9B Google, open | 46.6% |
| Closed detectors | Not measured (no access) |
Caveats: the strict set drops items Ghost P itself found hard (it is an upper bound for Ghost P); Ghost P’s bar and training used related data, the guards are off the shelf; so this favours Ghost P somewhat.
This is not an official leaderboard; the test set is ours.
04Against built-in safety
Nearly the same protection, far fewer wrongly refused questions.
The original model, with its safety training intact, compared with Ghost P on a model with safety removed. Same benchmark prompts.
| Benchmark | Setup | Harmful requests blocked | Safe questions wrongly refused or flagged |
|---|---|---|---|
| JailbreakBench, plain | Original model | 97% (better) | 31% |
| Safety removed, with Ghost P | 95.8% | 5.0% (better) | |
| XSTest | Original model | 99.5% (better) | 23.6% |
| Safety removed, with Ghost P | 95.5% | 2.4% (better) |
The original model blocks slightly more, but refuses roughly one in four to one in three of the safe questions. Ghost P blocks nearly as much on a model whose own refusals have been removed, and flags far fewer safe questions.
“Nearly the same protection, with far fewer wrongly refused questions, on a model with safety removed.”
05Limits
What these results do not show.
-
On smaller overlap sets, Llama Guard 3 was ahead.
Using each benchmark’s own benign prompts to set the bar, on the prompts both detectors could be scored on: JailbreakBench plain, Llama Guard 3 88.9% versus Ghost P 76.4% (n=72); XSTest, Llama Guard 3 96.6% versus Ghost P 93.3% (n=89). These sets are small and the gaps are within noise, but they are the honest counterweight to the headline table.
-
Only one model family has been tested.
All results are on Qwen3.5-9B. Other model families have not been measured yet.
-
No external audit yet.
The results have not been independently reproduced. They are reproducible from logs, available on request.
-
The test set is ours.
The clearly harmful set was assembled and filtered by Ulterior Safety from held-out harmful prompts. It is not an official leaderboard.
-
Deception is not measured.
Catching deception and scheming is a research direction. No result on this page is about deception.
06Method, in plain words
How the numbers were measured.
-
What the detector reads
Ghost P reads the model’s internal activity while it processes a request. How it does so is not published.
-
Held out
The test prompts were kept aside: the detector never trained on them. They include the public benchmarks JailbreakBench and XSTest.
-
The 1% false-alarm bar
A detector can catch everything by flagging everything. So each detector’s threshold is fixed first, so that it flags 1% of 3,398 pooled held-out safe prompts, and only then is its catch rate measured.
-
Why “clearly harmful”
Public benchmarks contain borderline items whose labels people disagree on. The clearly harmful set keeps only prompts whose harmful label was confirmed by a second, stronger judge model.
-
What “safety training removed” means
Open models can be modified so that they stop refusing harmful requests. Ghost P was tested on such a modified copy of Qwen3.5-9B. Every prompt in the clearly harmful set was one this modified model went on to answer, so the protection measured comes from the detector, not from the model refusing.
07Progress
Each detector, in order.
Each edition is scored on public tests it never trained on, at 1% false alarms unless stated.
-
First
Prototype
The first detector, used together with Llama Guard 3.
- JailbreakBench jailbreaks
- 89.2% at 1% false alarms; 96.7% at 5%
- XSTest
- 78.5% at 1% false alarms; 90.3% at 5%
-
Current
Ghost P
Runs on its own, on a model with safety training removed.
- Clearly harmful set
- 95.4% at 1% false alarms
- JailbreakBench plain / XSTest
- 95.8% / 95.5% at 1% false alarms
-
Next
Next edition
In progress. Its results will be added here once measured on the same held-out tests.
Contact
Talk to Ulterior Safety.
Investors, AI labs, teams deploying AI and press: we would like to hear from you.