Calibrating SusFactor for Your Traffic

A practical guide to tuning how aggressively SusFactor blocks prompts, so it catches attacks without getting in your real users' way. No statistics background needed.

---

What SusFactor Is

SusFactor is a guardrail that screens text prompts for jailbreaks and prompt injection attacks before they reach your LLM. You send it a prompt; it returns a suspicion score between 0 and 1:

• A score near 0 means the prompt looks benign.
• A score near 1 means the prompt looks like an attack.

SusFactor doesn't decide "block" or "allow" on its own. You set a cutoff, called the threshold, and anything scoring at or above it is treated as an attack:

if suspicion_score >= threshold:
    treat as an attack   # block it, or send it for review
else:
    let it through

The default threshold is 0.5. That's a reasonable starting point, but it's almost certainly not the best choice for your specific application. This guide shows you how to find a threshold that matches your traffic and your tolerance for mistakes.

Why the Threshold Matters

Where you set the threshold is a trade-off between two kinds of mistakes:

• False positives: a legitimate prompt gets wrongly flagged as an attack. Your real users hit friction, get blocked, and may give up.
• Missed attacks: a real attack scores below your threshold and slips through. Your LLM gets exploited.

These pull in opposite directions, and the threshold is the dial between them:

• Lower the threshold → you catch more attacks, but you also flag more legitimate prompts (more false positives).
• Raise the threshold → you flag fewer legitimate prompts, but more attacks slip through.

A common misconception: "If I lower the threshold, I'll get fewer false alarms." It's the opposite. Lowering the threshold flags more prompts as attacks, including more benign ones. To reduce false positives, you raise the threshold, and you pay for it in missed attacks.

There's no universally correct threshold. The right one depends on your traffic, how costly each kind of mistake is to you, and how you've wired SusFactor into your product. The rest of this guide walks you through finding it.

The Short Version

If you just want a sensible starting point:

1. Start at the default threshold of 0.5.
2. Collect a few hundred real prompts from your own traffic and label each one as an attack or not.
3. Run them through SusFactor (code below) and see how many attacks you'd catch and how many legitimate prompts you'd wrongly block at 0.5.
4. If too many legitimate prompts are blocked, raise the threshold. If too many attacks slip through, lower it.
5. Before you commit, read "What your flagged prompts will actually look like". At real-world attack rates, a threshold that looks fine on a test set can flood you with false alarms.

The sections below make each step concrete.

What You Need to Calibrate

A labeled sample of your own traffic: a few hundred prompts that look like what your application really sees, each marked as either an attack (1) or benign (0).

Why your own traffic, and not a public dataset of attacks? Because the threshold that's right for you depends on your mix of prompts. A public benchmark of hand-crafted jailbreaks looks nothing like the everyday prompts your users send, so a threshold tuned on it won't transfer.

How many examples?

More is better, but here's the practical guidance:

• 100 of each (100 attacks, 100 benign) is the minimum for numbers you can loosely trust.
• 300+ of each gives you noticeably more reliable numbers.
• 1,000+ of each is what you want if you need to tell the difference between, say, "2% of legitimate prompts blocked" and "5% blocked" with confidence.

With small samples, your measurements are noisy: a result that looks like "3% of legitimate prompts blocked" measured on only 50 benign examples could easily be anywhere from 0% to 10% in reality. That's why the code below also reports a confidence range (see Step 3): a plain measure of how much to trust each number.

Don't calibrate on data SusFactor was trained on

If your calibration prompts overlap with the data SusFactor was trained on, the model will look better than it really is. It has effectively seen the answers already, so your threshold estimates will be too optimistic.

SusFactor's fine-tuning data includes these open-source datasets. If you build a calibration set from public data, exclude any prompts that come from them:

Dataset	HuggingFace ID	What it contains
WildJailbreak (train split)	allenai/wildjailbreak	Adversarial harmful + adversarial-benign prompts
WildChat	allenai/WildChat-1M	Real-world benign chat prompts
jackhhao jailbreak-classification	jackhhao/jailbreak-classification	Labeled jailbreak vs. benign prompts

SusFactor is also built on a base model trained on additional jailbreak data that isn't fully enumerable here. This is exactly why calibrating on your own production traffic is the safest choice: it can't overlap with training data, and it tunes the threshold to your real distribution at the same time.

Step-by-Step

Step 1: Score your prompts with SusFactor

Run each of your labeled prompts through SusFactor to get its suspicion score. How you do this depends on which version of SusFactor you're using:

• On-prem: you're running the self-hosted SusFactor model weights yourself. You load the model locally and score prompts in-process.
• Hosted API (early-access beta): you're calling 0DIN's hosted SusFactor endpoint over HTTPS. You send each prompt to the API and read back its score. See the API reference for how to obtain a token and authenticate.

Either way, the goal is the same: end up with a scores list (numbers between 0 and 1) lined up with your labels list (1 = attack, 0 = benign). Steps 2 and 3 are identical from there.

On-prem (self-hosted weights)

SusFactor uses a text encoder plus a small classification head, so loading it takes a few lines. Point MODEL_DIR at wherever your SusFactor weights are: the directory contains an encoder/ subfolder (the encoder and tokenizer) and a head.pt file (the classification head):

import torch
import torch.nn as nn
from transformers import AutoModel, AutoTokenizer

MODEL_DIR = "path/to/susfactor-model"

# The classification head that sits on top of the encoder.
class ClassificationHead(nn.Module):
    def __init__(self, embedding_dim=1024, hidden_dim=256, num_classes=2):
        super().__init__()
        self.classifier = nn.Sequential(
            nn.Linear(embedding_dim, hidden_dim),
            nn.GELU(),
            nn.Linear(hidden_dim, num_classes),
        )

    def forward(self, embeddings):
        return self.classifier(embeddings)

# Load the encoder + tokenizer (in the "encoder/" subfolder) and the head weights.
encoder = AutoModel.from_pretrained(f"{MODEL_DIR}/encoder")
tokenizer = AutoTokenizer.from_pretrained(f"{MODEL_DIR}/encoder")
head = ClassificationHead(embedding_dim=encoder.config.hidden_size)
head.load_state_dict(
    torch.load(f"{MODEL_DIR}/head.pt", map_location="cpu", weights_only=True)
)
encoder.eval()
head.eval()

def suspicion_score(text: str) -> float:
    """Return SusFactor's suspicion score in [0, 1] for one prompt."""
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
    with torch.no_grad():
        hidden = encoder(**inputs).last_hidden_state
        # Mean-pool the token embeddings, respecting the attention mask.
        mask = inputs["attention_mask"].unsqueeze(-1).float()
        pooled = (hidden * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1e-9)
        probs = torch.softmax(head(pooled), dim=-1)
    return probs[0, 1].item()  # column 1 = "attack"

# Score every labeled prompt.
texts = ["your", "labeled", "prompts", "..."]
labels = [1, 0, 1, 0]  # 1 = attack, 0 = benign, aligned with texts
scores = [suspicion_score(t) for t in texts]

This simplified example truncates at 512 tokens for clarity. The production API instead chunks longer prompts and scores each chunk (see API).

Hosted API (early-access beta)

With the early-access beta you don't load any weights locally. You send each prompt to the SusFactor endpoint and read the score it returns. First set up authentication as described in the API reference: exchange your 0din.ai Portal API key for a short-lived JWT and make it available to your script (below it's read from the DEFENSE_JWT environment variable).

The response for each prompt looks like {"is_suspicious": true, "score": 0.997}. The score field is the number in [0, 1] you calibrate on, exactly the same quantity as the on-prem suspicion_score: the hosted early-access beta currently serves the same SusFactor model as the licensed self-hosted weights. For prompts under 512 tokens this is identical; for longer prompts, the on-prem example above truncates while the hosted API chunks, so calibrate against whichever deployment you'll actually run in production if your traffic includes long prompts.

import os
import requests

API_URL = "https://defense.0din.ai/api/v1/sus"
JWT = os.environ["DEFENSE_JWT"]  # see the API reference for how to mint this

def suspicion_score(text: str) -> float:
    """Return SusFactor's suspicion score in [0, 1] for one prompt."""
    response = requests.post(
        API_URL,
        headers={"Authorization": f"Bearer {JWT}"},
        json={"prompt": text},
        timeout=30,
    )
    response.raise_for_status()
    return response.json()["score"]

# Score every labeled prompt, one at a time.
texts = ["your", "labeled", "prompts", "..."]
labels = [1, 0, 1, 0]  # 1 = attack, 0 = benign, aligned with texts
scores = [suspicion_score(t) for t in texts]

note

The early-access beta has usage limits (exact figures TBD), so score your prompts sequentially as shown rather than firing many requests in parallel. A calibration set of a few hundred prompts is well within a sensible range. If a token expires mid-run (they're short-lived), mint a fresh one and continue.

You now have a scores list (numbers between 0 and 1) lined up with your labels list (1 = attack, 0 = benign).

Step 2: See the trade-off at every threshold

This builds a table showing, for each candidate threshold, how many attacks you'd catch and how many legitimate prompts you'd wrongly flag. Three numbers to know first:

• Recall: of the real attacks, what fraction did you catch? Higher is better. (Missing an attack means recall goes down.)
• False positive rate (FPR): of the legitimate prompts, what fraction did you wrongly flag? Lower is better.
• Precision: of everything you flagged, what fraction were real attacks? We'll come back to this one. It's the number most people get wrong, and it's covered in "What your flagged prompts will actually look like".

import numpy as np
import pandas as pd

scores = np.array(scores)
labels = np.array(labels)  # 1 = attack, 0 = benign

rows = []
thresholds = list(np.arange(0.05, 0.95, 0.05)) + [0.95, 0.96, 0.97, 0.98, 0.99]
for t in thresholds:
    flagged = (scores >= t)
    caught_attacks = ((flagged) & (labels == 1)).sum()
    flagged_benign = ((flagged) & (labels == 0)).sum()
    total_attacks = (labels == 1).sum()
    total_benign = (labels == 0).sum()

    recall = caught_attacks / total_attacks if total_attacks else 0
    fpr = flagged_benign / total_benign if total_benign else 0

    rows.append({"threshold": round(t, 2), "recall": recall, "FPR": fpr})

print(pd.DataFrame(rows).to_string(index=False))

Read down the table: as the threshold rises, both recall and FPR fall. You're looking for a row where the FPR is low enough to keep your users happy while recall is high enough to keep you safe.

What to expect from SusFactor specifically: its scores are strongly bimodal: most prompts score very close to 0 or very close to 1, with relatively few in the middle. In practice this means small threshold changes near 0.5 barely change anything, but pushing the false-positive rate really low requires thresholds up near 0.99 (and costs a lot of recall). Don't be surprised if the table jumps rather than moving smoothly.

Step 3: Check how reliable your numbers are

The numbers in your table are estimates from a limited sample, so they carry some uncertainty. This step measures that uncertainty and reports a 95% confidence range for each: the band your true value most likely falls within. A narrow band means you can trust the number; a wide band means you need more data before committing.

The technique below (called bootstrapping) just re-computes the numbers many times on random re-samples of your data and reports the spread. You don't need to understand the internals to use it.

def confidence_ranges(scores, labels, threshold, n_repeats=1000, seed=42):
    """Estimate 95% confidence ranges for FPR and recall at a threshold."""
    rng = np.random.default_rng(seed)
    attack_idx = np.where(labels == 1)[0]
    benign_idx = np.where(labels == 0)[0]

    recalls, fprs = [], []
    for _ in range(n_repeats):
        # Re-sample attacks and benign prompts separately, with replacement.
        a = rng.choice(attack_idx, size=len(attack_idx), replace=True)
        b = rng.choice(benign_idx, size=len(benign_idx), replace=True)
        s, y = scores[np.concatenate([a, b])], labels[np.concatenate([a, b])]
        flagged = (s >= threshold)
        recalls.append(flagged[y == 1].mean())
        fprs.append(flagged[y == 0].mean())

    return {
        "FPR_range": (np.percentile(fprs, 2.5), np.percentile(fprs, 97.5)),
        "recall_range": (np.percentile(recalls, 2.5), np.percentile(recalls, 97.5)),
    }

r = confidence_ranges(scores, labels, threshold=0.5)
print(f"FPR:    likely between {r['FPR_range'][0]:.1%} and {r['FPR_range'][1]:.1%}")
print(f"Recall: likely between {r['recall_range'][0]:.1%} and {r['recall_range'][1]:.1%}")

If the ranges are uncomfortably wide (say, "FPR is somewhere between 5% and 25%"), that's a signal to collect more labeled examples before locking in a threshold.

What Your Flagged Prompts Will Actually Look Like

This is the single most important, and most commonly misunderstood, part of setting up a guardrail. Read it before you pick a threshold.

Recall and FPR don't tell you the thing you most want to know: when SusFactor flags a prompt, how likely is it to actually be an attack? That's precision, and it depends heavily on something the earlier numbers ignore: how common attacks actually are in your traffic (the "attack rate," or prevalence).

Here's the trap. When you test on a balanced sample (half attacks, half benign), precision looks great, roughly equal to recall. But real traffic isn't balanced. In most applications, attacks are rare: maybe 1% or 5% of prompts. And when attacks are rare, most of the prompts SusFactor flags are actually benign ones caught by mistake, simply because there are so many more benign prompts to begin with.

A concrete example. Say your calibration shows SusFactor catches 80% of attacks (recall) while wrongly flagging 10% of benign prompts (FPR). Sounds solid. Now watch what happens as attacks get rarer:

Out of every...	...attacks	...benign	You flag (correctly)	You flag (by mistake)	Of everything you flagged, real attacks =
100 prompts, 5% attacks	5	95	4	~10	~30% (about 1 in 3)
100 prompts, 1% attacks	1	99	~1	~10	~7% (about 1 in 13)

Same model, same threshold, but at a 1% attack rate, only about 1 in 13 things you flag is a real attack. The other 12 are false alarms. The model didn't get worse; rare events are just like that.

Using SusFactor's actual published numbers. At SusFactor's real operating point (88% recall, 9% FPR at the default threshold of 0.5, the same threshold is_suspicious flips on; see Performance), a 1% attack rate works out to precision = (0.88 × 0.01) / (0.88 × 0.01 + 0.09 × 0.99) ≈ 9%, roughly 1 in 11 flagged prompts is a real attack. That means the default threshold is not a safe unsupervised auto-block setting on realistic traffic without accounting for this trade-off, whether that's routing flagged prompts to human review instead of auto-blocking, or raising the threshold as described in Choosing Your Threshold below.

You can compute this for your own numbers. If p is your attack rate (e.g. 0.01 for 1%):

precision = (recall × p) / (recall × p + FPR × (1 − p))

Why this matters for setup: you need at least a rough sense of your attack rate to choose a threshold sensibly. Even an order-of-magnitude guess ("attacks are maybe 1% of our traffic" vs. "maybe 10%") completely changes what a good threshold looks like. If you don't estimate it, you're choosing blind.

Choosing Your Threshold

Two things drive the choice: how you've deployed SusFactor, and how rare attacks are in your traffic.

Match the threshold to how you use SusFactor

• Blocking prompts automatically (a false positive means a real user is blocked): favor a higher threshold to keep false positives low, and accept that some attacks will slip through. There's no review queue to worry about. Blocked users have to retry or contact support, which is real friction for them.

• Sending flagged prompts to a human review queue (a false positive means wasted analyst time): here precision is what matters: it's literally how many false alarms your analysts wade through per real attack they find. Re-read the section above: at a 1% attack rate, a queue can easily be 90%+ false alarms, which burns out reviewers fast.

• A two-stage setup (auto-block the most obvious attacks at a high threshold, send the medium-scoring ones to human review): this gets you both high safety and a manageable review queue. The auto-block tier is high-precision; the review tier catches the rest. Given SusFactor's bimodal scores, this fits it well: pick a high auto-block threshold from your own calibration table (illustratively, something like 0.9, where the bimodal distribution puts most attacks) and a lower review-queue threshold below it.

As an illustration of a three-tier approach: pass below 0.5, flag for review between 0.5 and a high threshold, and auto-block above it. The exact cutoffs should come from your own calibration table, not these numbers.

Match the threshold to your priorities

If you care most about...	Then...	The cost
Not disrupting users	Raise the threshold until false positives are acceptably low	More attacks slip through
Catching every attack	Lower the threshold until recall is high enough	Many more false alarms, especially painful when attacks are rare
A middle ground	Start near the default (0.5) and adjust based on your calibration table	Reasonable, but always check precision at your real attack rate

Putting a number on it (optional)

If you can estimate what each mistake costs you, you can make the trade-off explicit. If a missed attack costs you $X and a false positive costs you $Y, the best threshold is roughly the one that minimizes:

(missed attacks) × $X  +  (false positives) × $Y

You don't need exact dollar figures: even rough ratios (e.g. "a missed attack is 20× worse than a false alarm") point you toward the right end of the range.

Keep It Calibrated

Calibration isn't one-and-done. Re-check your threshold periodically, because the things it depends on change:

1. Your traffic drifts: as your user base grows and attackers adapt, the mix of prompts shifts.
2. SusFactor gets updated: a new model version scores prompts differently, which can move the right threshold.
3. Attacks come in waves: a threshold tuned during a quiet stretch may be too loose during a coordinated campaign.

A practical rhythm: re-run this calibration on a fresh labeled sample about once a quarter, and any time you upgrade to a new SusFactor version. If the recommended threshold has moved a lot, dig into why before adopting it: it usually means either your traffic changed or your labeled sample wasn't representative.

When you recalibrate after a SusFactor update, re-check the training-data list above too, since a new version may have been trained on different data, which changes what's off-limits for your calibration set.