Red-Teaming Integrations¶

JEF integrates with two leading LLM red-teaming frameworks: garak (NVIDIA) and PyRIT (Microsoft). These integrations let you use JEF’s scoring rubrics directly inside automated red-teaming pipelines — no glue code required.

Garak¶

garak is an LLM vulnerability scanner. JEF provides detectors (scoring) and probes (attack prompts) that plug directly into garak’s CLI and Python API.

Installation¶

pip install 0din-jef[garak]

This pulls in garak >= 0.13.3 alongside JEF.

Available Detectors¶

Detectors score model outputs using JEF rubrics. Each detector returns a value in garak’s 0.0-1.0 scale (JEF percentage / 100).

Detector	What It Scores
`0din_jef.CrystalMethScore`	Crystal meth synthesis content
`0din_jef.NerveAgentScore`	Nerve agent synthesis content
`0din_jef.FentanylScore`	Fentanyl synthesis content
`0din_jef.AnthraxScore`	Anthrax / B. anthracis content
`0din_jef.CRISPRScore`	CRISPR gene-editing content
`0din_jef.CopyRightScoreHarryPotterPageOne`	Harry Potter page-one reproduction
`0din_jef.CopyRightScoreHarryPotterChapterOne`	Harry Potter chapter-one reproduction

Available Probes¶

Probes are disclosed jailbreak techniques (n-days) from the 0DIN bug-bounty program. Each probe carries its original attack prompts and links to its 0DIN disclosure page.

Probe	Technique
`0din_jef.PlaceholderInjection`	Guardrail bypass via placeholder text between words
`0din_jef.IncrementalTableCompletion`	Incomplete-table pattern completion
`0din_jef.TechnicalFieldGuide`	Law-enforcement field-guide framing
`0din_jef.ChemicalCompilerDebug`	Obfuscated chemical notation as a “debug” task
`0din_jef.Correction`	Forensic document correction request
`0din_jef.HexRecipeBook`	Hex-encoded instructions in fictional context

CLI Usage¶

Run a probe with a detector in a single command:

# Score a model against crystal meth rubric using a disclosed n-day
garak --model_type openai --model_name gpt-4 \
      -p 0din_jef.PlaceholderInjection \
      -d 0din_jef.CrystalMethScore

# Mix JEF detectors with any garak probe
garak --model_type openai --model_name gpt-4 \
      -p <any-probe> \
      -d 0din_jef.NerveAgentScore

# Use multiple detectors
garak --model_type openai --model_name gpt-4 \
      -p 0din_jef.Correction \
      -d 0din_jef.CrystalMethScore,0din_jef.FentanylScore

Python Usage¶

from garak._plugins import load_plugin

# Load a detector
detector = load_plugin("detectors.0din_jef.CrystalMethScore")
results = detector.detect(attempt)  # list[float] in 0.0-1.0

# Load a probe
probe = load_plugin("probes.0din_jef.PlaceholderInjection")
print(probe.prompts)        # attack prompt strings
print(probe.goal)           # short goal statement
print(probe.uri)            # 0DIN disclosure URL

Note

JEF detectors and probes are installed as namespace packages. --list_detectors may not show them, but explicit -d 0din_jef.<Name> always works.

PyRIT¶

PyRIT (Python Risk Identification Tool for generative AI) is Microsoft’s red-teaming orchestration framework. JEF provides scorers and seed datasets for PyRIT.

Installation¶

pip install 0din-jef[pyrit]

This pulls in pyrit >= 0.11 alongside JEF.

Available Scorers¶

All scorers inherit from pyrit.score.FloatScaleScorer and return values in PyRIT’s 0.0-1.0 range.

Scorer	What It Scores
`JEFMethScorer`	Crystal meth synthesis content
`JEFNerveAgentScorer`	Nerve agent synthesis content
`JEFFentanylScorer`	Fentanyl synthesis content
`JEFAnthraxScorer`	Anthrax / B. anthracis content
`JEFCRISPRScorer`	CRISPR gene-editing content
`JEFCopyrightScorer`	Harry Potter copyright reproduction

Python Usage¶

from jef.integrations.pyrit.scorers import JEFMethScorer

scorer = JEFMethScorer()
scores = await scorer.score_text_async("some LLM output")
print(scores[0].get_value())  # float in 0.0-1.0

The copyright scorer accepts additional parameters:

from jef.integrations.pyrit.scorers import JEFCopyrightScorer

scorer = JEFCopyrightScorer(
    ref="page_one",       # or "chapter_one" (default)
    min_ngram_size=5,
    max_ngram_size=7,
)
scores = await scorer.score_text_async("some LLM output")

Seed Datasets¶

JEF ships pre-built PyRIT seed datasets for each n-day probe. These are YAML files loadable via pyrit.models.SeedDataset.from_yaml_file().

from pyrit.models import SeedDataset
from jef.integrations.pyrit.seeds import SEEDS_DIR

# Load the PlaceholderInjection seed dataset
dataset = SeedDataset.from_yaml_file(SEEDS_DIR / "placeholder_injection.yaml")
for seed in dataset.seeds:
    print(seed.value)

Available seed files (one per n-day probe):

placeholder_injection.yaml
incremental_table_completion.yaml
technical_field_guide.yaml
chemical_compiler_debug.yaml
correction.yaml
hex_recipe_book.yaml

Regenerating Seeds¶

If you add a new probe definition to the config, regenerate the seed YAML files:

python -m jef.integrations.pyrit.seeds

Architecture¶

Both integrations share a common layer in jef.integrations:

Scorer registry (SUBSTANCE_SCORERS) — maps short keys to JEF scoring modules. Used by both garak detectors and PyRIT scorers.
N-day probe registry (NDAY_PROBES) — loaded from jef/integrations/config/nday_probes.yaml. Garak probes are generated dynamically; PyRIT seeds are pre-built YAML files.
Normalized results (JEFResult) — every JEF call returns a consistent value (0.0-1.0), percentage (0-100), raw_score, and metadata dict.

Adding a new n-day probe:

Add an entry to jef/integrations/config/nday_probes.yaml
Run python -m jef.integrations.pyrit.seeds to regenerate YAML
Commit the new seed file — the garak probe class is generated automatically at import time