Jailbreak

In depth

A jailbreak is a prompt written to get a model to produce output its safety training is meant to refuse. Wei, Haghtalab, and Steinhardt traced jailbreaks to two failure modes. The first is competing objectives, where being helpful conflicts with being safe. The second is mismatched generalization, where safety training doesn't cover inputs the model can still understand, such as encoded or obfuscated text. Unlike prompt injection, where an attacker hijacks an app through third-party content, a jailbreak is usually the user attacking the model's own safeguards.

Primary source: arxiv.org