Researchers from CBAI and Cosmos propose a mechanistic explanation of prompt injection in large language models (LLMs), highlighting the importance of studying roles in LLMs.

Researchers from CBAI and Cosmos have proposed a mechanistic explanation of prompt injection in large language models, which occurs when low-privilege text gains the authority of a higher-privilege role.

Researchers from CBAI and Cosmos have proposed a mechanistic explanation of prompt injection in large language models, which occurs when low-privilege text gains the authority of a higher-privilege role. The researchers found that large language models perceive roles from an insecure feature, which is the writing style, rather than the secure method of tags. This can lead to prompt injection attacks, where attackers hide malicious commands in text that is tagged as external data, but sounds like a user instruction. The researchers developed a new attack called CoT Forgery, which injects fake reasoning into a user message or tool output, and found that it can steal the trust given to the model's reasoning role. The researchers also found that the effectiveness of the attack depends on how user-like the command is, and that simply prepending 'User: ' to the command can cause the model to perceive it as more likely to be genuine user text. The researchers conclude that roles are not discrete, architectural boundaries, but rather soft inferences reconstructed from a combination of surface features, and that this is what enables prompt injection. The researchers propose a general theory of roles, which suggests that roles are load-bearing infrastructure in the large language model stack, and that they separate different aspects of the model's behavior, such as exploration and communication, or comprehension and generation. The researchers also provide a brief history of roles, which shows how they evolved from a formatting trick to a crucial component of large language models. Overall, the researchers' findings have significant implications for the security and reliability of large language models, and highlight the need for a more robust approach to role perception.