Prompt Injection.

Prompt injection is a critical security vulnerability that occurs when an attacker embeds malicious instructions within user input that an LLM processes, causing the model to deviate from its intended behavior. Unlike traditional code injection attacks, prompt injections exploit the natural language processing capabilities of LLMs by crafting inputs that appear legitimate but contain hidden commands. This attack vector has become increasingly important as LLMs are integrated into applications that process untrusted content from web pages, documents, emails, or user submissions.

The attack works by leveraging the LLM's inability to distinguish between legitimate instructions from the system and malicious instructions embedded in user data. Attackers use techniques like instruction smuggling, where harmful prompts are hidden within seemingly innocent content, or delimiter confusion, where they manipulate the model's understanding of where instructions end and data begins. Direct prompt injections target the user's own session, while indirect injections involve poisoning external data sources that the LLM later processes, making them particularly dangerous for applications using retrieval-augmented generation or web browsing capabilities.

Defending against prompt injection requires multiple layers of protection, including input sanitization, output filtering, and architectural changes like separating instruction and data channels. However, there is no foolproof defense due to the fundamental nature of how LLMs process text. Organizations must implement monitoring systems to detect suspicious outputs and consider the security implications when designing LLM-powered applications. Common misconceptions include believing that simple keyword filtering or prompt templates provide adequate protection, when in reality, sophisticated attacks can bypass most current defensive measures.