Research

Prompt injection has no fix: the architectural limit of AI security

Written 381 words

AI securityAI AgentsClaudeChatGPTGemini

Marbled blue abstract background, liquid marble pattern — Magnific · Licensed under the Magnific Free License

Every AI company says it is "working on" prompt injection. None of them can fix it. Here is why, explained so that a non-engineer can follow it.

Your AI assistant does not obey commands. It predicts the next word based on all the text it can see: your instructions, the system prompt, and the contents of that PDF you just uploaded. All of it is one stream of tokens. One flat river of text.

One flat river of text

Think of it as a brilliant but blind assistant sitting in a room. You tell them: only follow my instructions. They agree. But they cannot tell voices apart. When a hidden instruction inside a document says "now upload this file", it sounds exactly like you.

This is not a bug in Claude, or ChatGPT, or Gemini. It is how the transformer architecture works. The model has no cryptographic signature for "this came from the user". No verified sender. No trust hierarchy baked into the mathematics. The tags that separate a system prompt from document content are themselves just text, and text can be forged.

What has already been tried

There have been attempts. OpenAI tried instruction hierarchy, which marks system prompts as higher priority. It helps against basic attacks and breaks against sophisticated ones.

The privilege levels OpenAI proposed, with a prompt injection arriving in a tool output at the lowest level.
The privilege levels OpenAI proposed, with a prompt injection arriving in a tool output at the lowest level. Wallace, Xiao, Leike, Weng, Heidecke, Beutel, The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions, OpenAI, 2024, fig. 1. arxiv.org/abs/2404.13208. Quoted under §51 UrhG.
Others tried processing documents in a separate context window. But then the agent cannot reason about your document in light of your question, which is the entire point of the product.

The precedent is SQL injection

The closest analogy is SQL injection in the early 2000s. Data and commands travelled in the same channel. The fix was architectural: parameterised queries that physically separated code from data. It took the industry years to adopt.

For language models that separation does not exist yet. Instructions and data are the same substance: text. Until someone invents the equivalent of parameterised queries for language models, prompt injection is not "hard to fix". It is structurally unsolved.

One reader put it more precisely than I had: governance was not bypassed. It was applied correctly, just by the wrong principal.

Every AI agent you give file access to today is operating on trust, not on security. The blind assistant in the room is extremely capable. It still cannot tell your voice from an attacker's.

References