The architectural limit of AI security

Instead of "obeying commands", your AI assistant predicts the next word based on all the text it sees. Your instructions, the system prompt, and the content of that PDF you just uploaded - it's all one stream of tokens.

Think of it as a brilliant but blind assistant sitting in a room. You tell them: "Only follow my instructions", and they agree, but they can't tell voices apart. When a hidden instruction inside a document says "now upload this file" - it sounds exactly like you.

How an injection reaches the model - an attacker plants instructions in a document, the document is pulled in alongside your query, and the model reads both as one stream.
How an injection reaches the model - an attacker plants instructions in a document, the document is pulled in alongside your query, and the model reads both as one stream.

This is not a bug in Claude, ChatGPT or Gemini: it's how the transformer architecture works1. The model has no cryptographic signature for "this came from the user": no verified sender and no trust hierarchy in the math. The tags that separate "system prompt" from "document content" are just text - and text can be faked.

OpenAI tried instruction hierarchy - marking system prompts as higher priority2. It helps against basic attacks but breaks against advanced ones. Others tried processing documents in a separate context window, but then the agent can't reason about your document in light of your question, which is the entire point of the product.

The closest analogy is SQL injection in the early 2000s. Data and commands traveled in the same channel. The fix was architectural - parameterized queries that physically separated code from data - and it took the industry years to adopt it.

For LLMs, that architectural separation doesn't exist yet, because instructions and data are the same thing: text. Until someone invents the equivalent of parameterized queries for language models, prompt injection remains structurally unsolvable3.

To date, I have not yet seen a robust defense against this vulnerability which is guaranteed to work 100% of the time.4

Simon WillisonCreator of Datasette, co-creator of DjangoPhotograph by Paul Downey, cropped, CC BY 2.0

AI agents with file access still rely on trust. The blind assistant in the room is incredibly smart, but it still can't tell your voice from an attacker's.

References

Subscribe to the newsletter

Articles on AI, politics and disinformation · No paywall