04 / How It Goes Wrong
Prompt injection
Hidden instructions planted in something the AI reads, such as a web page, email or document, designed to make it do something it should not.
How you’ll hear it
“The agent forwarded a customer’s details.”
“An email it summarised contained hidden instructions. That was prompt injection.”
Avoid
Giving an agent that reads untrusted content the power to send or delete without approval.
The longer version
Prompt injection exploits the fact that a model cannot reliably tell instructions from data, so anything an agent reads can carry commands. It sits at the top of most AI security risk lists, and the main defence is limiting what an agent is able to do after reading untrusted material.
In depth
Prompt injection is an attack in which instructions are hidden inside content an AI system reads, such as a web page, an email, a document or a product review, so that the system follows them instead of its own rules.
It works because language models cannot reliably tell the difference between instructions and data. Everything is text. An agent asked to summarise an inbox may meet an email that says to forward the finance folder elsewhere, and treat it as a command.
The risk grows with what the system can do. A chatbot that only answers questions can be embarrassed. An agent that can send messages, move files or make payments can be turned against its owner. Prompt injection sits at the top of most lists of AI security risks, and there is no complete technical fix yet.
The main defence is design: limit what an agent can do after it has read untrusted content, and require human approval for anything consequential.
What to do
- List every source of untrusted content your AI systems read, including email, web pages and uploaded files.
- Remove or gate the ability to send, delete or pay after reading untrusted content.
- Include injection attacks in red teaming and in your evals.