Defenders test prompt injections as traps for AI hacking agents
Tracebit researchers said hidden prompts placed near AWS secrets can make attacking LLM agents hit their safety limits and stop.
By James Whitfield · Staff Writer
2 min read
Security researchers are testing prompt injections as a defensive tool against AI-driven attacks. Tracebit said Monday that prompts placed beside sensitive Amazon Web Services data often caused attacking large language model agents to halt rather than continue an intrusion.
According to Ars Technica, prompt injections have mostly been known as an attacker technique. The method involves hiding instructions inside content that an AI system may read, such as an email or calendar invitation, in an effort to make the model act against the user’s interests.
Ars Technica reported that a carefully written hidden command can be enough to push an LLM into leaking sensitive information or carrying out another harmful step. That risk has made prompt injection one of the main tools for abusing AI platforms connected to user data and external services.
Using the same weakness against attackers
Tracebit researchers said they found a defensive use for the same kind of instruction. In their tests, the researchers placed prompt injections near passwords, cryptographic keys and other secrets stored on AWS, according to the company’s Monday post.
The prompts were aimed at the attacking LLM rather than at a legitimate user’s model. Tracebit said the instructions told the AI agent to take an action that its safety rules prohibit.
According to Tracebit, that conflict was often enough to stop the agent. The company said the model would respond by shutting down after encountering the forbidden instruction.
The approach relies on the safety systems built into LLMs, Ars Technica reported. Those guardrails are designed by AI developers to block harmful actions, and Tracebit’s finding suggests defenders may be able to trigger those limits when an AI hacking agent reads protected material.
Tracebit’s report centers on AI agents used for hacking attempts, not on traditional human intruders. The company’s finding points to a possible defense in environments where attackers use LLMs to inspect cloud resources and search for secrets.
The technique also shows how prompt injection has become a two-sided issue. Ars Technica described it as a common way to turn AI tools against users, while Tracebit said a similar instruction can be positioned as a tripwire for models operating on behalf of attackers.
This story draws on original reporting from Ars Technica.