Can malicious instructions in external documents be stored in an AI agent's long-term memory? An AWS AgentCore Memory experiment(zenn.dev)
Writing language: KoreanRead in the original language
Plain summary
- A Japanese Zenn author tested with AWS AgentCore Memory whether bad instructions hidden in an external document can get into the feature where an AI agent picks out important points from conversations and remembers them. The author used a fictional customer and mock materials.
- An English sentence telling the agent to also read Customer B's materials when writing Customer A's report was stored as a "user preference." A request beyond the agent's authorization was organized as if it were a report-writing habit.
- In the pre-storage check, the Prompt Attack filter did not catch this sentence. After adding Denied Topics that prohibit referencing materials across customers, it was blocked. The author implemented the step that stops storage on the app side.
- The contaminated memory was passed to new sessions and run 5 times each, 10 times in total, but there was no request for Customer B's materials. The reason could not be determined. The author tested only one document and said false positives and bypasses are possible.
Found on
Zenn ↗ / Published by source
Have you run into something like this? Tell us in a line or two what you did. You do not need to know the product.