This month a vulnerability in an AI assistant for business was published that should make everyone building with AI stop and think. The researchers called it EchoLeak. An attacker could send a crafted email to a user. The user didn\u00b4t have to open it. Later, when the user asked the assistant an ordinary question, the assistant read the email as context, followed the hidden instructions in it and leaked internal data. Zero clicks.<\/p>\n
Microsoft fixed it on the server side and reported no evidence of exploitation. But the class of problem is here to stay. It is called prompt injection.<\/p>\n
The core issue is simple. A language model can\u00b4t reliably tell the difference between data and instructions. Everything is text. So when an AI reads an email, a document or a web page, any instruction hidden in that content may be followed, with the permissions of the AI.<\/p>\n
What can you do? Limit what each AI can reach. An assistant that can read everything is an assistant that can leak everything. Treat external content as untrusted and keep it apart from sensitive data where possible. Restrict how the AI can send data out: links, images, tool calls and connections to external services. Use sensitivity labels and DLP so the most sensitive data isn\u00b4t available to the AI at all. And monitor what the AI does, not just what users do.<\/p>\n
It is the oldest lesson in security, in a new form. Never trust input.<\/p>\n
In the age of AI, every document is potential code.<\/p>\n","protected":false},"excerpt":{"rendered":"
This month a vulnerability in an AI assistant for business was published that should make everyone building with AI stop and think. The researchers called it EchoLeak. An attacker could send a crafted email to a user. The user didn\u00b4t have to open it. Later, when the user asked the assistant an ordinary question, the […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[4,13],"tags":[165,176,175],"class_list":["post-1478","post","type-post","status-publish","format-standard","hentry","category-security-architecture","category-technology","tag-ai","tag-copilot","tag-prompt-injection"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/crowmoor.se\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1478","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/crowmoor.se\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/crowmoor.se\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/crowmoor.se\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/crowmoor.se\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1478"}],"version-history":[{"count":1,"href":"https:\/\/crowmoor.se\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1478\/revisions"}],"predecessor-version":[{"id":1511,"href":"https:\/\/crowmoor.se\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1478\/revisions\/1511"}],"wp:attachment":[{"href":"https:\/\/crowmoor.se\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1478"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/crowmoor.se\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1478"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/crowmoor.se\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1478"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}