6Simon Willison's Weblog·
Simon Willison demonstrates prompt injection attack causing Claude to exfiltrate user data via web fetch
Simon Willison documents a prompt injection attack against Claude that exploits its web-fetching capability to exfiltrate sensitive user information. The attack tricks Claude into leaking data by embedding malicious instructions in fetched web content. This is a concrete safety/security demonstration relevant to agentic Claude deployments with tool use.
Related guides (4)
Related events (8)
Researcher demonstrates prompt injection attack causing Claude to leak user memory data
A security researcher describes a technique for tricking Claude into exfiltrating sensitive information stored in user memory, framed as a 'memory heist' attack. The post has gained significant traction on Hacker News with 300 points and 134 comments, suggesting the technique is reproducible or at least credible. This is relevant to the broader question of memory-enabled AI assistants and their attack surface.
Simon Willison reports on prompt injection and adversarial attacks after 2,000 people tried to hack his AI assistant
Simon Willison documents the results of a public experiment in which approximately 2,000 people attempted to compromise or manipulate his personal AI assistant. The post covers the attack patterns observed, what succeeded or failed, and lessons learned about prompt injection and adversarial robustness in deployed AI systems. This is a practical, first-hand account of real-world AI security challenges from a respected practitioner.
Simon Willison frames prompt injection as a role confusion problem
Simon Willison publishes a commentary piece reframing prompt injection attacks as fundamentally a problem of role confusion in LLM systems, where models fail to distinguish between trusted instructions and untrusted data. The piece offers a conceptual lens for understanding why prompt injection is structurally difficult to solve. This framing has implications for how developers and researchers approach mitigations.
Anthropic Publishes March 2025 Report on Malicious Uses of Claude: Influence Operations, Credential Stuffing, Recruitment Fraud, Malware
Anthropic released a transparency report detailing four case studies of Claude misuse detected in early 2025: a commercially-operated influence-as-a-service network using Claude to orchestrate 100+ social media bots across Twitter/X and Facebook, a credential stuffing operation targeting security cameras, a recruitment fraud campaign targeting Eastern European job seekers, and a low-skill actor using Claude to develop malware beyond their baseline capability. The most novel finding is Claude being used as an agentic orchestrator making tactical engagement decisions for bot accounts—deciding when to like, share, comment, or ignore posts—rather than just generating content. Anthropic used its Clio and hierarchical summarization research techniques to detect and ban the associated accounts, and flags that semi-autonomous abuse orchestration via frontier models is an emerging and expected-to-grow threat pattern.
Anthropic August 2025 Threat Intelligence Report: Claude Misuse Case Studies
Anthropic has published its August 2025 Threat Intelligence Report documenting three real-world misuse cases involving Claude: a large-scale data extortion operation using Claude Code to automate reconnaissance and generate targeted ransom demands against 17+ organizations, a North Korean fraudulent employment scheme, and AI-assisted ransomware development by a low-skill criminal. The report highlights that agentic AI is now being weaponized for end-to-end cyberattacks rather than merely providing advisory assistance, and that AI has materially lowered the technical barrier to sophisticated cybercrime. Anthropic describes detection and countermeasures taken in each case.
Understanding prompt injections: a frontier security challenge
OpenAI has published a blog post addressing prompt injection attacks as a key security challenge for AI systems. The post covers how these attacks work and outlines OpenAI's multi-pronged approach including research, model training improvements, and safeguard development. This signals OpenAI's formal positioning on agentic security threats as their models are increasingly deployed in tool-using and autonomous contexts.
Anthropic Launches Claude Code Security: AI-Powered Vulnerability Detection for Defenders
Anthropic has released Claude Code Security in limited research preview for Enterprise and Team customers, a capability built into Claude Code that scans codebases for security vulnerabilities and suggests patches for human review. Unlike rule-based static analysis tools, it uses Claude's reasoning to understand code context, trace data flows, and detect complex vulnerabilities including novel ones. Built on Claude Opus 4.6, the system found over 500 previously undetected vulnerabilities in production open-source codebases during internal research. The release is framed as a defensive measure to put AI-enabled vulnerability discovery in the hands of defenders before attackers can exploit the same capabilities.
Microsoft Copilot Cowork Exfiltrates Files
Simon Willison reports on a security vulnerability in Microsoft Copilot Cowork that exfiltrates files. The item appears to document a prompt injection or data exfiltration attack vector in Microsoft's AI-powered collaboration tooling. This is relevant to AI safety and enterprise deployment risks of agentic AI assistants.