[Ayush Paul] posts about extracting data from Claude agents while it accesses web data to fulfill user requests.

But it wasn’t that easy. [Ayush] discovered that Anthropic anticipated many of the attacks, and set up guardrails in an attempt to keep the agent from accessing arbitrary web sites. For Claude to access a website, the user has to specify it, it must be the results of a web search, or it must be referenced by a website previously specified by the user or returned in a search.

To convince the Claude agent to navigate a malicious site designed to extract data, Ayush formed a false warning that Cloudflare was blocking the agent for authentication purposes, and asked it to spell out the name of the agents owner by clicking a list of alphabetical links. Of course Claude trusts Cloudflare and wants to be helpful, so it cheerfully completed the task.

Once the agent is trapped in the false authentication loop, it can be interrogated for all sorts of information it knows about the owner: Ayush was able to convince it to disclose employer, and even data about the user that could be linked to security questions, like their home town.

Since Claude can be detected by the user agent (the field attached to web requests that tells the web server what sort of browser is requesting the page) custom information can be fed to the bot while users see a normal website; clicking a link looks completely normal, but asking an agent to summarize the site triggers fooling the bot into spilling the beans.

After reporting the issue, Ayush was told that Anthropic had already identified the issue internally, and eventually prevented the attack for now by preventing the agent from following links on external pages.

Grok sends entire codebases

Cereblab discovered that the Grok coding agent uploaded the entire content of the codebase it’s working with – and all Git history – to xAI servers, almost immediately. Even when told to never upload a file, the agent would reply “OK”, and then begin uploading the code bundle anyway.

If uploading your code to a remote server isn’t bad enough, and this could be extremely bad in some situations involving sensitive company code, including the entire Git history means that previously deleted files, like accidentally committed secrets or authentication credentials and tokens, were also leaked. Attempting to opt out by disabling options to improve the model by uploading code had no effect.

After gaining attention, Grok has added a privacy option to opt out of data retention. xAI has committed to deleting the retained uploaded code, but it is unclear if users will be told when their data has been removed. To actually prevent the agent from uploading local files to the xAI servers, even temporarily, a global flag “disable_codebase_upload” is required. Watch out, the privacy retention flags are only per-session!

Continue reading “This Week In Security: AI Is A Mess, Hacking Car Chargers, An OpenSSL DoS, And Factories Under Attack” →