Somebody figured out that the fastest way past a robots allowlist is to just say you’re an AI crawler. Known Agents’ Agentic Web Index is now tracking an active campaign doing exactly that: fake Googlebot traffic at 0.5% of all requests, with ChatGPT-User, GPTBot, OAI-SearchBot and PerplexityBot each around 0.1%.
What they’re actually looking for
Not content. Credentials. The scanned paths read like a list of every AI coding tool people installed this year: /.config/anthropic/credentials/default.json, .claude.json, .claude/settings.json. Mixed in with the classics — /.aws/credentials, /.env.production, firebase-adminsdk.json, Docker configs, Terraform state files stuffed with infrastructure keys.
Why this one matters
This is the first scanning wave treating AI agent config files as a standard attack surface, right alongside .env. Those files sit in home directories on dev boxes and CI runners, and plenty of them hold live API keys.
The detection trick is the useful part: real ClaudeBot and GPTBot traffic verifies via reverse IP lookup or Web Bot Auth signatures. Fail that check and the User-Agent is a costume. Who’s behind it is unknown and probably will stay that way.
If your firewall whitelists AI crawlers by User-Agent string, that rule is now a hole.
You Might Also Like
- Google A2ui Agent to User Interface Finally a Standard way for ai Agents to Show you Things
- Eu Orders Google to Open Android to Rival ai Agents Chatgpt and Claude get Gemini Level Access
- Ai Agents Keep Deleting User Files jai Stanford ai Agent Sandbox Offers a one Command fix
- Addy Osmani Open Sources Agent Skills 19 Workflows That Make ai Agents Code Like Google Engineers
- Google Rolls out File Generation in Gemini Chasing Chatgpt Canvas and Claude Artifacts
Top AI Product 
Leave a comment