Blog·Craft·

Bots get named, not just flagged

AI crawlers resolve with a name, category, and producer, not just a bot flag.

A bot flag tells you something showed up. A name tells you who.

Call /useragent with a GPTBot request and bot flips true. Ask for ?deep=true and the bot resolves to a full record.

{
  "device": "bot",
  "bot": true,
  "deep": {
    "bot": {
      "name": "GPTBot",
      "category": "AI Data Scraper",
      "url": "https://openai.com/gptbot",
      "producer": "OpenAI"
    },
    "ai": true
  }
}

Same shape for search crawlers. Googlebot resolves with device: "bot" too, ai comes back false. The flag and the name do different jobs: one for a fast filter, one for the log line you actually read later.

One flag, one field

ai is the fast check. Filter it in a WAF rule or a logging pipeline without touching name or category. Need the detail? deep.bot carries who built it and what it is for, plus a producer URL when the crawler publishes one.

Try it

Call /useragent with a raw User-Agent string and ?deep=true. deep.bot has the name.