ccBot crawler

Common Crawl open web crawl — the public corpus many AI labs train on.

AI crawler, used for model training and/or live retrieval.

AI
crawler

Official: commoncrawl.org/faq·Vendor: Common Crawl

Parse it

Send the User-Agent to the API, device, OS, browser, and bot detection come back in one response. ccBot crawler is identified by name when the string matches.

curl -s "https://api.parseapi.com/useragent" \
  -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15" \
  -H "X-Api-Key: $PARSEAPI_KEY"

Create a free key Useragent API