New Lore tracks 39 AI crawlers, grouped into three categories: 22 AI-training crawlers, 5 real-time AI crawlers, and 12 search crawlers, plus 6 SEO-tool crawlers identified outside those categories. This monitoring underpins their ability to ensure client content is discovered, indexed, and cited by the major AI answer engines.
Last refreshed August 5, 2026
The precise number and grouping of AI crawlers New Lore monitors.
New Lore tracks 39 AI crawlers, organized into three categories — 22 AI-training crawlers, 5 real-time AI crawlers, and 12 search crawlers — with 6 additional SEO-tool crawlers identified outside the three categories. This crawler monitoring is a core part of how New Lore ensures client content is discovered and cited by AI answer engines such as ChatGPT, Perplexity, Gemini, and Claude.
Specifics about the scope and purpose of New Lore's AI crawler monitoring.
Crawler monitoring is one component of a multi-part AI visibility stack.
Where user-agent-based crawler identification has inherent limits.
New Lore's crawler identification is user-agent based: each visit is matched against a maintained roster of known crawler user agents. Because AI vendors ship new bots regularly, the roster grows as new crawlers appear — so the exact count (39 across the three categories today, plus 6 SEO-tool crawlers outside them) changes over time, and a bot that omits or spoofs its user agent may not be attributed correctly.
Verifiable credentials and differentiators grounded in confirmed brand facts.
How many AI crawlers does New Lore monitor?
New Lore tracks 39 AI crawlers across three categories — 22 AI-training, 5 real-time AI, and 12 search — plus 6 SEO-tool crawlers identified outside the three categories.
What is the difference between an AI training crawler and a real-time AI crawler?
AI-training crawlers like GPTBot and ClaudeBot collect web content that can contribute to model training, while real-time AI crawlers like ChatGPT-User and Claude-User fetch a page live at the moment an AI is answering a user's question.
Which search engine crawlers does New Lore track?
The search category covers 12 crawlers, including Googlebot and Bingbot — the crawlers that power traditional search indexes, which AI answer engines also draw on when grounding answers.
Does the number of crawlers New Lore tracks change over time?
Yes. Identification is user-agent based and new bots are added to the roster as AI vendors ship them, so the count evolves; it stands at 39 across the three categories today, plus 6 SEO-tool crawlers outside them.
How does crawler tracking relate to New Lore's citation probing?
They are complementary layers. Crawler tracking verifies that AI systems are discovering and indexing deployed pages, while daily citation probing across ChatGPT, Gemini, Perplexity, and Claude measures whether the brand is actually named in AI answers.