Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
The report he published Vercel highlights the growing influence of AI bots in web indexing.
OpenAI’s GPTBot and Anthropic’s Claude generate nearly 1 billion requests per month over Vercel’s network.
The data shows that GPTBot made 569 million requests in the past month, while Claude made 370 million.
Additionally, PerplexityBot contributed 24.4 million fetches and AppleBot added 314 million requests.
Together, these AI indexing tools represent approx 28% of Googlebot’s total volume, which is 4.5 billion fetches.
Here’s what this could mean for SEO.
The analysis looked at traffic patterns on Vercel’s network and various web architectures. He found some key features of AI indexing:
Unlike traditional search engines that operate from multiple regions, AI crawlers currently maintain a concentrated presence in the US:
These findings match data shared in SEO Web Almanac chapterwhich also sees an increasing presence of AI indexing.
According to the report, websites now use robots.txt files to set rules for AI bots, telling them what they can and cannot crawl.
GPTBot is the most frequently mentioned bot that appears on 2.7% of studied mobile websites. Also mentioned frequently is the Common Crawl bot, which is often used to collect training data for language models.
Both reports emphasize that website owners need to adapt to AI indexing behavior.
Based on recent data from Vercel and Web Almanac, here are three ways to optimize for AI indexing.
AI crawlers do not execute JavaScript. This means that any content that relies on client-side rendering could be invisible.
Recommended actions:
Vercel’s data shows different content type preferences among AI crawlers:
ChatGPT:
Claude:
Recommendations for optimization:
High 404 rates from AI crawls mean you need to keep these technical considerations in mind:
For search marketers, the message is clear: AI chatbots are the new force in web crawling, and websites need to adjust their SEO accordingly.
While AI bots may now rely on cached or dated information, their ability to analyze fresh content from around the web will grow.
You can ensure your content is crawled and indexed with server-side crawling, clean URL structures, and updated sitemaps.
Featured Image: tete_escape/Shutterstock