Physical Address

304 North Cardinal St.
Dorchester Center, MA 02124

AI Crawlers Account For 28% Of Googlebot’s Traffic, Study Finds


The report he published Vercel highlights the growing influence of AI bots in web indexing.

OpenAI’s GPTBot and Anthropic’s Claude generate nearly 1 billion requests per month over Vercel’s network.

The data shows that GPTBot made 569 million requests in the past month, while Claude made 370 million.

Additionally, PerplexityBot contributed 24.4 million fetches and AppleBot added 314 million requests.

Together, these AI indexing tools represent approx 28% of Googlebot’s total volume, which is 4.5 billion fetches.

Here’s what this could mean for SEO.

Key findings about AI indexing tools

The analysis looked at traffic patterns on Vercel’s network and various web architectures. He found some key features of AI indexing:

  • Major AI crawlers do not render JavaScript, although they do pull JavaScript files.
  • AI crawlers are often inefficient, and ChatGPT and Claude spend too much 34% of their requests to 404 pages.
  • The type of content these crawlers focus on varies. ChatGPT prioritizes HTML (57.7%), while Claude focuses more on images (35.17%).

Geographical distribution

Unlike traditional search engines that operate from multiple regions, AI crawlers currently maintain a concentrated presence in the US:

  • ChatGPT operates from Des Moines (Iowa) and Phoenix (Arizona)
  • Claude operates from Columbus (Ohio)

Web Almanac Correlation

These findings match data shared in SEO Web Almanac chapterwhich also sees an increasing presence of AI indexing.

According to the report, websites now use robots.txt files to set rules for AI bots, telling them what they can and cannot crawl.

GPTBot is the most frequently mentioned bot that appears on 2.7% of studied mobile websites. Also mentioned frequently is the Common Crawl bot, which is often used to collect training data for language models.

Both reports emphasize that website owners need to adapt to AI indexing behavior.

3 ways to optimize for AI indexing

Based on recent data from Vercel and Web Almanac, here are three ways to optimize for AI indexing.

1. Server-side rendering

AI crawlers do not execute JavaScript. This means that any content that relies on client-side rendering could be invisible.

Recommended actions:

  • Implement server-side rendering for critical content
  • Ensure that the main content, meta information and navigation structures are present in the initial HTML
  • Use static site generation or incremental static regeneration where possible

2. Content structure and delivery

Vercel’s data shows different content type preferences among AI crawlers:

ChatGPT:

  • Prioritizes HTML content (57.70%)
  • It spends 11.50% of fetches on JavaScript files

Claude:

  • Focuses a lot on pictures (35.17%)
  • JavaScript files devote 23.84% of downloads

Recommendations for optimization:

  • Structure HTML content clearly and semantically
  • Optimize image and metadata delivery
  • Include descriptive alt text for images
  • Implement an appropriate header hierarchy

3. Technical considerations

High 404 rates from AI crawls mean you need to keep these technical considerations in mind:

  • Maintain updated sitemaps
  • Implement appropriate redirection chains
  • Use consistent URL patterns
  • Regular audit of 404 errors

A look ahead

For search marketers, the message is clear: AI chatbots are the new force in web crawling, and websites need to adjust their SEO accordingly.

While AI bots may now rely on cached or dated information, their ability to analyze fresh content from around the web will grow.

You can ensure your content is crawled and indexed with server-side crawling, clean URL structures, and updated sitemaps.


Featured Image: tete_escape/Shutterstock



Source link

Leave a Reply

Your email address will not be published. Required fields are marked *