AKARUMI INSIGHTS: AI Bots Access 'robots.txt' and 'sitemap' Most Frequently

ipe inc. (hereafter ipe) analyzed the AI bot access to the website of its proprietary AI writing tool 'DeepEditor' over the past 30 days, using 'AKARUMI', a visibility tool designed for the AI search era.

With information gathering increasingly originating from Generative AI platforms like ChatGPT and Gemini, it has become crucial for companies to understand how they are recognized and referenced by AI.

Here, we introduce excerpts from the report analyzing which pages AI bots were accessing.

Summary of Findings

In this analysis, the pages most frequently accessed by AI bots were not article or service pages, but site structure-related pages such as /robots.txt and /sitemap.xml.

Additionally, differences in access behavior were observed among ClaudeBot, GPTBot, and PerplexityBot. ClaudeBot predominantly accessed site structure pages, while GPTBot's access was more distributed, including pricing and article pages.

In the era of AI search, understanding exactly which pages AI bots are looking at has become essential.

'robots.txt' and Sitemap-related Pages Received the Most Overall AI Bot Access

Aggregating access from all AI bots, the most visited page was /robots.txt. This was followed by the root domain (/), /sitemap.xml, and /sitemap.rss.

According to the data, the total visits to /robots.txt, /sitemap.xml, and /sitemap.rss reached 249, accounting for approximately 61.5% of the access within the top 10 pages.

These results suggest that AI bots are checking not only the main text of the pages but also crawlability, URL structure, and update information.

Access Trends Vary by AI Bot

When looking at individual AI bots, distinct differences in access trends emerged.

ClaudeBot had a high ratio of access to /robots.txt, /sitemap.xml, and /sitemap.rss, focusing primarily on site structure pages.

On the other hand, GPTBot accessed sitemap-related pages but also visited pricing pages, article pages, case study pages, and news pages.

Meanwhile, PerplexityBot focused mostly on the top page and /robots.txt. These results indicate that different AI bots may be verifying different types of information and serving varying roles.

Website Improvement Points for the AI Search Era

This survey confirmed AI bot access to pricing pages, feature pages, solution pages, and article pages.

In the AI search era, it is increasingly important to organize pricing, feature differences, use cases, and implementation conditions to design information in a way that AI can easily understand.

Furthermore, clearly organizing the top page to define 'what the service is,' 'who it is for,' and 'what problems it solves' is considered critical for helping AI grasp the site's content.

'AKARUMI': Visualizing AI Bot Access

With AKARUMI, businesses can visualize access from AI bots like ChatGPT, Claude, and Perplexity, and analyze exactly which pages are being viewed by AI.

By confirming AI bot access, which is difficult to grasp using only GA4 or GSC, companies can better understand which pages to strengthen in the AI search era and prioritize improvements for LLMO.

About ipe inc.

ipe inc. is a digital marketing company that supports businesses through SEO and LLMO consulting to ensure they are correctly recognized, quoted, and selected by search engines and generative AI.

For over 13 years, the company has provided SEO support, primarily for large-scale and database-driven sites, assisting more than 300 companies. Currently, ipe leverages this expertise to enhance LLMO support while offering 'AKARUMI,' an analysis platform that visualizes brand exposure on AI search engines and LLMs.

Company Name: ipe inc.

Headquarters: Aoyama Tower Building 5F, 2-24-15 Minamiaoyama, Minato-ku, Tokyo 107-0062

Established: October 2013

Business Operations: Web consulting, media business, advertising business, etc.

Corporate Website: https://ipeinc.jp/

Phone: 03