All features

AI Crawlers: Which AI Bots Read Your Site, and Which Pages

Connect Cloudflare and see every visit AI bots make to your site: when an AI reads a page to answer someone, when a search engine indexes it, when a model collects it for training. Page by page, next to the citations each page earns.

Start free trial
AI crawlers · last 7 days

Answer reads

312

Search indexing

1,140

Training crawls

2,486

Most crawled pages

/pricing418cited 9 times
/lemlist-vs-instantlyGenerated236cited 4 times
/blog/cold-email-guide191not cited

From Cloudflare, synced every night.

Key Takeaways

  • AI bots visit your site for three different reasons: to read a page while answering someone, to add it to an AI search index, or to collect it to train a model. Mentionable counts each kind separately.
  • The AI crawlers page shows every visit over the period, the trend per day, the most crawled pages with how often each one is cited in AI answers, the visits per bot and the errors your site served to AI bots.
  • For a page published with Mentionable, you follow the delay from publication to the first AI visit, then to the first citation.
  • The visits come from your Cloudflare account, through a read-only connection. History starts 7 days before you connect, then syncs every night.
  • This page covers bots. Human visitors who click through from an AI answer are on the AI visits page, read from Google Analytics.

You published a comparison page two weeks ago. Your tracking says no AI engine cites it yet. Fair enough. But here is the question nobody can answer for you: has any AI even read it?

If ChatGPT never opened the page, the problem is not your content. It is that the page was never in the running.

What does the AI crawlers page show?

Mentionable reads the visits AI bots make to your site and shows, on one page, who came, what they read and what it led to.

Four numbers at the top. Total AI bot visits over the period, compared with the previous one, then the same visits split into answer reads, search indexing and training crawls.

Visits over time. A chart per day and per kind of visit. Orange lines mark the days you published a page or logged an action, so you compare before and after without guessing.

The most crawled pages. Every page AI bots loaded, with its visits, its answer reads, the number of times it is cited in AI answers on your tracked questions, and the day of its last visit. Pages published with Mentionable carry a marker. You can search a page, or filter by the HTTP code your site served.

Visits per bot. Which bots came, and how often. Filters keep one company (OpenAI, Anthropic, Perplexity…) or one bot.

Errors served to AI bots. The pages that answered an AI bot with an error. A page that cannot be read cannot be cited.

What are the three kinds of AI bot visits?

Not every AI visit means the same thing, which is why Mentionable never adds them up into one vanity number.

  • Answer reads. An AI fetched the page while answering someone: ChatGPT-User, Claude-User, Perplexity-User. A real question is behind the visit. It is the closest signal to an answer built on your page.
  • Search indexing. An engine adds the page to its search index: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot. Without indexing, the page cannot come up in answers.
  • Training crawls. A company collects the page to train future models: GPTBot, ClaudeBot, Bytespider, CCBot. No direct effect on today's answers.

A page with plenty of training crawls and no answer reads is known to the models, but nobody's question leads to it yet.

Which AI bots does Mentionable track?

The bots of the companies whose assistants your customers use:

Company Bots
OpenAI GPTBot, OAI-SearchBot, ChatGPT-User
Anthropic ClaudeBot, Claude-SearchBot, Claude-User
Perplexity PerplexityBot, Perplexity-User
Mistral MistralAI-User
DuckDuckGo DuckAssistBot
Apple Applebot
Amazon Amazonbot
Meta Meta-ExternalAgent, Meta-ExternalFetcher
ByteDance Bytespider
Common Crawl CCBot

How long does it take for a new page to be read, then cited?

Open any page from the table and you get its full story: visits per day, the HTTP codes it served to AI bots, and the engines that cite it.

For a page you published with Mentionable, a timeline lines up three dates: the publication, the first AI visit (and which bot made it), and the first citation (and which engine gave it), each counted in days after publication. When a step has not happened yet, the timeline says so.

That is how you tell a page that is waiting to be found from a page that was read and still not chosen.

How do I connect Cloudflare?

Your site has to be served through Cloudflare with the proxy on, the orange cloud. A site that only uses Cloudflare for DNS sends no visits.

  1. Open AI crawlers in your project and click Connect Cloudflare.
  2. Approve the read-only access in Cloudflare. It reads your zones and their analytics, nothing else.
  3. Pick the zone that serves your site.

Mentionable then fetches the last 7 days of visits, which takes a few seconds, and syncs every night after that. You can change the zone, reconnect or disconnect from the bottom of the page. Disconnecting stops the sync and keeps the history.

What is the difference between AI crawlers and AI traffic?

AI crawlers are bots. AI traffic is people.

The AI crawlers page counts the visits AI bots make to read, index or collect your pages, from Cloudflare. The AI visits page counts the people who click a link in an AI answer and land on your site, from Google Analytics. One tells you whether the engines read you, the other whether their answers send you visitors.

Can my own AI read the crawler data?

Yes, through the Mentionable MCP server. The get_ai_crawler_overview, list_ai_crawler_pages, get_ai_crawler_page and list_ai_crawler_visits tools return the same visits, split by kind, crossed with each page's citations, with the delay from publication to first visit and first citation for pages published with Mentionable.

Which plans include AI crawler tracking?

All of them, and the 7-day free trial too. The only requirement is a site served through Cloudflare. Read-only team members see the page once someone with edit rights has connected it.

Curious if AI mentions your brand?

Run a free scan and see where you stand on ChatGPT.

Free AI Scan

Keep Reading

Frequently Asked Questions

Which AI bots visit my website?

The main AI companies each run several bots. OpenAI runs GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic runs ClaudeBot, Claude-SearchBot and Claude-User. Perplexity runs PerplexityBot and Perplexity-User. Mentionable also counts MistralAI-User, DuckAssistBot, Applebot, Amazonbot, Meta-ExternalAgent, Meta-ExternalFetcher, Bytespider and CCBot. Once your site is connected, the AI crawlers page shows which ones came and which pages they read.

What is the difference between ChatGPT-User and GPTBot?

ChatGPT-User fetches a page while ChatGPT is answering someone: a real question is behind the visit. GPTBot collects pages to train future OpenAI models and has no direct effect on today's answers. Mentionable calls the first kind an answer read and the second a training crawl, and counts them apart, along with search indexing bots such as OAI-SearchBot.

Do I need Cloudflare to track AI crawlers?

Yes. Mentionable reads AI bot visits from Cloudflare, so your site must be served through Cloudflare with the proxy on (the orange cloud). A site that only uses Cloudflare for DNS sends no visits. If your site is not behind Cloudflare, the page explains how to connect it and stays empty until then.

Is the Cloudflare connection safe?

The connection is read-only: it reads your zones and their analytics, and changes nothing on your site or in your Cloudflare settings. You can revoke it from Cloudflare at any time, or disconnect it from Mentionable. Disconnecting stops the nightly sync and keeps the history already fetched.

How far back does the AI crawler history go?

When you connect, Mentionable fetches the last 7 days of visits. After that, it syncs every night, so the page shows data through yesterday, and the history grows from the day you connected.

Can I see whether an AI read a page before citing it?

Yes. The most crawled pages table puts, next to each page, its visits, its answer reads and how many times it is cited in AI answers on your tracked questions. Opening a page shows its visits per day, the HTTP codes it served and which engines cite it. For a page published with Mentionable, a timeline shows the publication date, the first AI visit and the first citation.

Is AI crawler tracking included in every plan?

Yes. The AI crawlers page is included in every Mentionable plan and in the 7-day free trial. It only needs a site served through Cloudflare.

You can see where you stand today.

Free trial. Start tracking your AI visibility, no credit card.