AI Crawler Control and Management

Understand AI search, user-requested retrieval, and model-training access on your changelog.

All Articles

AI Crawler Control and Management

Manage access for AI search, user-requested retrieval, and model-training crawlers. These uses are different: appearing in an AI search answer does not require allowing model training.

Search, Retrieval, and Training

  • AI search: OAI-SearchBot, PerplexityBot, and Claude-SearchBot discover content for search answers.
  • User-requested retrieval: ChatGPT-User, Perplexity-User, and Claude-User may visit a page when someone asks an assistant about it.
  • Model training: GPTBot and ClaudeBot collect content that may be used to develop AI models.
  • Google-Extended: Controls Gemini training and grounding. It does not control inclusion in Google Search, AI Overviews, or AI Mode.

Google Search's AI features use Googlebot and normal search indexing controls. User-requested fetchers may not honor robots.txt in the same way as automatic crawlers.

Paid Account Settings

The current AI Crawler Access setting covers supported search, retrieval, and training agents together. It is separate from the Search Engine Indexing setting.

  • Block: Requests that supported AI agents do not access the changelog, including search and training crawlers.
  • Allow: Adds no AI-specific rules. Agents follow the general robots.txt rules, and content may be used for training.
  • Optimize: Explicitly allows supported AI agents, including training crawlers, and adds structured data. A sitemap is advertised when search indexing is also set to Optimize.

These rules include the current OpenAI, Perplexity, and Anthropic agent names, alongside legacy and other supported agents. Access does not guarantee indexing, citations, or recommendations.

Free Accounts with Custom Domains

A free changelog on its verified, active custom domain is automatically eligible for search indexing when it is public and has no password. On that hostname, ChangeCrab explicitly allows OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User, Perplexity-User, and Claude-User.

This automatic policy keeps GPTBot, ClaudeBot, Google-Extended, and the other supported agents outside that search and retrieval group restricted. It does not open model-training access, even if an older saved AI mode was Allow or Optimize.

The exception applies only to the active custom hostname. Free ChangeCrab-hosted copies and unverified custom domains remain excluded from indexing and do not receive this exception. Crawler settings remain a paid feature.

Changing Your Settings

  1. Open your changelog Settings.
  2. Find AI Crawler Access in the privacy and visibility settings.
  3. Choose Block, Allow, or Optimize.
  4. Save your changes.

Privacy and Access

Robots.txt expresses crawler preferences; it is not access control and cannot guarantee that every agent respects your rules. Private and password-protected changelogs remain protected regardless of AI settings or custom-domain status. Use those features for content that should not be public.

Checking Your Rules

Visit /robots.txt on the hostname you want to check. For each supported agent, Allow: / permits crawling and Disallow: / requests exclusion. Search indexing also depends on the page's robots meta tag. Check server logs to confirm that permitted crawlers can retrieve your public pages; crawler traffic alone does not prove AI citations.

Related Features

Crawler Documentation