AI Bot & Search Permissions Builder (robots.txt)
Decide who is welcome on your website. Keep Google, Bing, and AI answer citations turned on — or block AI companies from using your writing to train future models.
Googlebot, Bingbot, OAI-SearchBot, and PerplexityBot set to Allow so your links keep getting citations and clicks, while setting model-training crawlers like GPTBot, ClaudeBot, and Google-Extended to whichever policy fits your goals.AI Search Citations vs. AI Model Training Bots
Many website owners accidentally block every AI bot when they really only want to opt out of AI training. Major platforms maintain separate user-agents for search citations versus training datasets:
| Provider | Search / Live Citation Bot | Model Training Bot | Suggested Setting |
|---|---|---|---|
| OpenAI / ChatGPT | OAI-SearchBot, ChatGPT-User | GPTBot | Allow Search; pick per preference |
| Googlebot, Googlebot-Image | Google-Extended | Always Allow Googlebot & Image | |
| Anthropic / Claude | Claude-SearchBot, Claude-User | ClaudeBot | Allow Search for citations |
| Perplexity | PerplexityBot, Perplexity-User | None separate | Allow for citation links |
| Apple | Applebot | Applebot-Extended | Always Allow Applebot |
Hiding Checkout, Cart, and Private Folders Safely
Even on small websites, crawlers spend unnecessary requests scanning shopping carts, account logins, admin panels, and internal search pages. Listing folders like /admin/, /cart/, and /checkout/ keeps bots focused on the public articles and product pages you actually want indexed.
Remember never to block /favicon*.png, /favicon.ico, or preview image routes in robots.txt, because Google requires access to display your brand logo next to search results. Want ready-made presets? Check our templates library.