robots.txt generator
A valid robots.txt, with the AI crawlers listed explicitly. Free and no account. Nothing leaves your browser.
Google ignores this directive entirely. Bing and Yandex honour it.
Training crawlers
Blocking these keeps your content out of model training. It does not affect whether assistants cite you.
Retrieval crawlers
Blocking these removes you from AI answers. Most people who block them did not mean to.
robots.txt
User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /cart Disallow: /*?s= Sitemap: https://example.com/sitemap.xml
Save this as robots.txt at the root of your domain. Note that it controls crawling and not indexing: a disallowed URL can still appear in results if other sites link to it. Use a noindex meta tag to keep a page out of the index.
What it tells you
Separates training from retrieval crawlers
The distinction almost nobody explains, and the one that decides whether blocking a bot costs you anything.
Every AI crawler named
GPTBot, Google-Extended, ClaudeBot, CCBot, OAI-SearchBot and PerplexityBot, each with what it actually does.
Standard directives too
Disallow paths, a sitemap line, and an optional crawl delay with a note on who honours it.
Copy and paste
Valid output, ready for the root of your domain.
How to use it
- 1Set your sitemap and disallow pathsAdmin areas, carts and internal search results are the usual candidates.
- 2Decide about AI crawlersRead the two groups carefully. Blocking training crawlers costs you nothing today; blocking retrieval crawlers removes you from AI answers.
- 3Copy to your domain rootIt must live at yourdomain.com/robots.txt. Anywhere else and it is ignored.
Questions people ask
Should I block GPTBot?
It depends what you are protecting. GPTBot collects training data, so blocking it keeps your writing out of future models and costs you nothing in visibility. It is not the crawler that fetches your page when ChatGPT answers a question: that is OAI-SearchBot. People block both and then wonder why they are never cited.
Does blocking Google-Extended hurt my search rankings?
No. Google-Extended controls Gemini training only and is separate from Googlebot. Blocking it has no effect on Google Search crawling, indexing or ranking, which Google states directly.
Does robots.txt stop a page being indexed?
No, and this is the most common misunderstanding. It controls crawling. A disallowed URL can still appear in results if other sites link to it, usually with no description. To keep a page out of the index, allow the crawl and use a noindex meta tag, because a blocked page cannot be read to discover the noindex.
Where does the file go?
The root of the domain, at yourdomain.com/robots.txt, served as plain text. A robots.txt in a subdirectory is ignored, and each subdomain needs its own.
Does crawl delay work?
Bing and Yandex honour it. Google ignores it entirely and you set the crawl rate in Search Console instead.
Other free tools
Word counter
Words, characters, sentences, paragraphs and reading time, as you type.
Readability checker
Flesch reading ease and grade level, with the sentences dragging you down.
Keyword density analyzer
Which terms you actually used, and how often, including two and three word phrases.
SERP preview
How your title and meta description will truncate in Google, in pixels.
Character counter
Live character count against the limits for every field that has one.
Headline checker
Scores a headline on length, structure and the words that make people click.
Image resizer
Resize and compress an image without it ever leaving your machine.
Permalink generator
A clean, lowercase, ASCII URL slug from any headline.
UTM link builder
Tagged campaign links, with the casing normalised so reports do not split.
Title case converter
AP, Chicago, APA or sentence case, because they genuinely disagree.
Schema markup generator
Valid JSON-LD for FAQ, Article, Product and Organization.
llms.txt generator
The file that tells assistants what your site is and which pages matter.
Meta tag generator
Title, description, canonical and the social tags, in one block.
XML sitemap generator
Paste your URLs, get a valid sitemap.xml.
Hreflang tag generator
Language and region tags that point at each other properly.
301 redirect generator
Old URLs to new ones, as Apache, Nginx or Next.js rules.
Keyword mixer
Combine lists into every permutation, deduplicated.
AI answer readiness
Whether a page is shaped to be quoted in an AI answer.
AI meta description writer
Three descriptions at the right length, from what the page actually covers.
AI FAQ generator
The questions people actually type, with answers short enough to be quoted.
AI keyword intent classifier
What each keyword wants, and the one page type that should target it.
Want a tool like this on your own domain?
This page is the argument for the offer. A small useful utility gets embedded and linked, and every embed is a link back. kiterank builds one for your audience and hosts it on your domain, so the links accrue to you rather than to us.