Guide
robots.txt: what it blocks, and what it doesn't
robots.txt guides crawlers, not visitors. Here is what it blocks, what it doesn't, and why it does not protect your data.
Same URL, different link order depending on Instagram/TikTok/YouTube. Configure once. Korli applies the rest automatically.
- Examples: TikTok visitor sees offer first, Instagram visitor sees proof first
- Common mistakes: too many links, same order for everyone, no tracking
- Setup: create page → set Smart Rules → measure clicks (30 seconds)
Proof demo: source switch → different order
Same URL, but a different rendering based on where the visitor comes from. Configure once. Korli applies the rest automatically.
Definition: an instruction file, not a wall
- robots.txt is a text file at the root of the domain. It gives directives to crawlers visiting your site.
- It does NOT prevent visits. A human visitor sees whatever they want.
- Disallow means "don't crawl", not "doesn't exist". A blocked page stays reachable if someone knows the URL.
What it actually does on korli.fr
- The file allows the whole site and blocks /dashboard, /admin, /api: areas that should not be in an index.
- It explicitly allows GPTBot, ClaudeBot, PerplexityBot: those crawlers can read the site so it can be cited in AI answers.
- It declares two sitemaps so Google discovers new pages faster.
What robots.txt does NOT do
- It protects nothing. A sensitive file stays readable if you block crawlers but leave the URL public.
- It does not deindex. To deindex a page you need noindex in its <head>, not robots.txt.
- Blocking an already-indexed page makes it SLOWER to disappear: Google must recrawl to notice the block.
Concrete example
Example: why Korli lets AI crawlers in
- Your content can be cited by an AI assistant when someone asks about the topic.
- Blocking GPTBot prevents those citations: you become invisible in generated answers.
- Korli allows those crawlers explicitly while keeping /dashboard and /api blocked.
- Result: you are readable everywhere, but your dashboards are not indexed.
How it works (3 steps)
- Write the file at the root: https://korli.fr/robots.txt
- List directories to block (dashboard, admin, api) with Disallow.
- Declare sitemaps at the end of the file to speed up discovery.
Common mistakes
- Blocking the whole site thinking you protect against copy-paste.
- Putting noindex in robots.txt: that tag is not read by Google.
- Blocking GPTBot out of habit and silently losing AI citations.
In the same category
Click-through rate (CTR): measuring it without fooling yourself
Click-through rate is clicks divided by impressions. Here is the formula, what moves it, and why yours can look low at the same position.
Deep link: opening the app instead of the browser
A deep link opens a mobile app directly instead of a browser. Here is how Korli uses it and why it changes your click-through rate.
Link in bio: the definition, and what actually changes
A link in bio is one page holding all your links. Here is the definition, the limits of the static version, and what Korli changes.
Shortlink: shortening a URL and measuring the click
A shortlink shortens a long URL. Here is what it brings, what it hides, and why a link-in-bio does not need one.
Conversion rate: measuring it properly on your page
Conversion rate is actions over visitors. Here is the formula, what makes it misleading, and the right method on Korli.
Tip: read 2 pages in the same cluster, then refine your link order by source.
Read next
Too many links (cluttered page): quick diagnosis + action plan
An actionable guide: 3-point diagnosis, one concrete scenario (Instagram order vs newsletter order), 3-step how-it-works, common mistakes, and useful links.
Link in bio zero clicks: diagnosis + quick fixes
An actionable guide: common causes, a Korli example (Instagram vs TikTok), a 3-step method, common mistakes, and a short FAQ.
Canonical: the attribute that prevents duplicate content
The canonical attribute tells Google which URL is the reference version. Here is when it matters, and the classic mistake on sites with similar pages.
Tip: read 2 pages in the same cluster, then refine your link order by source.
FAQ
How do I check my robots.txt?
Open https://korli.fr/robots.txt directly in a browser to see the raw text. You can also use Google Search Console's robots testing tool to verify blocks.
Does blocking a URL in robots.txt remove it from Google?
No, and it is counter-intuitive. If the page was already indexed, blocking only prevents seeing the change. To deindex it, add noindex in the page's <head>.
Should I allow GPTBot and other AI crawlers?
If you want to be cited in ChatGPT, Perplexity or Claude answers, yes. Blocking GPTBot makes you invisible there. It is a choice: generative visibility vs full crawl control.
CTA
Configure your smart rules once. Korli applies the rest automatically.