robots.txt Explained: What It Actually Controls (and What It Doesn't)
One misplaced line in this file can quietly de-index an entire website. Here's what robots.txt really does — and the one thing people always assume it does but doesn't.
Key takeaways
- robots.txt controls crawling — whether a bot visits a page — not indexing, whether it appears in search results.
- A disallowed page can still appear in search results (usually without a snippet) if it's linked from elsewhere.
- "Disallow: /" with no matching Allow blocks that entire crawler from the whole site.
- The file must live at the exact root of the domain — example.com/robots.txt, never in a subfolder.
- It's a voluntary standard; well-behaved crawlers (Google, Bing) respect it, but it can't enforce access control.
The mistake that's easy to make and expensive to leave unnoticed
robots.txt is one of the smallest, plainest files on the internet — a handful of lines of text at the root of a domain — and also one of the easiest to get catastrophically wrong. A single "Disallow: /" left over from a staging environment, copied into production, tells every well-behaved crawler to stop visiting the entire site. Traffic quietly drops, and because nothing throws an error, it can go unnoticed for weeks.
What robots.txt actually controls
robots.txt is a plain-text file that tells crawlers which parts of a site they may and may not request. It sits at the domain root (example.com/robots.txt — not in any subfolder) and uses a simple syntax: a User-agent line naming which crawler the rule applies to, followed by Disallow and Allow lines naming paths.
- User-agent: * — applies to all crawlers by default
- Disallow: /private/ — blocks that path
- Allow: /private/public-page — carves out an exception within a blocked path
- Sitemap: https://example.com/sitemap.xml — points crawlers to the full page list
The single biggest misconception
robots.txt controls crawling, not indexing — and that distinction trips up almost everyone at some point. A page disallowed in robots.txt can still show up in Google's search results, usually as a bare URL with no title or description, if other pages link to it. Google knows the page exists (from the link) even though it never crawled the page's actual content, because robots.txt told it not to.
If the actual goal is keeping a page fully out of search results, the correct tool is a noindex meta tag or HTTP header on that page — not a robots.txt Disallow rule. The two solve different problems and aren't interchangeable.
“A disallowed page can still appear in search results — Google just won't have crawled what's on it.”
It's a request, not a lock
robots.txt is a voluntary standard. Major search engines and well-behaved crawlers respect it as a strong convention, but nothing about the file technically prevents a request from reaching a disallowed URL — it's not access control, and it shouldn't be relied on to keep anything genuinely private or secure. Anything that needs real protection needs authentication, not a robots.txt line.
Build one without the risk of a costly typo
Open Tools Library's robots.txt Generator starts from safe presets — allow everything, block everything, or block common admin paths — and specifically checks for the one mistake that costs the most: a rule that disallows "/" with no offsetting Allow, flagging it before you publish rather than after the traffic drop.
Mentioned in this post
robots.txt Generator
Generate a valid robots.txt file using quick presets or fully custom per-crawler Allow/Disallow rules, with a built-in warning if a rule would accidentally block your entire site from search engines.
XML Sitemap Generator
Paste a list of page URLs and generate a valid XML sitemap, with per-page control over last-modified date, change frequency, and priority — ready to copy or download as sitemap.xml.
Frequently asked questions
Does robots.txt stop a page from appearing in Google?
Not reliably — it stops crawling, but a linked page can still appear in search results without a crawled snippet. Use a noindex tag for pages that must stay out of results entirely.
Where does robots.txt need to be placed?
At the exact root of the domain — https://yourdomain.com/robots.txt. It won't be read from any subfolder.
What does a blank "Disallow:" line mean?
It means disallow nothing — functionally the same as allowing everything for that user-agent.
Can robots.txt block a specific bad bot?
You can add a rule group targeting that bot's specific user-agent name, but since the standard is voluntary, it only works against crawlers that choose to respect it.