Short answer: Robots.txt is a plain-text file at the root of your domain that tells search engine crawlers which parts of your site they may or may not crawl. It controls crawling, not indexing — a blocked page can still appear in results, so use it for crawl management, not for hiding pages.
What is robots.txt?
The robots.txt file follows the Robots Exclusion Protocol, a standard every major crawler respects. It lives at https://yourdomain.com/robots.txt and is the first file a well-behaved bot requests before crawling your site. Inside, you write rules that grant or deny access to specific paths for specific user agents.
User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /
Sitemap: https://cyberdefence.org.in/sitemap.xmlThis example tells all bots (User-agent: *) to stay out of /admin/ and /cart/, allows everything else, and points them to the sitemap.
Crawling vs indexing: the crucial distinction
The single biggest misunderstanding about robots.txt is thinking it hides pages from Google. It does not. Disallow stops a bot from crawling a URL, but the URL can still be indexed if other pages link to it — Google just shows it without a description.
- To stop crawling: use
robots.txtDisallow. - To stop indexing: use a
<meta name="robots" content="noindex">tag or anX-Robots-TagHTTP header — and let the page stay crawlable so Google can see the noindex directive.
Blocking a page in robots.txt and adding noindex is a classic mistake: if the bot cannot crawl the page, it never reads the noindex, so the page can linger in the index.
Core robots.txt directives
| Directive | Purpose | Example |
|---|---|---|
| User-agent | Which crawler the rules apply to | User-agent: Googlebot |
| Disallow | Path the bot should not crawl | Disallow: /private/ |
| Allow | Exception inside a disallowed path | Allow: /private/public.html |
| Sitemap | Location of the XML sitemap | Sitemap: https://site.com/sitemap.xml |
| Crawl-delay | Seconds between requests (Bing/Yandex; Google ignores) | Crawl-delay: 10 |
How to write a robots.txt file
1. Allow crawling of what you want ranked
By default, if there is no robots.txt or it is empty, crawlers assume everything is fair game. Only add Disallow for paths you genuinely want kept out of the crawl.
2. Block low-value and private paths
Typical candidates: admin dashboards, internal search results, cart and checkout, staging URLs, faceted-navigation parameters that create infinite crawl traps, and duplicate print views.
3. Never block CSS and JS
Google renders pages like a browser. If you block your stylesheets or scripts, it may see a broken layout and misjudge quality or mobile-friendliness. Keep resource folders crawlable.
4. Always declare your sitemap
Add a Sitemap: line so crawlers find your URL list even if you have not submitted it in Search Console yet.
Robots.txt and AI crawlers in 2026
A newer job for robots.txt is controlling AI crawlers that gather training and answer-engine data. You can allow or block bots like GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /This is a business decision, not a pure SEO one. Blocking AI crawlers can protect content but may reduce your chances of being cited in ChatGPT, Perplexity or AI Overviews — the growing field of Generative Engine Optimization (GEO). We help clients weigh this trade-off case by case.
Robots.txt vs meta robots vs canonical
Beginners often reach for robots.txt to solve problems it was never designed for. Each tool has a distinct job, and using the wrong one causes the classic "why is my page still in Google" headache.
| Tool | Controls | Use it to |
|---|---|---|
| robots.txt Disallow | Crawling | Stop bots wasting budget on low-value paths |
| meta robots noindex | Indexing | Keep a crawlable page out of search results |
| rel=canonical | Duplication | Point ranking signals to the preferred URL |
| X-Robots-Tag | Indexing (non-HTML) | Noindex PDFs, images and other files |
The rule of thumb: use robots.txt for crawl management, meta robots or X-Robots-Tag for index control, and canonical tags for duplication. They are complementary layers, not interchangeable switches.
A sample robots.txt for a typical business site
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?s=
Sitemap: https://cyberdefence.org.in/sitemap.xmlThis blocks the admin area and cart, keeps the AJAX endpoint reachable, stops internal search-result URLs from being crawled, and points bots to the sitemap. It leaves everything else — including CSS and JS — fully crawlable, which is exactly what you want.
Common robots.txt mistakes
- Blocking the whole site with
Disallow: /left over from staging — a catastrophic and common launch bug. - Blocking CSS/JS and breaking how Google renders the page.
- Using robots.txt to hide sensitive data — the file is public, so you are advertising the paths you want hidden.
- Combining Disallow with noindex so the noindex is never read.
- Case and path errors — directives are case-sensitive and match from the path start.
How to test your robots.txt
- Open
yourdomain.com/robots.txtin a browser to confirm it loads. - Use the robots.txt report and URL Inspection tool in Google Search Console.
- Recheck after every deployment — a bad push can silently block your whole site.
How crawl budget relates to robots.txt
On large websites, robots.txt is also a crawl-budget tool. Every request Googlebot spends on a useless URL — an internal search page, a faceted-navigation combination, a session-parameter duplicate — is a request it did not spend on a page you actually want ranked. Disallowing those low-value patterns concentrates crawling on your important content. For an ecommerce store with millions of filter combinations, sensible Disallow rules can be the single biggest factor in whether new products get discovered quickly. Just remember the trade-off: a disallowed URL cannot pass its link signals or be re-evaluated, so reserve blocking for paths that genuinely add no search value.
Robots.txt works hand in hand with your XML sitemap and your indexing directives. Together they form the crawl-control layer of technical SEO — see how they connect in our guide to crawling and indexing.
At Cyber Defence (cyberdefence.org.in), technical SEO is part of our SEO and AI-SEO service from ₹15,000/month. Founder Amit Kumar (CEH, CRTA, AI-SEO expert) leads the team from Hisar, working remotely across India. We are ISO-certified and GeM-registered. We are honest about timelines: SEO takes months, and no one can guarantee a #1 rank. If you want to learn this yourself, our SEO and AI-SEO course covers it hands-on. Call +91-75175-72000.
For a large site — a multi-location IT company in Delhi or an ecommerce catalogue — getting robots.txt wrong can deindex thousands of pages overnight, so it is worth a professional review. New to the basics? Start with what SEO is.
FAQ
What is robots.txt used for?
It tells search engine crawlers which parts of your site they may or may not crawl, helps manage crawl budget, and points bots to your sitemap. It controls crawling, not indexing.
Does robots.txt stop a page from appearing in Google?
No. Blocking a URL stops crawling, but the page can still be indexed if other pages link to it. To keep a page out of results, use a noindex tag and leave it crawlable.
Where should robots.txt be located?
It must sit at the root of your domain, at yourdomain.com/robots.txt. Placing it in a subfolder means crawlers will not find or apply it.
Should I block CSS and JavaScript in robots.txt?
No. Google renders pages using those files. Blocking them can make your page look broken to Google and hurt rankings and mobile evaluation.
Can robots.txt block AI crawlers like GPTBot?
Yes. You can add rules for AI user-agents such as GPTBot, ClaudeBot and Google-Extended. Just weigh the loss of potential AI citations against your content-protection goals.
Is robots.txt a security tool?
No. The file is publicly visible, so listing sensitive paths actually reveals them. Protect private data with authentication, not robots.txt.
Staging vs production robots.txt
One of the most damaging bugs in web launches is shipping a staging robots.txt to production. During development, teams often add Disallow: / to keep the unfinished site out of Google. If that line survives the go-live, the entire site is blocked from crawling and traffic collapses within days. Make removing it a mandatory step in your launch checklist, and verify yourdomain.com/robots.txt immediately after deployment. Automated drift monitoring or a simple post-deploy check that fetches the file and fails the build on a blanket Disallow can save you from this expensive, silent mistake.
Want your crawl setup audited before it costs you rankings? Cyber Defence does it as part of SEO and AI-SEO from ₹15,000/month, led by Amit Kumar (CEH, CRTA), Hisar and remote across India. Call +91-75175-72000.