Short answer: Crawling is when search engines like Google discover and download your web pages using bots. Indexing is when they analyse and store those pages in a giant database. A page must be crawled and indexed before it can rank — if either step fails, the page is invisible in search.
How Google search works: the three stages
Every page that appears in Google passes through three distinct stages. Understanding them is the foundation of technical SEO, because most "my page isn't ranking" problems are really "my page isn't indexed" problems.
- Crawling — Googlebot discovers URLs and downloads their content.
- Indexing — Google processes and stores the page in its index.
- Serving (ranking) — when someone searches, Google picks and orders relevant indexed pages.
What is crawling?
Crawling is the discovery phase. Googlebot follows links from pages it already knows, reads your XML sitemap, and downloads the HTML, CSS and JavaScript it finds. Modern Googlebot renders pages much like a browser, so it executes JavaScript to see content loaded dynamically.
How crawlers discover URLs
- Following internal and external links.
- Reading sitemaps you submit.
- URLs submitted via the Indexing API or IndexNow.
- Previously known URLs re-crawled on a schedule.
Crawl budget
Crawl budget is the number of URLs Googlebot will crawl on your site in a given period. Small sites rarely worry about it, but large sites do. You protect it by fixing broken links, avoiding infinite URL parameter traps, removing duplicate pages and keeping your server fast.
What is indexing?
Once crawled, a page is analysed: Google reads its content, evaluates quality, identifies the canonical version among duplicates, processes structured data, and decides whether to store it in the index. Only indexed pages are eligible to rank.
Crucially, crawled does not mean indexed. Google may crawl a page and then choose not to index it if it is thin, duplicate, low-value, or explicitly told not to.
| Stage | What happens | What can block it |
|---|---|---|
| Crawling | Bot discovers and downloads the URL | robots.txt Disallow, server errors, no links |
| Indexing | Page is analysed and stored | noindex tag, canonical to another URL, thin/duplicate content |
| Serving | Page ranked for a query | Relevance, authority, competition |
How to control crawling and indexing
Robots.txt (crawling)
Use robots.txt to stop bots crawling low-value paths. Remember it controls crawling, not indexing — a disallowed page can still be indexed if linked from elsewhere.
Meta robots / X-Robots-Tag (indexing)
To keep a page out of the index, use a noindex directive and leave the page crawlable so Google can read it.
<meta name="robots" content="noindex, follow">Canonical tags (duplication)
When similar pages exist, a rel="canonical" tag tells Google which version to index, consolidating ranking signals onto one URL.
<link rel="canonical" href="https://cyberdefence.org.in/blog/what-is-seo/">How JavaScript affects crawling and indexing
Modern sites lean heavily on JavaScript, and that changes how crawling works. Googlebot processes JS in two waves: it first crawls the raw HTML, then queues the page for rendering, where it executes JavaScript to see content loaded on the client. This second wave can be delayed, so content that only appears after JS runs may be discovered and indexed later — or, if rendering fails, not at all.
The safest approach for SEO-critical content is server-side rendering (SSR) or static generation, so the important HTML is present in the first response. If you rely on client-side rendering, test with the URL Inspection tool's rendered-HTML view to confirm Google actually sees your content. Many "not indexed" mysteries on React or single-page apps trace back to content that never made it into the rendered output.
Crawl budget: when it actually matters
Crawl budget is the number of URLs Googlebot will fetch from your site in a given window. For a site under a few thousand pages it is rarely a concern — Google can comfortably crawl everything. It becomes critical for large sites, where wasted crawling on duplicates, parameters and dead URLs means important pages get visited less often.
| Factor | Effect on crawl budget |
|---|---|
| Fast, reliable server | Increases crawl rate |
| Duplicate and parameter URLs | Wastes budget |
| Broken links and 404s | Wastes budget |
| Strong internal linking | Directs budget to key pages |
| Clean sitemap and lastmod | Helps prioritise re-crawls |
Why pages fail to get indexed
- Blocked by robots.txt so Google never crawls them.
- Noindex tag left over from staging or a plugin default.
- Canonicalised to a different URL, so only that one is indexed.
- Thin or duplicate content Google judges not worth storing — shown as "Crawled – currently not indexed."
- Orphan pages with no internal links pointing to them.
- Slow or unreliable servers that return errors during crawling.
How to get pages crawled and indexed faster
- Submit an up-to-date XML sitemap in Google Search Console and Bing Webmaster Tools.
- Build strong internal links so no page is an orphan.
- Use the URL Inspection tool to request indexing for important new pages.
- Adopt IndexNow to notify Bing and Copilot instantly when content changes.
- Earn backlinks — new sites with links get discovered faster.
- Keep the site fast and free of crawl errors.
Be realistic: indexing can take hours or weeks, and Google is under no obligation to index every page. This is one reason SEO takes months — discovery and indexing are only the entry ticket, before ranking competition even begins.
Monitoring indexation in Google Search Console
Search Console is the authoritative place to understand your site's crawl and index health. The Pages report groups every URL into "indexed" and "not indexed" buckets, and the not-indexed reasons are diagnostic gold: "Crawled – currently not indexed" usually signals quality or duplication problems; "Discovered – currently not indexed" often means crawl-budget or priority issues; "Excluded by noindex tag" and "Blocked by robots.txt" point to configuration you can fix directly. Review these categories monthly, fix the largest buckets first, and use the URL Inspection tool to confirm individual pages after changes. Treat indexation as an ongoing metric to manage, not a one-time setup — on an active site, pages constantly move between these states as content, links and site health change.
Crawling, indexing and AI answer engines
AI engines have their own crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — that must also access your pages to cite them. The same fundamentals apply: if a page is blocked or non-indexable, it cannot be crawled or quoted by AI. Making content crawlable, fast and well-structured serves both Google and Generative Engine Optimization (GEO). See how these disciplines connect in our overview of the types of SEO.
At Cyber Defence (cyberdefence.org.in), technical SEO is part of our SEO and AI-SEO service from ₹15,000/month. Founder Amit Kumar (CEH, CRTA, AI-SEO expert) leads the team from Hisar, working remotely across India. We are ISO-certified and GeM-registered. We are honest about timelines: SEO takes months, and no one can guarantee a #1 rank. If you want to learn this yourself, our SEO and AI-SEO course covers it hands-on. Call +91-75175-72000.
If you run a large site — a multi-city IT company in Delhi or a big content library — crawl and index management is often where the biggest, fastest SEO wins hide. New to the subject? Start with what SEO is, then let our web development team make sure the site is built to be crawled cleanly.
FAQ
What is the difference between crawling and indexing?
Crawling is Google discovering and downloading your page. Indexing is Google analysing and storing it. A page must be crawled first, then indexed, before it can rank in search results.
Why is my page crawled but not indexed?
Usually because Google judged the content thin, duplicate or low-value, or a canonical tag points elsewhere. Improve content quality, uniqueness and internal linking to help it get indexed.
How long does it take Google to index a page?
Anywhere from a few hours to several weeks. It depends on your site's authority, crawl frequency, internal linking and content quality. Submitting the URL in Search Console can speed it up.
How do I stop a page from being indexed?
Add a noindex meta tag or X-Robots-Tag header and keep the page crawlable so Google can read the directive. Do not block it in robots.txt, or the noindex may never be seen.
Does more crawl budget mean better rankings?
No. Crawl budget affects how many pages get discovered, not how they rank. It mainly matters for very large sites where important pages might otherwise be missed.
How do I check if my page is indexed?
Use the URL Inspection tool in Google Search Console, or search site:yourdomain.com/page-url in Google. Search Console gives the authoritative indexing status.
Pages not getting indexed? Cyber Defence diagnoses and fixes crawl and index issues as part of SEO and AI-SEO from ₹15,000/month, led by Amit Kumar (CEH, CRTA), Hisar and remote across India. Call +91-75175-72000.