Crawling depth determines how many links deep a scanner follows from your homepage before it stops – and for most sites, the default depth misses more than it finds. A blog with 40 posts and a support portal with 4,000 articles look identical from the homepage, but a scanner set to crawl three levels deep might index 200 URLs on the first site and 200 URLs on the second, leaving 3,800 pages completely untested on the support portal.
What crawling depth actually measures
Depth counts hops, not pages. The homepage is depth 0. A link from the homepage is depth 1. A link found on that depth-1 page is depth 2, and so on. A scanner configured for depth 3 will follow /products/ → /products/category-a/ → /products/category-a/item-42/ and then stop, regardless of what item-42’s page links to next.
This matters because most content management systems don’t put everything within three clicks of the homepage. WooCommerce stores routinely bury individual product variants five or six links deep once you account for category, subcategory, filter, and pagination layers. A knowledge base built on Zendesk or a custom Next.js app with client-side routing can hide entire sections behind a search box that a crawler never triggers, because there’s no static link to follow at all.
Why deep pages are often the riskiest ones
Security teams tend to assume the homepage and top navigation pages carry the most risk because they get the most traffic. That’s backwards for a lot of vulnerability classes.
Old admin tools, staging pages left in production, and orphaned upload forms are almost never linked from the homepage – that’s precisely why they survive unnoticed for years. A WordPress site migrated in 2021 might still have /wp-content/uploads/old-migration-tool.php sitting on the server, reachable only because someone bookmarked it or a search engine indexed it before the link was removed. Shallow crawls never touch it. A file upload endpoint from a discontinued plugin, buried four directories deep, is a more attractive target to an attacker running mass automated scans than the contact form on your homepage, precisely because nobody is watching it.
Common reasons scanners stop short
A handful of technical patterns account for most of the pages that never get tested:
JavaScript-rendered links – if navigation is built with client-side JS and the crawler doesn’t execute scripts, it sees an empty shell and stops there.
Pagination without direct links – infinite-scroll product listings or “load more” buttons that fire an AJAX call instead of exposing a page=2 URL.
Authentication walls – member dashboards, account settings, and admin panels that require a session cookie the crawler doesn’t have.
robots.txt disallow rules – sections deliberately blocked from search engines that a security scanner should still check for vulnerabilities, but often skips out of the same politeness rules as an SEO crawler.
Depth or time limits – a hard cap of 500 pages or a 10-minute crawl window, both of which favor breadth over the parts of the site that matter for security.
The myth that a sitemap.xml solves this
A common assumption is that submitting a sitemap.xml file guarantees full coverage. It doesn’t. A sitemap is a list the site owner chose to publish – usually generated by an SEO plugin like Yoast or RankMath – and it reflects what the site wants indexed for search, not what actually exists on the server.
Orphaned pages, old backup files, exposed .env or config remnants, and admin interfaces are excluded from sitemaps by definition; nobody adds /admin/debug.php to their sitemap.xml on purpose. Relying on the sitemap as a crawl seed means a scanner inherits the site owner’s blind spots instead of testing independently of them. Sitemaps are a useful starting point for discovering the intended content structure, but they should supplement crawling, not replace it.
How to check whether your own scans are hitting a depth wall
A practical way to test this is to count pages independently and compare. Pull the total URL count from Google Search Console’s Pages report, or run `site:yourdomain.com` on Google, then compare that number against how many URLs your security scan actually reports as tested. A gap of more than 15-20% usually means something structural is blocking the crawler – JS rendering, pagination, or a depth cutoff.
A seasoned security auditor also checks server access logs for 200-status requests to paths that never appear in the scan report. If the web server logs show traffic to /internal/reports/2024/ but the scanner never touched it, that’s a clear sign the crawl configuration, not the target site, is the limiting factor.
Practical steps to widen coverage
Increasing depth alone isn’t free – a crawl that goes 10 levels deep on a site with faceted search filters can spiral into tens of thousands of near-duplicate URLs (same products, different sort order) and waste scan time on noise instead of real coverage. The fix is usually structural, not just a bigger number:
Feed the scanner multiple entry points instead of relying on the homepage alone – submit category pages, the sitemap, and any known deep sections as separate seeds. For JavaScript-heavy sites, confirm the scanner executes rendered DOM content rather than parsing raw HTML; this is the single biggest source of missed pages on React, Vue, and Angular front ends. Where authentication-gated areas hold sensitive functionality, provide test credentials so the crawl can log in and cover account pages, since attackers who obtain leaked credentials will absolutely go there. This kind of layered coverage overlaps with how automated scans map onto OWASP’s vulnerability categories – breadth of testing only matters if the crawl actually reaches the pages where those categories apply.
Running scans on a recurring schedule also helps catch pages that get added after the initial crawl baseline, since site structure changes constantly as content teams publish new sections; see how often you should realistically be scanning for a sense of the right cadence.
Frequently asked questions
Does a deeper crawl mean a slower scan?
Yes, proportionally. Each additional depth level can multiply the number of URLs discovered, especially on e-commerce sites with filters and sort parameters. A scan that takes 12 minutes at depth 3 might take 45 minutes at depth 6 on a large catalog site. The trade-off is worth it when the alternative is leaving entire sections untested, but it’s worth tuning depth per site rather than using one setting everywhere.
Can a scanner find pages that aren’t linked from anywhere?
Only indirectly. True orphan pages – no internal links, not in the sitemap, blocked from search engines – require other discovery methods, like checking historical DNS records, certificate transparency logs for subdomains, or old backup archives. A crawl-based scanner finds what’s reachable by following links; it won’t magically discover a forgotten URL that exists in isolation.
Should robots.txt block a security scanner the same way it blocks Googlebot?
No. robots.txt is a convention for search engine indexing, not an access control mechanism, and it shouldn’t be applied to security testing. A path disallowed for SEO reasons, like an old staging directory, is often exactly the kind of forgotten area that needs testing most, since attackers ignore robots.txt entirely.
Depth is one of the least visible settings in any scanning setup, and it’s rarely mentioned in a scan report – which is exactly why it goes unchecked for years. Before trusting a clean scan result, confirm the crawler actually reached the parts of the site nobody thinks about anymore.