Why is my page not being crawled?

If a page you expect to see in your audit is missing, one of seven things is happening. Work through them in this order – they're ranked by how often we see each cause in real audits.

1. robots.txt blocks our crawler

The fastest check: visit https://yoursite.com/robots.txt and look for any line that disallows our user agent or all bots. Our default user agent is SEOAuditBot/1.0. Common patterns that block us:

text
User-agent: *
Disallow: /

User-agent: SEOAuditBot
Disallow: /admin/

If you intentionally block bots, allow our specific user agent or temporarily switch off "Respect robots.txt" in Settings → Crawl while the audit runs.

2. The page requires authentication

Members-only pages, admin areas and staging environments behind HTTP basic auth all look like 401/403 errors to our crawler. Add credentials in Settings → Crawl → Authentication (basic auth username/password or session cookie) and the audit will rerun against the authenticated origin.

3. The page is not linked from anywhere we crawled

Our crawler discovers URLs by following links. If a page isn't linked from any other page within your crawl depth, we won't find it. Two fixes:

• Add a link from your sitemap.xml or robots.txt – we follow both.
• Submit the URL list directly under Audits → New audit → Advanced → Seed URLs.

4. Your CDN is rate-limiting us

Cloudflare, Akamai and similar CDNs sometimes flag our requests as suspicious and start returning 429s or CAPTCHAs. Symptoms: the audit completes but shows hundreds of HTTP 403/429 errors. The fix is to allow-list our IP range or our user agent. Contact support and we'll send you the current IP block.

5. The page returns a non-2xx status

A page that returns 404, 410, 500 or 503 won't end up in your audit even if we find it. Check the audit's "Issues" tab – non-200 pages are listed there separately, not in the main URL list. If you expected a 200, the page is genuinely broken.

6. The page has a noindex meta or X-Robots-Tag

We crawl pages with noindex, but they appear in a separate "Noindex" report rather than the main URL list because they're intentionally hidden from search engines. Check the Issues → Noindex pages section.

7. JavaScript hides the link from our renderer

Some Single Page Apps add internal links only after a user action (e.g. opening a menu). We render pages with a headless browser, but we don't click around. The fix is to make sure every internal link is in the initial HTML or rendered DOM, ideally before any user interaction. SSR or SSG frameworks like Next.js, Nuxt and Astro do this automatically.

Diagnostic checklist

1. Open the missing URL in an incognito window – does it load with a 200?
2. Check robots.txt for blocks.
3. Search for the URL in the audit's Issues tab to see if it appeared with an error.
4. Make sure at least one other crawled page links to it.
5. If the page is behind auth, add credentials in Settings → Crawl.
6. If your site uses Cloudflare/Akamai, check for 403/429 errors and allow-list us.