robots.txt

robots.txt uses a simple Disallow / Allow syntax scoped by user-agent. It controls crawling, not indexing — a URL blocked by robots.txt can still appear in the index if Google discovers it through external links.

Common mistakes

• Blocking CSS or JavaScript, which prevents Google from rendering the page correctly.
• Using robots.txt to hide sensitive URLs — the file itself is public.
• Applying Disallow when you meant noindex.

Example

A retailer blocks /search/ to prevent crawler waste on internal search-result pages, keeps /product/ open, and lists the sitemap at the bottom of the file.

How SEM Optimiser reports it

The audit fetches robots.txt on every crawl and warns when critical resources are blocked or when the file returns a non-200 response.