Crawlers and technical
How does robots.txt work?
robots.txt works by publishing crawl directives that compliant automated agents read before fetching parts of a website.
Rules can address named user-agents and paths. They govern requested crawling behaviour, not whether a human can open a page.
The file is not an access-security mechanism or a reliable removal tool. Blocking crawling does not necessarily remove an already known address from an index.
In Seenford
Seenford uses robots.txt as technical context, while its answer verdicts remain linked to the exact sentences engines wrote.
Example
A rule allows a search crawler to fetch public service pages.
Related terms
Updated
See what AI says about your brand →