What Is Robots.txt?
Robots.txt is a plain-text file that gives search engine crawlers instructions about which website URLs they may request. The file sits at the website’s root address, such as example.com/robots.txt. It can guide crawling behaviour, but it does not remove pages from search results, protect private information, or replace page-level indexing controls.
Quick Facts About Robots.txt
Category
Technical SEO file
Used for
Crawler access instructions
Common confusion
It does not secure confidential content
Also called
Robots exclusion file, Robots exclusion protocol
Often discussed with
Technical SEO For Law Firms, Law Firm SEO Audits
Key Takeaways About Robots.txt
- Robots.txt controls crawler requests, not human visitor access.
- A blocked URL can still appear in search results.
- Robots.txt is not the same as a noindex tag.
How Robots.txt Works
Robots.txt contains rules for named crawlers, or for crawlers generally. A rule can allow or disallow requests for a path, folder, or file pattern. Search engines read the file before crawling, then apply the instructions they support.
Related glossary terms: Crawl budget, Crawl error, Noindex tag.
Website owners often use the file to reduce crawling of duplicate, temporary, or low-value URLs. A blocked page may still be discovered through links or other signals, so Robots.txt does not guarantee removal from search results. A noindex tag controls indexing after a crawler can access the page. The file also does not restrict people, stop data collection, or replace access controls. A sitemap serves a different purpose by listing URLs that a site wants search engines to discover.
Expert Note
When we audit Robots.txt, we check the file’s location, syntax, crawler rules, and relationship with important legal service pages. We compare blocked paths with the site’s sitemap and indexing controls. Our team also checks whether a rule unintentionally restricts assets that search engines need to understand a page.
Robots.txt in Practice: A Real-World Example
A Melbourne family law firm may use Robots.txt to prevent crawlers from requesting internal search results and staging URLs. The production pages for family law services remain available for crawling, while confidential client material stays protected through proper access controls.
Related Services
Related Terms
Crawl budget
Crawl budget is the attention and processing capacity a search engine assigns to crawling a website…
Crawl error
A crawl error is a problem that prevents a search engine crawler from accessing, reading, or…
Noindex tag
A Noindex tag is an instruction in a webpage’s HTML or HTTP response that asks search…
Sitemap
A Sitemap is a structured file or page that lists URLs a website wants search engines…
SEO-for-Lawyers.com.au
Have Questions About Robots.txt?
Contact SEO-for-Lawyers.com.au for practical guidance on Robots.txt and related seo for lawyers work in Melbourne.