What is robots.txt?
Robots.txt is a plain text file placed at the root of a website that tells search engine crawlers which URLs or folders they should not crawl. It guides well-behaved bots only and is not a security measure, since anyone can still open the blocked pages directly.
A robots.txt file is a short text file saved at the top level of a domain, such as example.com/robots.txt. Crawlers from search engines like Google and Bing check it before they fetch pages from the site. Each rule names a user agent and lists paths to allow or disallow, for example blocking a staging folder, a cart page, or internal search result URLs that add nothing to search.
The file matters because it shapes how crawl budget is spent. On a large Malaysian e-commerce or property site with thousands of filter combinations, a clean robots.txt keeps crawlers focused on the product and category pages you want indexed. It can also point crawlers to an XML sitemap, which helps them find those pages faster.
Robots.txt has a limit that people often miss. It is a request, not a lock. Compliant bots honour it, but nothing stops a person or a rogue scraper from visiting a disallowed URL. A page blocked in robots.txt can also still appear in search results if other sites link to it, which is why noindex tags or password protection are the right tools when content must stay private.
Errors in this file can be costly. A single stray "Disallow: /" can hide an entire site from search engines. Checking the file in Google Search Console after changes is a simple way to catch mistakes early.
If you want help reviewing crawl rules or site structure, the technical SEO providers in the directory handle this kind of work.
"