Title: robots.txt Author: Kriko Published: Mar 24, 2021 Last modified: Jul 12, 2026 --- 1. [Home](https://kriko.io/) / 2. [Glossary](https://kriko.io/glossary) / 3. R Letter # robots.txt **Robots.txt** is a text file located in the root directory of a website that tells search engine bots which parts of the site they are allowed or not allowed to crawl. Also known as the robots exclusion protocol, it helps bots from search engines such as Google, Bing and others understand which URL paths they can access while crawling a website. Robots.txt is one of the basic technical SEO tools used to guide crawling behaviour. However, it does not guarantee that pages will be indexed or removed from the index. The main purpose of robots.txt is to limit or guide the crawling of specific areas within a website. For example, admin panels, filtered URLs, cart pages, internal search result pages, parameter-based URLs or low-value directories can be blocked from crawling. This can help search engines focus more on important pages. Robots. txt can be especially useful for large e-commerce websites, news websites and platforms with a high number of URLs, as it can support more efficient crawl budget management. Robots.txt is often one of the first files that search engine bots check when they visit a website. Commands such as `User-agent`, `Disallow` and `Allow` can be used to define which bots are allowed or blocked from accessing specific paths. For example, the `Disallow` directive can be used to prevent a specific directory from being crawled. The location of the sitemap can also be specified in robots.txt using the` Sitemap` line. Blocking crawling with robots.txt does not necessarily mean that the page will be removed from search results. If a URL is linked from other pages or was previously indexed, a search engine may still display that URL in search results with limited information. To prevent a page from appearing in search results, methods such as a `noindex` tag, an HTTP header directive or access control may be required. For this reason, robots.txt should not be confused with index management. Robots.txt should also not be treated as a security measure. The file is publicly accessible, and malicious bots may ignore the rules written in it. Therefore, private, confidential or sensitive content should not be protected only through robots.txt. Membership areas, admin panels, personal data or confidential files should be protected with password systems, authorisation, server-side access controls and proper security measures. Preparing robots.txt correctly is important because an incorrect rule can prevent important pages from being crawled by search engines. For example, accidentally blocking the entire site with `Disallow: /` can severely restrict crawling. Similarly, blocking CSS and JavaScript files unnecessarily can make it difficult for search engines to render the page correctly. Robots.txt updates should therefore be tested carefully before they are published. The robots.txt file is usually located under the main domain in the format `domain. com/robots.txt`. It is a simple text file and can be prepared in UTF-8 format. If a website uses multiple subdomains, each subdomain may need its own robots.txt file. For example, if `www`, `blog` or `shop` subdomains are managed separately, each should have its own robots.txt file in the relevant root directory. A properly configured **robots.txt** file helps search engine bots crawl a website more efficiently. It can reduce crawling of unimportant or repetitive URLs, indicate the location of the sitemap and make technical SEO management easier. However, this file is not a complete solution for indexing, security or privacy. For a successful crawling and indexing strategy, robots.txt should be evaluated together with XML sitemaps, canonical tags, noindex directives, internal linking structure and broader technical SEO controls. ## Discover it in the dictionary ### Internal Linking Internal linking refers to links that connect one page of a website to another page under the same domain. In other words, an internal… ### Time Series Analysis Time series analysis is an analytical method used to examine data recorded over time. A time series consists of values related to a specific… ### Google Index Google Index is the index system that contains web pages Google has discovered, crawled, processed and evaluated for potential display in search results. For… ### Conversion Rate Conversion Rate, commonly abbreviated as CR, is a performance metric that shows the percentage of users who complete a target action on a website,… ### Pagination Pagination is a web design and development method used to divide long content, product listings, comments or data sets into multiple pages. This structure… ### Rich Snippet A rich snippet is an enhanced search result that displays additional information alongside the standard title, URL and description. To understand this concept clearly,… ### Application Programming Interface (API) An Application Programming Interface, commonly abbreviated as API, is a software interface that allows controlled access to specific functions of an application, service, operating… ### Data Security Data security refers to the protection of data against risks such as unauthorised access, use, modification, disclosure, loss or destruction. Its primary objective is… ### Geospatial Analysis Geospatial analysis is a method used to examine data through location, distance, area and geographical relationships. It helps identify where events occur, how different… ## Track the digital heartbeat with Kriko Subscribe to receive curated insights, news, and ideas shaping the digital landscape. By checking this box, you acknowledge and accept our [privacy policy](https://kriko.io/privacy-policy).