Title: robots.txt Author: Kriko Published: Mar 24, 2021 Last modified: Jul 12, 2026 --- 1. [Home](https://kriko.io/) / 2. [Glossary](https://kriko.io/glossary) / 3. R Letter # robots.txt **Robots.txt** is a text file located in the root directory of a website that tells search engine bots which parts of the site they are allowed or not allowed to crawl. Also known as the robots exclusion protocol, it helps bots from search engines such as Google, Bing and others understand which URL paths they can access while crawling a website. Robots.txt is one of the basic technical SEO tools used to guide crawling behaviour. However, it does not guarantee that pages will be indexed or removed from the index. The main purpose of robots.txt is to limit or guide the crawling of specific areas within a website. For example, admin panels, filtered URLs, cart pages, internal search result pages, parameter-based URLs or low-value directories can be blocked from crawling. This can help search engines focus more on important pages. Robots. txt can be especially useful for large e-commerce websites, news websites and platforms with a high number of URLs, as it can support more efficient crawl budget management. Robots.txt is often one of the first files that search engine bots check when they visit a website. Commands such as `User-agent`, `Disallow` and `Allow` can be used to define which bots are allowed or blocked from accessing specific paths. For example, the `Disallow` directive can be used to prevent a specific directory from being crawled. The location of the sitemap can also be specified in robots.txt using the` Sitemap` line. Blocking crawling with robots.txt does not necessarily mean that the page will be removed from search results. If a URL is linked from other pages or was previously indexed, a search engine may still display that URL in search results with limited information. To prevent a page from appearing in search results, methods such as a `noindex` tag, an HTTP header directive or access control may be required. For this reason, robots.txt should not be confused with index management. Robots.txt should also not be treated as a security measure. The file is publicly accessible, and malicious bots may ignore the rules written in it. Therefore, private, confidential or sensitive content should not be protected only through robots.txt. Membership areas, admin panels, personal [data](https://kriko.io/services/data-analysis-insight) or confidential files should be protected with password systems, authorisation, server-side access controls and proper security measures. Preparing robots.txt correctly is important because an incorrect rule can prevent important pages from being crawled by search engines. For example, accidentally blocking the entire site with `Disallow: /` can severely restrict crawling. Similarly, blocking CSS and JavaScript files unnecessarily can make it difficult for search engines to render the page correctly. Robots.txt updates should therefore be tested carefully before they are published. The robots.txt file is usually located under the main domain in the format `domain. com/robots.txt`. It is a simple text file and can be prepared in UTF-8 format. If a website uses multiple subdomains, each subdomain may need its own robots.txt file. For example, if `www`, `blog` or `shop` subdomains are managed separately, each should have its own robots.txt file in the relevant root directory. A properly configured **robots.txt** file helps search engine bots crawl a website more efficiently. It can reduce crawling of unimportant or repetitive URLs, indicate the location of the sitemap and make technical SEO management easier. However, this file is not a complete solution for indexing, security or privacy. For a successful crawling and indexing strategy, robots.txt should be evaluated together with XML sitemaps, canonical tags, noindex directives, internal linking structure and broader technical SEO controls. ## Discover it in the dictionary ### Artificial Intelligence Artificial intelligence, commonly abbreviated as AI, is a field of technology that aims to enable computer systems to perform tasks associated with human intelligence,… ### Data Architecture Data architecture is the structure that defines how an organisation collects, stores, manages, integrates and uses data. It brings data systems, technologies, rules, standards… ### Domain Authority Domain Authority, commonly abbreviated as DA, is a third-party SEO metric developed by Moz. This metric attempts to predict how likely a website is… ### Largest Contentful Paint (LCP) Largest Contentful Paint, commonly abbreviated as LCP, is a Core Web Vitals metric that measures how long it takes for the largest visible content… ### Data Gravity Data gravity refers to the effect created when large volumes of data become concentrated within a particular system, platform or location. As data grows… ### Programmatic Advertising Programmatic advertising is a digital advertising model that enables ad inventory to be purchased through software, data and automation technologies. In traditional media buying,… ### Google Analytics Google Analytics is a Google analytics platform used to measure, analyse and report user interactions on websites and mobile applications. Its standard version can… ### Cookie A cookie is a small piece of data that a web server sends to a user's browser and that the browser can store. On… ### Push Notification A push notification is a short message that a mobile application, web application or website sends to a user through their device. These notifications… ## Track the digital heartbeat with Kriko Subscribe to receive curated insights, news, and ideas shaping the digital landscape. By checking this box, you acknowledge and accept our [privacy policy](https://kriko.io/privacy-policy).