Robots.txt

A text file placed at the root of a website that instructs search engine crawlers which pages or directories they should or should not index. Proper robots.txt configuration ensures that search engines focus their crawl budget on the most important B2B content pages.