Using robots.txt on your website gives you the control to tell search engines which pages to crawl for indexing and which to skip. It is a plain text file which is located in a website’s root.

If you are using a blogging platform like Blogger or Tumblr to blog, you do not have a robots.txt in general. But wait, you can now add robots.txt to Blogger or blogspot blogs with ease, however you need to switch to new Blogger Interface in case if you’re using the old Blogger. This short tutorial shows how to add robots.txt to blogger blogs:

Add robots.txt to Blogger blogs: Step-by-step guide

1. Sign into your Blogger account and click the drop-down triangle as shown in the below screenshot. Then select Settings.

2. Now, under settings, click the sub-option Search Preferences. You’ll see the option Custom robots.txt option therein, set it to “Yes” if you already have a custom robots.txt file for your Blogger blog.

If you have want to write a custom robots.txt file for your blog, you should go through this guide to know robots.txt basics.

3. You don’t need to worry if you don’t know how to get a custom robots.txt file for your blogger blog, just leave the above option disabled (i.e. leave it set to “No”). Now, come to the next option i.e. Custom robots header tags and enable it by setting it to “Yes”.

You can select the indexing, following, archiving, availability and other robots parameters for the default posts and pages, home page, archive pages and search pages (/search/lablel/...) as per your need. If you’re confused, let me simplify what you’re should do here: Set your archive and search pages to noindex and then click the “Save Changes” button. This will set an automatic robots.txt file to your blogger blog according to the different parameters you’ve just chosen. This is the recommended method to add robots.txt to your blogger blog. Using this option will keep your Blogger blog’s search labels away from indexing on search engines and hence avoid the content duplication problem for your blog.

You can locate your robots.txt file by going to this path: http://YOUR_BLOGGER_BLOG_URL/robots.txt/. Hope you’ve utilized this guide to add robots.txt to your Blogger blog. Also add Breadcrumbs to Blogger for an improved navigation.