May 11, 2018

How to add Custom Robots.txt file?

For gaining organic traffic to a blog/website, it is necessary that we design the same to be SEO ready. One efficient step in building SEO is adding Robots.txt file. It tells the search engines crawlers about what page to crawl and what not to.

Overview:




How to add Custom Robots.txt file?
How to add Custom Robots.txt file?

Before moving straight into How to add Custom Robots txt file? let's first have a clear idea about Robots TXT file. In this article, I have tried to explain about what a Robots.txt file is, why is it important for blog's SEO and how to create one.

Best Deals

What is Robots.txt file?

Robots.txt is a text format file that directs search engine crawlers, which page to crawl and the one's not to. Search engine crawlers always  scan this file before crawling any blog.

How to add Custom Robots.txt file to your blog?

Follow the steps below carefully as if you mess with this part, your site might be deindexed:

  1. Sign in to your blog.
  2. Navigate to Settings>>Search Preferences.
  3. Under Crawlers and Indexing>>Custom Robots.txt>>Click [Edit].
  4. Now paste from below robots.txt file code in the box.


User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://letslearnblogging.blogspot.com/feeds/posts/default?orderby=UPDATED


Remember to change letslearnblogging to your blog URL.


Click on Save Changes button.


Example of Custom Robots.txt
Example of Custom Robots.txt

You've successfully added custom Robots.txt file for your blog.

Check your live Robots.txt file

If you want to check if Robots file you uploaded was fine properly or not? then go to the address bar of your browser and type your blog URL and add /robots.txt in the end. Press enter.
It will appear like this:


All set! I hope you could easily add the custom robots.txt file in your blog. If you have any query related to this, drop a comment below.