March 27, 2013

How to Enable Custom Robots.txt File in Blogger

Robot.txt is a way to tell search engines whether they are allow to index a page in the search result or not. The bots are automatic, and before they could access your site, they check the Robot.txt file to make sure whether they are allowed to crawl this page or not.   Sometime, people do not want to stop searching engine from crawling their whole website. On the other hand, they want to specify few pages, which should not be indexed in the search results. Therefore, today in this article, we will show you How to Enable Custom Robot.txt File in Blogger?

First of all, go to Blogger.com >> your site >> Settings >> Search engine Preference >> Crawlers and indexing. Now you will be able to see two options i.e. Custom robots.txt and Custom robots header tags. These two options would provide you the flexibility to customize your robot.txt file. 
  • Custom robots.txt: This option provides you the ability to edit your whole robot.txt file. You can just type your content whether you want the content to be crawled by spiders or not. However, You can always undo your actions and can always revert back to normal. 
  • Custom robots header tags: This option is a bit completed. It does not provide you the ability to write your codes instead it provides few options with the check boxes. So, if have no idea about head tags then stay away from it. 
Now you have to enable the custom robot.txt file so press the edit button which is present next to the “Custom robots.txt” option. After selecting the edit button, it would ask you to enable custom robots.txt content so press “Yes” and proceed to the next step. 
In the text area, type the content which you want to exclude from being crawled. After editing the file according to your needs, press the save button to conclude. However, if you want to revert back to the default robot file then, you can select “No” instead of “Yes. 
For Example:

User-agent: *
Disallow: /about

User-agent: *
Disallow: /contact

User-agent: *
Disallow: /services

We hope this tip would help you. If you are having any issue regarding crawling then, feel free to leave your questions in the comments below and our experts would try to help you in solving them. 

55 comments:

  1. Life blogMarch 27, 2013 at 4:20 PM

    Thanks Syed Bro well explanations. We will just only write the name of page like About me.
    or
    We have to write full url of that page and put it here. How?

    ReplyDelete
    Replies
    1. Syed Faizan AliMarch 27, 2013 at 11:23 PM

      No, you have to write the URL. For example, if your want to stop robots from crawling this URL (www.mybloggerlab.com/p/about.html) then, in the robot.txt file you will enter this command.

      User-agent: *
      Disallow: p/about.html

      Exclude the main domain and just type the remaining part of it.

      Delete
    2. UnknownMarch 28, 2013 at 11:31 AM

      You have to include this as disallow condition.

      Disallow: /p/about.html

      Delete
  2. R S DasMarch 27, 2013 at 11:23 PM

    pls explain the examples mentioned in detail

    ReplyDelete
    Replies
    1. Syed Faizan AliMarch 27, 2013 at 11:30 PM

      We will create a separate article to explain the commands which we use in robot.txt file.

      Delete
  3. UnknownMarch 28, 2013 at 12:31 AM

    Hi,previous when i publish a post i could see it in google search results in a few minutes,it was gettting indexed automatically .But now donno what happened,i dont see my published post in google even after a day,and if I manually submit my blog post to google ,it is getting indexed.Pease let me know what would the problem.

    my blog: genuinecontent.blogspot.com

    ReplyDelete
    Replies
    1. Syed Faizan AliMarch 28, 2013 at 1:21 AM

      It is not only you who is facing this issue. Almost the majority of BlogSpot blogs are getting index lately. So don't get dishearten and hope of the best.

      Delete
  4. Life blogMarch 28, 2013 at 1:32 PM

    Syed i have insert my robot.txt file in blogger dashboard in crawling section like this is it OK.

    User-agent: Mediapartners-Google
    Disallow:

    User-agent: *
    Disallow: /search
    Allow: /

    Sitemap: http://www.probloggertricks.com/feeds/posts/default?orderby=UPDATED

    ReplyDelete
    Replies
    1. Syed Faizan AliMarch 28, 2013 at 2:22 PM

      Yup, its fine

      Delete
    2. UnknownMarch 28, 2013 at 2:38 PM

      Yes, This is default robots.txt file. If you keep disable custom file, then the same file will work for blogger. You can verify your default robots.txt file using:
      www.yourblog.com/robots.txt

      Delete
    3. Jawad SaqibFebruary 13, 2016 at 8:59 PM

      use this sitemap in your blog robots.txtfile like www.yourblogaddress.com/sitemap.xml it is best for seo...

      Delete
  5. UnknownApril 23, 2013 at 8:51 PM

    I'm not much know about robots.text . If it's possible please tell me what should become robots.text for my blog and how I wrote it my blog : http://seo-supports.blogspot.com

    ReplyDelete
    Replies
    1. Syed Faizan AliApril 24, 2013 at 1:29 AM

      You are using Default Robots.txt file Its fine there is no need to edit. Cheers

      Delete
  6. Syed Faizan AliMay 11, 2013 at 12:49 AM

    You should add Description specially while publishing a post. Its better to it manually.

    ReplyDelete
  7. Syed Faizan AliMay 11, 2013 at 12:56 AM

    This is a Default Robot file so its just fine.

    ReplyDelete
    Replies
    1. Midhat ImranOctober 15, 2014 at 11:23 PM

      User-agent: Mediapartners-Google
      Disallow:

      User-agent: *

      Disallow: /search
      Allow: /yahoo.com


      Sitemap: http://fun-with-fun.blogspot.com/feeds/posts/default?orderby=UPDATED

      Delete
    2. Midhat ImranOctober 15, 2014 at 11:23 PM

      is this ri8?

      Delete
    3. Syed Faizan AliOctober 17, 2014 at 3:12 AM

      Yes Diya

      Delete
  8. UnknownMay 28, 2013 at 2:03 AM

    why should we add robot txt ?

    ReplyDelete
    Replies
    1. Syed Faizan AliMay 29, 2013 at 1:18 AM

      Through it, you can control the crawling of your site. You can let search engine index few but not index everything. its like a key to your site for crawllers

      Delete
  9. Greg_1948 in_WAJune 3, 2013 at 11:55 AM

    I had a robot.txt file, but I did not like how it worked, so I DISABLED it to get back to normal. How long will it take to get back to the default condition as far as operation is concerned? It is not immediate.

    ReplyDelete
    Replies
    1. Syed Faizan AliJune 3, 2013 at 11:39 PM

      On Disabling the Robot.txt file it gets back to normal within seconds. Cheers

      Delete
  10. Syed Faizan AliJune 4, 2013 at 4:43 PM

    Your Robot file is default. and I don't see any problems from here. I also Google'd your site and every thing seems to work fine at my end. Cheers

    ReplyDelete
  11. UnknownJune 9, 2013 at 12:19 PM

    Thanks

    ReplyDelete
  12. Kang NoeJune 12, 2013 at 5:41 AM

    nice info, thanks

    ReplyDelete
  13. Ejaz KhanJuly 3, 2013 at 12:20 AM

    Thanks very much, much helpful

    ReplyDelete
  14. New Beauty StoreAugust 26, 2013 at 3:34 AM

    please tell me what should be the code if i want all my pages and posts being crawled by search engines ...
    is this code is ok
    User-agent: Mediapartners-Google
    Disallow:
    User-agent: *
    Disallow: /search?q=*
    Disallow: /*?updated-max=*

    should i also add the sitemap?
    Allow: /


    ReplyDelete
  15. New Beauty StoreAugust 26, 2013 at 3:46 AM

    Adding to my previous comment pl tell is this code is alright,
    User-agent: Mediapartners-Google
    Allow:/
    User-agent: Googlebot
    Disallow: /?m=1
    Disallow: /?m=0
    Disallow: /*?m=1
    Disallow: /*?m=0
    User-agent: *
    Disallow: /search?*
    User-agent: Twitterbot
    Allow:/
    User-agent: *
    Disallow: /search
    Disallow: /p/*
    Disallow: /view/*
    Allow: /
    Sitemap: http://aboutwomenhealthandbeauty.blogspot.com/feeds/post/default?orderby=UPDATED

    it would be grateful to you, if you answer my query,
    Regards,

    ReplyDelete
    Replies
    1. Syed Faizan AliAugust 29, 2013 at 4:36 PM

      There no need to do anything, Just use default robot.txt file as provided by blogger itself

      Delete
    2. Syed Faizan AliSeptember 8, 2013 at 11:29 AM

      Here you go

      User-agent: Mediapartners-Google
      Disallow:

      User-agent: *
      Disallow: /search
      Allow: /

      Sitemap: http://www.mybloggerlab.com/feeds/posts/default?orderby=UPDATED

      Replace: MyBloggerlab.com with your domain.

      Delete
  16. New Beauty StoreAugust 29, 2013 at 11:28 PM

    Thanks for your answer but I found no default setting there, robot.txt file is disabled by default. What I am supposed to do in search preferences then?

    ReplyDelete
  17. VideosOctober 29, 2013 at 8:11 AM

    Sir, this is harsha. My web page is not displayed in search engine and also it is not opening. In some systems our website is opening which do not have the security for systems and the systems which have avast security then it is not opening. could you please help me sir.....

    ReplyDelete
  18. UnknownNovember 4, 2013 at 12:17 AM

    Hello. I'm using blogspot as my platform. I want to disallow three pages from search engine crawlers. Can you show sample I can follow in the Customs Robot.txt box?. Appreciate your response. Thank you.

    ReplyDelete
    Replies
    1. Syed Faizan AliNovember 4, 2013 at 1:03 AM

      Here's the example

      User-agent: *
      Disallow: /search
      Disallow: /2013/03/how-to-remove.html
      Allow: /

      Delete
  19. UnknownNovember 6, 2013 at 1:21 AM

    Hey MBL,
    Just wanted to ask, how long does it take for a custom robots.txt file to show up/start working?
    I applied one on my blog recently but when i navigate to mysite.com/robots.txt it still shows the default robots.txt content and not the one i added.

    So wanted to know how long does it take for a custom file to come in effect.

    Thanks

    ReplyDelete
    Replies
    1. Syed Faizan AliNovember 6, 2013 at 5:11 AM

      It takes seconds to implement. Check whether you're adding it correctly or not.

      Delete
  20. UnknownDecember 12, 2013 at 2:19 AM

    sir,
    since i updated me robots.txt file on my blog i lost traffic from google...plz help ...
    my blog traffic has drop down to 10% as it was before...what to do....plz help...is there any other reasion ..plz let me know..or plz tell the errors in my robots.txt file..

    my blog - www.supper-android.blogspot.com/

    ReplyDelete
    Replies
    1. Syed Faizan AliDecember 12, 2013 at 9:02 PM

      This is your Robots.txt File

      User-agent: Mediapartners-Google
      Disallow:

      User-agent: *
      Disallow:
      Allow: /

      Sitemap: http://www.supper-android.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
      Sitemap: http://www.supper-android.blogspot.com/feeds/posts/default?orderby=UPDATED

      Replace the Disallow with this

      Disallow: /search

      And Then, It might help you out.

      Delete
  21. AaliJaahFebruary 15, 2014 at 5:26 AM

    how long it takes time to effect?

    ReplyDelete
  22. NuevaJanuary 15, 2015 at 5:37 PM

    Hello, I am having problems with Twitter Cards and Blogger. My blog is www.bondadanonima.com.
    Twitter's Troubleshooting page suggests: Configure your web servers to allow Twitter access via the robots.txt file.
    witter uses the User-Agent of Twitterbot (with version, such as Twitterbot/1.0), which can be used to create an exception in your robots.txt file.

    User-agent: Twitterbot
    Disallow:

    How/where do I do this?

    Many thanks in advance

    ReplyDelete
  23. UnknownJune 20, 2015 at 10:40 PM

    Sir Create A blog And Set Up A domain After This I am Apply for Adsense They Are ask me for This Error ( http://www.selectomobile.tk/robots.txt?m=1#150 ) How I can solve this???Please helpme

    ReplyDelete
  24. UnknownJuly 20, 2015 at 2:01 PM

    will this affect traffic ?

    ReplyDelete
  25. Khawer KhanDecember 10, 2015 at 10:17 PM

    Like wordpress i also want robot.txt to allow blogger labels to appear in search result. How to allow label and suppose if i do this the result will be better or not.

    ReplyDelete
    Replies
    1. Syed Faizan AliDecember 18, 2015 at 3:48 AM

      If you want your labels to be indexed, replace your robots.txt file with this

      User-agent: Mediapartners-Google
      Disallow:

      User-agent: *
      Disallow: /search
      Allow: /search/label
      Allow: /


      Sitemap: http://www.yoursite.com/sitemap.xml

      Delete
    2. Syed Faizan AliDecember 18, 2015 at 3:49 AM

      Its to mention that indexing Labels won't help in rankings.

      Delete
  26. UnknownDecember 27, 2015 at 10:06 AM

    hey thanks bussy for such information, ihave a blog on blogspot ( http://whatsappstatus-mystatus.blogspot.in/ )
    nd my robot.text is
    # Blogger Sitemap generated on 2015.12.16
    User-agent: *
    Disallow: /search
    Allow: /
    Sitemap: http://whatsappstatus-mystatus.blogspot.in/atom.xml?redirect=false&start-index=1&max-results=500
    but when im searching my blog in google it always shows
    A description for this result is not available because of this site's robots.txt.
    for all my labels.. i dont know what is getting wrong could u help me figure out this problem.
    thanks aarav

    ReplyDelete
    Replies
    1. Syed Faizan AliDecember 28, 2015 at 4:13 AM

      Labels doesn't provide any description in the search, this is how Blogger site works. You can't do anything about it.

      Delete
    2. UnknownDecember 30, 2015 at 8:24 AM

      thanks for reply on my post.
      my secound question is this
      how could i get better rank in google?
      after submiting my blog in google webmasters im not getting any traffic from google . its 1 month now . what is the problem ?

      Delete
    3. Syed Faizan AliDecember 30, 2015 at 11:30 AM

      Webmaster tool just helps you index your pages in search, but to get traffic from Google you need to rank your website on keywords that are searched by people.

      1. You need backlinks to get better ranking.
      2. Better Social Score.
      3. Great Content.

      Read this post to know about different ways to get traffic:

      http://www.mybloggerlab.com/2012/06/how-to-bring-real-targeted-usa-visitors.html

      http://www.mybloggerlab.com/2012/02/5-fantastic-tips-to-increase-website.html

      Delete
  27. Jawad SaqibFebruary 13, 2016 at 9:08 PM

    i have two robots.txt file which one is correct

    User-agent: Mediapartners-Google
    Disallow:
    User-agent: *
    Disallow: /search
    Disallow: /search?q=*
    Disallow: /*?updated-max=*
    Allow: /

    Sitemap: http://lifehacklife.blogspot.com/sitemap.xml


    2:

    User-agent: Mediapartners-Google
    Disallow:
    User-agent: *
    Disallow: /search
    Disallow: /search?q=*
    Disallow: /*?updated-max=*
    Allow: /

    Sitemap: http://lifehacklife.blogspot.com/feeds/posts/default

    ReplyDelete
    Replies
    1. Syed Faizan AliFebruary 14, 2016 at 10:27 AM

      Both looks fine, You have just changed the sitemap link. It depends on you which sitemap link you choose.

      Cheers

      Delete

We’re eager to see your comment. However, Please Keep in mind that all comments are moderated manually by our human reviewers according to our comment policy, and all the links are nofollow. Using Keywords in the name field area is forbidden. Let’s enjoy a personal and evocative conversation.