How to Enable Custom Robots.txt File in Blogger

Advertisements
Robot.txt is a way to tell search engines whether they are allow to index a page in the search result or not. The bots are automatic, and before they could access your site, they check the Robot.txt file to make sure whether they are allowed to crawl this page or not.   Sometime, people do not want to stop searching engine from crawling their whole website. On the other hand, they want to specify few pages, which should not be indexed in the search results. Therefore, today in this article, we will show you How to Enable Custom Robot.txt File in Blogger?

First of all, go to Blogger.com >> your site >> Settings >> Search engine Preference >> Crawlers and indexing. Now you will be able to see two options i.e. Custom robots.txt and Custom robots header tags. These two options would provide you the flexibility to customize your robot.txt file. 
  • Custom robots.txt: This option provides you the ability to edit your whole robot.txt file. You can just type your content whether you want the content to be crawled by spiders or not. However, You can always undo your actions and can always revert back to normal. 
  • Custom robots header tags: This option is a bit completed. It does not provide you the ability to write your codes instead it provides few options with the check boxes. So, if have no idea about head tags then stay away from it. 
Now you have to enable the custom robot.txt file so press the edit button which is present next to the “Custom robots.txt” option. After selecting the edit button, it would ask you to enable custom robots.txt content so press “Yes” and proceed to the next step. 
In the text area, type the content which you want to exclude from being crawled. After editing the file according to your needs, press the save button to conclude. However, if you want to revert back to the default robot file then, you can select “No” instead of “Yes. 
For Example:

User-agent: *
Disallow: /about

User-agent: *
Disallow: /contact

User-agent: *
Disallow: /services

We hope this tip would help you. If you are having any issue regarding crawling then, feel free to leave your questions in the comments below and our experts would try to help you in solving them. 
Advertisements

The Editorial Team of MyBloggerLab consists of a group of Professional Blogger geeks Led by Syed Faizan Ali (Founder of MyBloggerLab).

Comment With:
OR
Choose Wisely!

55 comments

March 27, 2013 at 4:20 PM

Thanks Syed Bro well explanations. We will just only write the name of page like About me.
or
We have to write full url of that page and put it here. How?

Editorial Team MOD
March 27, 2013 at 11:23 PM

No, you have to write the URL. For example, if your want to stop robots from crawling this URL (www.mybloggerlab.com/p/about.html) then, in the robot.txt file you will enter this command.

User-agent: *
Disallow: p/about.html

Exclude the main domain and just type the remaining part of it.

March 27, 2013 at 11:23 PM

pls explain the examples mentioned in detail

Editorial Team MOD
March 27, 2013 at 11:30 PM

We will create a separate article to explain the commands which we use in robot.txt file.

March 28, 2013 at 12:31 AM

Hi,previous when i publish a post i could see it in google search results in a few minutes,it was gettting indexed automatically .But now donno what happened,i dont see my published post in google even after a day,and if I manually submit my blog post to google ,it is getting indexed.Pease let me know what would the problem.

my blog: genuinecontent.blogspot.com

Editorial Team MOD
March 28, 2013 at 1:21 AM

It is not only you who is facing this issue. Almost the majority of BlogSpot blogs are getting index lately. So don't get dishearten and hope of the best.

March 28, 2013 at 11:31 AM

You have to include this as disallow condition.

Disallow: /p/about.html

March 28, 2013 at 1:32 PM

Syed i have insert my robot.txt file in blogger dashboard in crawling section like this is it OK.

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://www.probloggertricks.com/feeds/posts/default?orderby=UPDATED

Editorial Team MOD
March 28, 2013 at 2:22 PM

Yup, its fine

March 28, 2013 at 2:38 PM

Yes, This is default robots.txt file. If you keep disable custom file, then the same file will work for blogger. You can verify your default robots.txt file using:
www.yourblog.com/robots.txt

April 23, 2013 at 8:51 PM

I'm not much know about robots.text . If it's possible please tell me what should become robots.text for my blog and how I wrote it my blog : http://seo-supports.blogspot.com

Editorial Team MOD
April 24, 2013 at 1:29 AM

You are using Default Robots.txt file Its fine there is no need to edit. Cheers

Editorial Team MOD
May 11, 2013 at 12:49 AM

You should add Description specially while publishing a post. Its better to it manually.

Editorial Team MOD
May 11, 2013 at 12:56 AM

This is a Default Robot file so its just fine.

May 28, 2013 at 2:03 AM

why should we add robot txt ?

Editorial Team MOD
May 29, 2013 at 1:18 AM

Through it, you can control the crawling of your site. You can let search engine index few but not index everything. its like a key to your site for crawllers

June 3, 2013 at 11:55 AM

I had a robot.txt file, but I did not like how it worked, so I DISABLED it to get back to normal. How long will it take to get back to the default condition as far as operation is concerned? It is not immediate.

Editorial Team MOD
June 3, 2013 at 11:39 PM

On Disabling the Robot.txt file it gets back to normal within seconds. Cheers

Editorial Team MOD
June 4, 2013 at 4:43 PM

Your Robot file is default. and I don't see any problems from here. I also Google'd your site and every thing seems to work fine at my end. Cheers

June 9, 2013 at 12:19 PM

Thanks

June 12, 2013 at 5:41 AM

nice info, thanks

July 3, 2013 at 12:20 AM

Thanks very much, much helpful

August 26, 2013 at 3:34 AM

please tell me what should be the code if i want all my pages and posts being crawled by search engines ...
is this code is ok
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search?q=*
Disallow: /*?updated-max=*

should i also add the sitemap?
Allow: /


August 26, 2013 at 3:46 AM

Adding to my previous comment pl tell is this code is alright,
User-agent: Mediapartners-Google
Allow:/
User-agent: Googlebot
Disallow: /?m=1
Disallow: /?m=0
Disallow: /*?m=1
Disallow: /*?m=0
User-agent: *
Disallow: /search?*
User-agent: Twitterbot
Allow:/
User-agent: *
Disallow: /search
Disallow: /p/*
Disallow: /view/*
Allow: /
Sitemap: http://aboutwomenhealthandbeauty.blogspot.com/feeds/post/default?orderby=UPDATED

it would be grateful to you, if you answer my query,
Regards,

Editorial Team MOD
August 29, 2013 at 4:36 PM

There no need to do anything, Just use default robot.txt file as provided by blogger itself

August 29, 2013 at 11:28 PM

Thanks for your answer but I found no default setting there, robot.txt file is disabled by default. What I am supposed to do in search preferences then?

Editorial Team MOD
September 8, 2013 at 11:29 AM

Here you go

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://www.mybloggerlab.com/feeds/posts/default?orderby=UPDATED

Replace: MyBloggerlab.com with your domain.

Editorial Team MOD
September 8, 2013 at 11:30 AM

Check the above comment!

October 29, 2013 at 8:11 AM

Sir, this is harsha. My web page is not displayed in search engine and also it is not opening. In some systems our website is opening which do not have the security for systems and the systems which have avast security then it is not opening. could you please help me sir.....

Editorial Team MOD
November 3, 2013 at 4:22 AM

Can you provide your blog URL?

November 4, 2013 at 12:17 AM

Hello. I'm using blogspot as my platform. I want to disallow three pages from search engine crawlers. Can you show sample I can follow in the Customs Robot.txt box?. Appreciate your response. Thank you.

Editorial Team MOD
November 4, 2013 at 1:03 AM

Here's the example

User-agent: *
Disallow: /search
Disallow: /2013/03/how-to-remove.html
Allow: /

November 6, 2013 at 1:21 AM

Hey MBL,
Just wanted to ask, how long does it take for a custom robots.txt file to show up/start working?
I applied one on my blog recently but when i navigate to mysite.com/robots.txt it still shows the default robots.txt content and not the one i added.

So wanted to know how long does it take for a custom file to come in effect.

Thanks

Editorial Team MOD
November 6, 2013 at 5:11 AM

It takes seconds to implement. Check whether you're adding it correctly or not.

December 12, 2013 at 2:19 AM

sir,
since i updated me robots.txt file on my blog i lost traffic from google...plz help ...
my blog traffic has drop down to 10% as it was before...what to do....plz help...is there any other reasion ..plz let me know..or plz tell the errors in my robots.txt file..

my blog - www.supper-android.blogspot.com/

Editorial Team MOD
December 12, 2013 at 9:02 PM

This is your Robots.txt File

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow:
Allow: /

Sitemap: http://www.supper-android.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: http://www.supper-android.blogspot.com/feeds/posts/default?orderby=UPDATED

Replace the Disallow with this

Disallow: /search

And Then, It might help you out.

February 15, 2014 at 5:26 AM

how long it takes time to effect?

Editorial Team MOD
February 16, 2014 at 12:02 PM

It affects as soon as your site is recrawled.

October 15, 2014 at 11:23 PM

User-agent: Mediapartners-Google
Disallow:

User-agent: *

Disallow: /search
Allow: /yahoo.com


Sitemap: http://fun-with-fun.blogspot.com/feeds/posts/default?orderby=UPDATED

October 15, 2014 at 11:23 PM

is this ri8?

Editorial Team MOD
October 17, 2014 at 3:12 AM

Yes Diya

January 15, 2015 at 5:37 PM

Hello, I am having problems with Twitter Cards and Blogger. My blog is www.bondadanonima.com.
Twitter's Troubleshooting page suggests: Configure your web servers to allow Twitter access via the robots.txt file.
witter uses the User-Agent of Twitterbot (with version, such as Twitterbot/1.0), which can be used to create an exception in your robots.txt file.

User-agent: Twitterbot
Disallow:

How/where do I do this?

Many thanks in advance

June 20, 2015 at 10:40 PM

Sir Create A blog And Set Up A domain After This I am Apply for Adsense They Are ask me for This Error ( http://www.selectomobile.tk/robots.txt?m=1#150 ) How I can solve this???Please helpme

Editorial Team MOD
June 23, 2015 at 9:36 AM

Apply At Adsense.com

July 20, 2015 at 2:01 PM

will this affect traffic ?

December 10, 2015 at 10:17 PM

Like wordpress i also want robot.txt to allow blogger labels to appear in search result. How to allow label and suppose if i do this the result will be better or not.

Editorial Team MOD
December 18, 2015 at 3:48 AM

If you want your labels to be indexed, replace your robots.txt file with this

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /search/label
Allow: /


Sitemap: http://www.yoursite.com/sitemap.xml

Editorial Team MOD
December 18, 2015 at 3:49 AM

Its to mention that indexing Labels won't help in rankings.

December 27, 2015 at 10:06 AM

hey thanks bussy for such information, ihave a blog on blogspot ( http://whatsappstatus-mystatus.blogspot.in/ )
nd my robot.text is
# Blogger Sitemap generated on 2015.12.16
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://whatsappstatus-mystatus.blogspot.in/atom.xml?redirect=false&start-index=1&max-results=500
but when im searching my blog in google it always shows
A description for this result is not available because of this site's robots.txt.
for all my labels.. i dont know what is getting wrong could u help me figure out this problem.
thanks aarav

Editorial Team MOD
December 28, 2015 at 4:13 AM

Labels doesn't provide any description in the search, this is how Blogger site works. You can't do anything about it.

December 30, 2015 at 8:24 AM

thanks for reply on my post.
my secound question is this
how could i get better rank in google?
after submiting my blog in google webmasters im not getting any traffic from google . its 1 month now . what is the problem ?

Editorial Team MOD
December 30, 2015 at 11:30 AM

Webmaster tool just helps you index your pages in search, but to get traffic from Google you need to rank your website on keywords that are searched by people.

1. You need backlinks to get better ranking.
2. Better Social Score.
3. Great Content.

Read this post to know about different ways to get traffic:

http://www.mybloggerlab.com/2012/06/how-to-bring-real-targeted-usa-visitors.html

http://www.mybloggerlab.com/2012/02/5-fantastic-tips-to-increase-website.html

February 13, 2016 at 8:59 PM

use this sitemap in your blog robots.txtfile like www.yourblogaddress.com/sitemap.xml it is best for seo...

February 13, 2016 at 9:08 PM

i have two robots.txt file which one is correct

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Disallow: /search?q=*
Disallow: /*?updated-max=*
Allow: /

Sitemap: http://lifehacklife.blogspot.com/sitemap.xml


2:

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Disallow: /search?q=*
Disallow: /*?updated-max=*
Allow: /

Sitemap: http://lifehacklife.blogspot.com/feeds/posts/default

Editorial Team MOD
February 14, 2016 at 10:27 AM

Both looks fine, You have just changed the sitemap link. It depends on you which sitemap link you choose.

Cheers

Post a Comment

We’re eager to see your comment. However, Please Keep in mind that all comments are moderated manually by our human reviewers according to our comment policy, and all the links are nofollow. Using Keywords in the name field area is forbidden. Let’s enjoy a personal and evocative conversation.

Helping You to do Cool Things With Blogger Since 2012™.

© Copyright 2012 - 2018. MBL Networks, All Rights are Strictly Reserved.