First of all, go to Blogger.com >> your site >> Settings >> Search engine Preference >> Crawlers and indexing. Now you will be able to see two options i.e. Custom robots.txt and Custom robots header tags. These two options would provide you the flexibility to customize your robot.txt file.
- Custom robots.txt: This option provides you the ability to edit your whole robot.txt file. You can just type your content whether you want the content to be crawled by spiders or not. However, You can always undo your actions and can always revert back to normal.
- Custom robots header tags: This option is a bit completed. It does not provide you the ability to write your codes instead it provides few options with the check boxes. So, if have no idea about head tags then stay away from it.
Now you have to enable the custom robot.txt file so press the edit button which is present next to the “Custom robots.txt” option. After selecting the edit button, it would ask you to enable custom robots.txt content so press “Yes” and proceed to the next step.
In the text area, type the content which you want to exclude from being crawled. After editing the file according to your needs, press the save button to conclude. However, if you want to revert back to the default robot file then, you can select “No” instead of “Yes.
For Example:
User-agent: *
Disallow: /about
User-agent: *
Disallow: /contact
User-agent: *
Disallow: /services
We hope this tip would help you. If you are having any issue regarding crawling then, feel free to leave your questions in the comments below and our experts would try to help you in solving them.
About the Editorial TeamThe Editorial Team of MyBloggerLab consists of a group of Professional Blogger geeks Led by Syed Faizan Ali (Founder of MyBloggerLab).
59 comments
Thanks Syed Bro well explanations. We will just only write the name of page like About me.
or
We have to write full url of that page and put it here. How?
No, you have to write the URL. For example, if your want to stop robots from crawling this URL (www.mybloggerlab.com/p/about.html) then, in the robot.txt file you will enter this command.
User-agent: *
Disallow: p/about.html
Exclude the main domain and just type the remaining part of it.
pls explain the examples mentioned in detail
We will create a separate article to explain the commands which we use in robot.txt file.
Hi,previous when i publish a post i could see it in google search results in a few minutes,it was gettting indexed automatically .But now donno what happened,i dont see my published post in google even after a day,and if I manually submit my blog post to google ,it is getting indexed.Pease let me know what would the problem.
my blog: genuinecontent.blogspot.com
It is not only you who is facing this issue. Almost the majority of BlogSpot blogs are getting index lately. So don't get dishearten and hope of the best.
You have to include this as disallow condition.
Disallow: /p/about.html
Syed i have insert my robot.txt file in blogger dashboard in crawling section like this is it OK.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://www.probloggertricks.com/feeds/posts/default?orderby=UPDATED
Yup, its fine
Yes, This is default robots.txt file. If you keep disable custom file, then the same file will work for blogger. You can verify your default robots.txt file using:
www.yourblog.com/robots.txt
I'm not much know about robots.text . If it's possible please tell me what should become robots.text for my blog and how I wrote it my blog : http://seo-supports.blogspot.com
You are using Default Robots.txt file Its fine there is no need to edit. Cheers
hey please tell me if i want all my pages and posts being crawled by search engines then what would be the code???
i am using this one...please tell me is this right???
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search?q=*
Disallow: /*?updated-max=*
Allow: /
Sitemap: http://www.downloadmediazone.blogspot.com/feeds/posts/default?orderby=updated
hey please help me i have a problem with my blog. when i search my url in search engines then the description of all posts appears to be same there is nothing related to my post in its description. what should i do now????
You should add Description specially while publishing a post. Its better to it manually.
This is a Default Robot file so its just fine.
why should we add robot txt ?
Through it, you can control the crawling of your site. You can let search engine index few but not index everything. its like a key to your site for crawllers
I had a robot.txt file, but I did not like how it worked, so I DISABLED it to get back to normal. How long will it take to get back to the default condition as far as operation is concerned? It is not immediate.
On Disabling the Robot.txt file it gets back to normal within seconds. Cheers
Sir.
I Search In Google My Site http://www.desipakimaza.com/
I See This Line About My Site.
A description for this result is not available because of this site's robots.txt
And I enter Custom robots.txt
User-agent: *
Allow: /
User-agent: Mediapartners-Google
Disallow:
Allow: /
Sitemap: http://www.desipakimaza.com/feeds/posts/default?orderby=UPDATED
But Same Problem Please Tell Me I Enter Custom robots.txt Is Wrong?
Your Robot file is default. and I don't see any problems from here. I also Google'd your site and every thing seems to work fine at my end. Cheers
Thanks
nice info, thanks
Thanks very much, much helpful
please tell me what should be the code if i want all my pages and posts being crawled by search engines ...
is this code is ok
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search?q=*
Disallow: /*?updated-max=*
should i also add the sitemap?
Allow: /
Adding to my previous comment pl tell is this code is alright,
User-agent: Mediapartners-Google
Allow:/
User-agent: Googlebot
Disallow: /?m=1
Disallow: /?m=0
Disallow: /*?m=1
Disallow: /*?m=0
User-agent: *
Disallow: /search?*
User-agent: Twitterbot
Allow:/
User-agent: *
Disallow: /search
Disallow: /p/*
Disallow: /view/*
Allow: /
Sitemap: http://aboutwomenhealthandbeauty.blogspot.com/feeds/post/default?orderby=UPDATED
it would be grateful to you, if you answer my query,
Regards,
There no need to do anything, Just use default robot.txt file as provided by blogger itself
Thanks for your answer but I found no default setting there, robot.txt file is disabled by default. What I am supposed to do in search preferences then?
Here you go
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://www.mybloggerlab.com/feeds/posts/default?orderby=UPDATED
Replace: MyBloggerlab.com with your domain.
Check the above comment!
Sir, this is harsha. My web page is not displayed in search engine and also it is not opening. In some systems our website is opening which do not have the security for systems and the systems which have avast security then it is not opening. could you please help me sir.....
Can you provide your blog URL?
Hello. I'm using blogspot as my platform. I want to disallow three pages from search engine crawlers. Can you show sample I can follow in the Customs Robot.txt box?. Appreciate your response. Thank you.
Here's the example
User-agent: *
Disallow: /search
Disallow: /2013/03/how-to-remove.html
Allow: /
Hey MBL,
Just wanted to ask, how long does it take for a custom robots.txt file to show up/start working?
I applied one on my blog recently but when i navigate to mysite.com/robots.txt it still shows the default robots.txt content and not the one i added.
So wanted to know how long does it take for a custom file to come in effect.
Thanks
It takes seconds to implement. Check whether you're adding it correctly or not.
sir,
since i updated me robots.txt file on my blog i lost traffic from google...plz help ...
my blog traffic has drop down to 10% as it was before...what to do....plz help...is there any other reasion ..plz let me know..or plz tell the errors in my robots.txt file..
my blog - www.supper-android.blogspot.com/
This is your Robots.txt File
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow:
Allow: /
Sitemap: http://www.supper-android.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: http://www.supper-android.blogspot.com/feeds/posts/default?orderby=UPDATED
Replace the Disallow with this
Disallow: /search
And Then, It might help you out.
how long it takes time to effect?
It affects as soon as your site is recrawled.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /yahoo.com
Sitemap: http://fun-with-fun.blogspot.com/feeds/posts/default?orderby=UPDATED
is this ri8?
Yes Diya
Hello, I am having problems with Twitter Cards and Blogger. My blog is www.bondadanonima.com.
Twitter's Troubleshooting page suggests: Configure your web servers to allow Twitter access via the robots.txt file.
witter uses the User-Agent of Twitterbot (with version, such as Twitterbot/1.0), which can be used to create an exception in your robots.txt file.
User-agent: Twitterbot
Disallow:
How/where do I do this?
Many thanks in advance
Sir Create A blog And Set Up A domain After This I am Apply for Adsense They Are ask me for This Error ( http://www.selectomobile.tk/robots.txt?m=1#150 ) How I can solve this???Please helpme
Apply At Adsense.com
will this affect traffic ?
I need a perfect robot.txt file for my blogging tips, blogger widget, templates related site http://khatimethi.com/
Like wordpress i also want robot.txt to allow blogger labels to appear in search result. How to allow label and suppose if i do this the result will be better or not.
If you want your labels to be indexed, replace your robots.txt file with this
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /search/label
Allow: /
Sitemap: http://www.yoursite.com/sitemap.xml
Its to mention that indexing Labels won't help in rankings.
hey thanks bussy for such information, ihave a blog on blogspot ( http://whatsappstatus-mystatus.blogspot.in/ )
nd my robot.text is
# Blogger Sitemap generated on 2015.12.16
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://whatsappstatus-mystatus.blogspot.in/atom.xml?redirect=false&start-index=1&max-results=500
but when im searching my blog in google it always shows
A description for this result is not available because of this site's robots.txt.
for all my labels.. i dont know what is getting wrong could u help me figure out this problem.
thanks aarav
Labels doesn't provide any description in the search, this is how Blogger site works. You can't do anything about it.
thanks for reply on my post.
my secound question is this
how could i get better rank in google?
after submiting my blog in google webmasters im not getting any traffic from google . its 1 month now . what is the problem ?
Webmaster tool just helps you index your pages in search, but to get traffic from Google you need to rank your website on keywords that are searched by people.
1. You need backlinks to get better ranking.
2. Better Social Score.
3. Great Content.
Read this post to know about different ways to get traffic:
http://www.mybloggerlab.com/2012/06/how-to-bring-real-targeted-usa-visitors.html
http://www.mybloggerlab.com/2012/02/5-fantastic-tips-to-increase-website.html
use this sitemap in your blog robots.txtfile like www.yourblogaddress.com/sitemap.xml it is best for seo...
i have two robots.txt file which one is correct
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Disallow: /search?q=*
Disallow: /*?updated-max=*
Allow: /
Sitemap: http://lifehacklife.blogspot.com/sitemap.xml
2:
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Disallow: /search?q=*
Disallow: /*?updated-max=*
Allow: /
Sitemap: http://lifehacklife.blogspot.com/feeds/posts/default
Both looks fine, You have just changed the sitemap link. It depends on you which sitemap link you choose.
Cheers
Post a Comment
We’re eager to see your comment. However, Please Keep in mind that all comments are moderated manually by our human reviewers according to our comment policy, and all the links are nofollow. Using Keywords in the name field area is forbidden. Let’s enjoy a personal and evocative conversation.