While starting a new blog or website, we always desire to stop Search engines from crawling or indexing the content. The intention may appear to be a wrong one because, who would like to stop search engines from indexing their newly born site. Well, we do have some solid reason. When a website is new, it has lots of creepy errors that kill the beauty of a site. As we all say, First impression of a person is the last impression. Similarly, search engines do care about the structure, and how well the site is optimized not only for Search Engines but also for the human users. Keeping that in mind, it’s necessary to hide your websites from the Search engines until it gets ready for release.
Before launching or what should we say, releasing it publicly, we always do a few experiments like testing the Headings, Fonts, Color Scheme and more developing stuff. Since, the search engines are able to crawl your website. Therefore, they index those pages which are only meant for testing purpose, not for visitors or search engines. These things could hurt a website. The affect would be diminished, but it would be there until or unless, your site gets fare enough response from the audience.
Go to Blogger.com >> Settings >> Basic >> Privacy. Now there you will able to see two options i.e. one for stopping Search robots from crawling your site and second for not allowing your blog to be listed in the Blogger Profile.
Now Select “NO” on both options i.e. Add your blog to our listings, Let search engines find your blog and hit the “Save” button to conclude. Now search engines would continue to crawl your site, but your Robots.txt file would stop them from indexing your testing posts/pages.
Once your site is ready for the release, you can follow the same steps as mentioned above but this time select “YES”. It would give access to the robots, and they will start crawling and indexing your site.
Note: Only apply the following instructions if your site is either new or, you want to test a few things on your site. By applying the following technique, we are not stopping search bots to crawl our site instead we are not allowing them to index the site in a search result. Once the site is ready to be launched we can easily again give access to the search bots.