setting up custom robots.txt for blogger - Seo Optimization Tips

setting up custom robots.txt for blogger - Seo Optimization Tips, blogger custom robots.txt file has allowed us to block or diallow any folders or pages on our website or blog by our own selves.we just have to edit custom robots.txt file and rest of work will do crawlers.


by default,custom settings for robots.txt for blogger is as follows.


User-agent: Mediapartners-Google
Disallow: 


user-agent: *

disallow: /search

allow: /

Sitemap: http://usmanmohiuddin123.blogspot.com/atom.xml redirect=false&start-index=1&max-results=500



where as the the terms are explained as;

user-agent is for all robots.do not change this code or you will have troubles and troubles.

google mediapartners-google: is a google adsense crawler from google.its crawles your website for matching the ads that fits best with your content.changing this will destroy your adsense earning.


disallow: you can use this file to allow or block any pages or folders on your website.by default.blog labels are not indexed as it is blocked by robots.txt.


latest update:

after google pagination update,we use the following configuration file to block google pagination page.



User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search?updated-min=
Disallow: /search?updated-max=
Disallow: /search/label/*?updated-min=
Disallow: /search/label/*?updated-max=
Allow: /

Sitemap: http://your site name/atom.xml redirect=false&start-index=1&max-results=500