Help regarding Robot.txt (google search)

Ranjith
from hyderabad
11 years ago

Your robots.txt is perfect. That is the default for all blogs and there is nothing that you need to change.

 

User-agent: Mediapartners-Google  // for the user agent Mediapartners-Google

Disallow:  // you are not blocking anything

 

User-agent: *  // for user agents other than  Mediapartners-Google  

Disallow: /search // you are blocking search pages on your blog and they SHOULD be blocked

Allow: / // allow all other pages

 

Sitemap: http://perpetualthinkings.blogspot.com/feeds/posts/default?orderby=UPDATED // you are simply stating the URL of your sitemap, you are not blocking anything

 

Uma Maheswari Anandane
from Karaikal
11 years ago

@ranjith

Thanks ! But why is my blog not indexed by search engine. I have created a sitemap also. One of my other blog is working quite well than this one. Any reasons as to why?

And even if I type the full title of my post along with my blog name, I find nothing in google search instead it shows the same title which I submitted as links to bloggers.com and indiblogger.com.

 

 

Aditya Dey
from Kolkata
11 years ago

I hope you haven't done anything silly with your search description tab under blogger settings.......Try to disable your robot.txt file and check the results.....resubmit your sitemap....let me know what happens

Uma Maheswari Anandane
from Karaikal
11 years ago

@aditya

I hope I did not

these r my settings for meta tags and crawler

-------------

Description ? [Disabled]  Custom robots.txt ? [Disabled]  Custom robots header tags ? [Disabled] 

-----------

I don't know how to disable robot.txt in blogger blogspot , tried but in vain 

can u guide on how to do?

 

Uma Maheswari Anandane
from Karaikal
11 years ago

also i don't think disabling robot.txt file will work

just found out all web domain and sub domain has the same robot txt  http://yoururl.blogspot.co.uk/robots.txt

so what might be the reason for google not indexing or even showing up my blog post? 

Aditya Dey
from Kolkata
11 years ago

Try one thing, sign in to your Webmaster tools account, click on your blog/site. Then click on "Health" (It will be on the left)....Check whaether there is any crawl error or not.....After that click on "Fetch as Google".....and Fetch your Home page, and all other linked page.....Sometimes it works....

Uma Maheswari Anandane
from Karaikal
11 years ago

Yes finally found the problem - solved ! - after deleting and adding my blog again in webmaster :)

Ranjith
from hyderabad
11 years ago

So, is your blog being indexed now?

Aditya Dey
from Kolkata
11 years ago

Happy to hear that......Happy blogging....

Uma Maheswari Anandane
from Karaikal
11 years ago

@ranjith @aditya - Haha not all the urls , I do find some still missing .I have to wait for a couple of days to see the changes though.  And it seems google uses relevent urls from g+, digg and all that pings to consider the urls to be highlighted in search result, I am waiting for the actual outcome .

Harsh Agrawal
from New Delhi
11 years ago

@Uma

Here are few things you could do for better indexing:

• Ping your blog manually using Pingler: https://pingler.com
• Comment on few active WordPress blog, as it will help search engine bots to follow link and crawl your blog.
• Follow a posting schedule, as this way search engine bots will fix a crawling rate for your blog

Aditya Dey
from Kolkata
11 years ago

@Harsh-Nice to have you back here again.......

Uma Maheswari Anandane
from Karaikal
11 years ago

@harsh 

Thank you ! But I think the problem is not with indexing. I did not find any crawl errors on my webmaster too and my specific url seems to be cached 

It is with search result. Instead of showing my blog post title (with the respective page) , it shows my blog title ( my home page with description matching my search result ) and other linked in pages with my title (like the one I submitted the link for indiblogger, bloggers, etc) but not my original page

I tried changing the  code in the <title> html but still i dont see any change