Jump to content
Sign in to follow this  
bizint

Googlebot is blocked from my robots.txt

Recommended Posts

bizint

Hi I've just gone live and followed the verification instruction (using yoast mainly for SEO) and in google wemaster tools its telling me google bot if blocked? http://www.businessinteriors.co.uk/robots.txt'>http://www.businessinteriors.co.uk/robots.txt 8 hours ago 200 (Success) Googlebot is blocked from http://www.businessinteriors.co.uk/ What do I do? What is robots.txt? and why is it blocked...I dont see any settings?

Share this post


Link to post
Share on other sites
Rob
Robots.txt is a file found in your document root. You can edit it manually using FTP and a text editor. PageLines does not edit this file under any circumstances I know about, however Yoast SEO may have done so. I'd suggest you communicate directly with them. Googlebot searches your site for content so it's rather important.

Former PageLines Moderator, Food Expert and Raconteur

Share this post


Link to post
Share on other sites
josh_g0uk
Robots.txt is a text file that gives web-bots instructions about how to index (or not index) all or parts of your site. Your robots.txt file says: User-agent: * Disallow: This is fine (http://www.robotstxt.org/robotstxt.html) as the syntax allows all robots complete access. It shouldn't be blocking Google. Is Yoast SEO reporting an error? If you go to your WordPress Dashboard > Settings > Privacy, have you got this option selected? "I would like my site to be visible to everyone, including search engines (like Google, Bing, Technorati) and archivers"

Share this post


Link to post
Share on other sites
bizint
Yoast is nt showing an error...but when I access wemaster tools at Google...it tells me this is bad...shall I just change it to allow?! Using a text editor?

Share this post


Link to post
Share on other sites
bizint
also...using filezilla and text wrangler and connecting by ftp....and i cannot see any file called robots.tx...however: http://www.businessinteriors.co.uk/robots.txt This is showing...how do edit it?! And what should i put...and why does it say disallow anyway?!

Share this post


Link to post
Share on other sites
Simon
That is auto generated by wordpress. Disallow: means disallow nothing. No need to edit anything

Share this post


Link to post
Share on other sites
bizint
But if I go to webmaster tools in google....it gives me a big red box and says "Some import page is blocked by robot.txt [u]more details[/u]" Then when I click throguh on more details I get: Then this line: http://www.businessinteriors.co.uk/robots.txt 14 hours ago 200 (Success) Googlebot is blocked from http://www.businessinteriors.co.uk/ If there is no problem, why is google webmaster tool telling me there is?!

Share this post


Link to post
Share on other sites
Simon
If wordpress is set to private your robots.txt will be [code]User-agent: * Disallow: /[/code]

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this  

×