Robots.txt blocking google search
Recently switched my site from being hidden to searchable on google. This is now a few days ago.
It shows up in the results, but the content copy underneath reads…
“A description for this result is not available because of this site’s robots.txt – learn more.”
I have webmaster tools for google and it says robots.txt is fine and nothing is blocked, but trying to do instant previews fails with an errors message.
When I check my robots.txt file, when I refresh my browser it alternates between this one…
# This file was generated on Sun, 02 Sep 2012 08:27:28 +0000
And this one…
# This file was generated on Mon, 03 Sep 2012 21:28:42 +0000
# If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
# Please see http://en.wordpress.com/firehose/ for more details.
# har har
# MT refugees
Anyone know how to fix this?
The blog I need help with is darrenfishman.com.
I suggest you jumpstart Google’s indexing your site by a) posting new content and b) pinging manually via Pingomatic.com. Technically neither should be necessary, but then Google doesn’t crawl your blog every day, apparently. I have found that these two together will help force it to update.
Forgot to add that I submitted a site map and got this message…
“Sitemap contains urls which are blocked by robots.txt.” with 0 errors but 108 issues.
Thanks. Have done both.
We’ll see what happens.
The robots.txt file is correct and won’t block search engines from your blog.
It is blocking search engines from indexing our internal resources, like the log in page, signup form, etc.
Any reason why google is having issues with it then?
I’m not sure, you’ll need to contact them about this.
Hi everyone, I’m having the same issue (http://anyaworksmart.com/)
fishcoteque, Did you happen to get to the point what is needed to be done? did it help the posting new content and pinging?
Any suggestion appreciated,
A combination of new content, pinging and resubmitting to Google has seem to have done the trick. It can take a day or two though.
Thank you fishcoteque. I’m following the same steps, did the verification and content item.
Can anyone explain to me how I do “pinging”? I know it sounds naive, but I don’t have a background of developer and that is new to me
Still having issues with this.
The linked pages come up in google, but the homepage (the most important) does not appear.
Not sure if all the different robots.txt messages I see when I keep refreshing the page http://darrenfishman.com/robots.txt has anything to do with it?
The topic ‘Robots.txt blocking google search’ is closed to new replies.