Robots.txt blocking google search

  • Author
    Posts
  • #998985

    fishcoteque
    Member

    Hi guys.

    Recently switched my site from being hidden to searchable on google. This is now a few days ago.

    It shows up in the results, but the content copy underneath reads…
    “A description for this result is not available because of this site’s robots.txt – learn more.”

    I have webmaster tools for google and it says robots.txt is fine and nothing is blocked, but trying to do instant previews fails with an errors message.

    When I check my robots.txt file, when I refresh my browser it alternates between this one…

    # This file was generated on Sun, 02 Sep 2012 08:27:28 +0000
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    And this one…

    # This file was generated on Mon, 03 Sep 2012 21:28:42 +0000
    # If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
    # Please see http://en.wordpress.com/firehose/ for more details.

    Sitemap: http://darrenfishman.com/sitemap.xml

    User-agent: IRLbot
    Crawl-delay: 3600

    User-agent: *
    Disallow: /next/

    User-agent: *
    Disallow: /mshots/v1/

    # har har
    User-agent: *
    Disallow: /activate/

    User-agent: *
    Disallow: /wp-login.php

    User-agent: *
    Disallow: /signup/

    User-agent: *
    Disallow: /related-tags.php

    User-agent: *
    Disallow: /public-api/

    # MT refugees
    User-agent: *
    Disallow: /cgi-bin/

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Anyone know how to fix this?

    Thanks

    The blog I need help with is darrenfishman.com.

    #999127

    raincoaster
    Member

    I suggest you jumpstart Google’s indexing your site by a) posting new content and b) pinging manually via Pingomatic.com. Technically neither should be necessary, but then Google doesn’t crawl your blog every day, apparently. I have found that these two together will help force it to update.

    #999128

    fishcoteque
    Member

    Forgot to add that I submitted a site map and got this message…

    “Sitemap contains urls which are blocked by robots.txt.” with 0 errors but 108 issues.

    #999134

    fishcoteque
    Member

    Hi raincoaster.
    Thanks. Have done both.
    We’ll see what happens.

    #999137

    macmanx
    Staff

    The robots.txt file is correct and won’t block search engines from your blog.

    It is blocking search engines from indexing our internal resources, like the log in page, signup form, etc.

    #999139

    fishcoteque
    Member

    Thanks macmanx.

    Any reason why google is having issues with it then?

    #999143

    macmanx
    Staff

    I’m not sure, you’ll need to contact them about this.

    #999339

    Hi everyone, I’m having the same issue (http://anyaworksmart.com/)
    fishcoteque, Did you happen to get to the point what is needed to be done? did it help the posting new content and pinging?
    Any suggestion appreciated,
    THNX

    #999340

    fishcoteque
    Member

    A combination of new content, pinging and resubmitting to Google has seem to have done the trick. It can take a day or two though.

    #999341

    Thank you fishcoteque. I’m following the same steps, did the verification and content item.
    Can anyone explain to me how I do “pinging”? I know it sounds naive, but I don’t have a background of developer and that is new to me
    Thanks!

    #999342

    fishcoteque
    Member

    Pingomatic.com

    #999343

    much appreciated!

    #999380

    fishcoteque
    Member

    Still having issues with this.

    The linked pages come up in google, but the homepage (the most important) does not appear.

    Not sure if all the different robots.txt messages I see when I keep refreshing the page http://darrenfishman.com/robots.txt has anything to do with it?

    #999381

    pcrumm
    Member

    Your robots.txt file is correct; the two different forms that are appearing intermittently shouldn’t be a problem.

    As @macmanx indicated, you’ll need to ask Google about the problems you’re experiencing. You can find Google Webmaster Tools’ help here:
    http://support.google.com/webmasters/?hl=en

The topic ‘Robots.txt blocking google search’ is closed to new replies.