Need help? Check out our Support site, then


WordPress.com Forums » Support

Robots.txt blocking google search

  1. fishcoteque
    Member

    Hi guys.

    Recently switched my site from being hidden to searchable on google. This is now a few days ago.

    It shows up in the results, but the content copy underneath reads...
    "A description for this result is not available because of this site's robots.txt – learn more."

    I have webmaster tools for google and it says robots.txt is fine and nothing is blocked, but trying to do instant previews fails with an errors message.

    When I check my robots.txt file, when I refresh my browser it alternates between this one...

    # This file was generated on Sun, 02 Sep 2012 08:27:28 +0000
    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    And this one...

    # This file was generated on Mon, 03 Sep 2012 21:28:42 +0000
    # If you are regularly crawling WordPress.com sites, please use our firehose to receive real-time push updates instead.
    # Please see http://en.wordpress.com/firehose/ for more details.

    Sitemap: http://darrenfishman.com/sitemap.xml

    User-agent: IRLbot
    Crawl-delay: 3600

    User-agent: *
    Disallow: /next/

    User-agent: *
    Disallow: /mshots/v1/

    # har har
    User-agent: *
    Disallow: /activate/

    User-agent: *
    Disallow: /wp-login.php

    User-agent: *
    Disallow: /signup/

    User-agent: *
    Disallow: /related-tags.php

    User-agent: *
    Disallow: /public-api/

    # MT refugees
    User-agent: *
    Disallow: /cgi-bin/

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

    Anyone know how to fix this?

    Thanks

    The blog I need help with is darrenfishman.com.

  2. raincoaster
    Member

    I suggest you jumpstart Google's indexing your site by a) posting new content and b) pinging manually via Pingomatic.com. Technically neither should be necessary, but then Google doesn't crawl your blog every day, apparently. I have found that these two together will help force it to update.

  3. fishcoteque
    Member

    Forgot to add that I submitted a site map and got this message...

    "Sitemap contains urls which are blocked by robots.txt." with 0 errors but 108 issues.

  4. fishcoteque
    Member

    Hi raincoaster.
    Thanks. Have done both.
    We'll see what happens.

  5. macmanx
    Happiness Engineer

    The robots.txt file is correct and won't block search engines from your blog.

    It is blocking search engines from indexing our internal resources, like the log in page, signup form, etc.

  6. fishcoteque
    Member

    Thanks macmanx.

    Any reason why google is having issues with it then?

  7. macmanx
    Happiness Engineer

    I'm not sure, you'll need to contact them about this.

  8. anyaworksmart
    Member

    Hi everyone, I'm having the same issue (http://anyaworksmart.com/)
    fishcoteque, Did you happen to get to the point what is needed to be done? did it help the posting new content and pinging?
    Any suggestion appreciated,
    THNX

  9. fishcoteque
    Member

    A combination of new content, pinging and resubmitting to Google has seem to have done the trick. It can take a day or two though.

  10. anyaworksmart
    Member

    Thank you fishcoteque. I'm following the same steps, did the verification and content item.
    Can anyone explain to me how I do "pinging"? I know it sounds naive, but I don't have a background of developer and that is new to me
    Thanks!

  11. fishcoteque
    Member

    Pingomatic.com

  12. anyaworksmart
    Member

    much appreciated!

  13. fishcoteque
    Member

    Still having issues with this.

    The linked pages come up in google, but the homepage (the most important) does not appear.

    Not sure if all the different robots.txt messages I see when I keep refreshing the page http://darrenfishman.com/robots.txt has anything to do with it?

  14. pcrumm
    Staff

    Your robots.txt file is correct; the two different forms that are appearing intermittently shouldn't be a problem.

    As @macmanx indicated, you'll need to ask Google about the problems you're experiencing. You can find Google Webmaster Tools' help here:
    http://support.google.com/webmasters/?hl=en

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags

No tags yet.