Need help? Check out our Support site, then


WordPress.com Forums » Support

robots.txt file is stopping google from accessing my blog - how do I delete it?

  1. jonhw
    Member

    This file is on my wordpress blog: http://holfordwatch.info/robots.txt. It's stopping google from indexing the blog, and I presume other search engines too. Any idea how I get rid of, or change, this file.

    My privacy options are set so that the blog should be indexable by search engines.

  2. options
    Member

    HTTP/1.1 404 Not Found

  3. jonhw
    Member

    sorry - the file's at http://holfordwatch.info/robots.txt

    The full stop I put in broke the link. One of those days...

  4. options
    Member

    I can't beleive it. any idea about how this file has been put out there?

  5. barry
    Staff

    Fixed! Just had to toggle the privacy options in Options --> Discussion

  6. jonhw
    Member

    thanks barry - google doesn't seem to have realised there's a new robots.txt file yet :( but when it does it should work OK?

  7. options
    Member

    he-he...

    PS
    everything will be ok, john. don't worry. next time when bot visit your blog it'll crawl whole site and reindex it.

  8. sunburntkamel
    Member


  9. todayyesterdayandtomorrow
    Member

    I receive this same file-type name and have this message at my google account -

    User-agent: IRLbot
    Crawl-delay: 3600

    User-agent: *
    Disallow:

    is that not normal? I didn't ever set up the - http://todayyesterdayandtomorrow.wordpress.com/robots.txt

    I've noticed some "not found'" responses in response to some of my posts in the google account when viewing the stats -

    Thanks much -

  10. barry
    Staff

    @johnw

    Google will read the changes soon and it should be fine then

    @todayyesterdayandtomorrow

    Yes, that is the default robots.txt file on all WordPress.com blogs. Since I don't have access to your Google webmaster account (which is I assume what you are talking about), I can't really comment on the "not found" messages.

  11. jonhw
    Member

    thanks all for the help - very much appreciated. I was tearing my hair out trying to figure out why the robots.txt file was there...

  12. scopettg
    Member

    Mine's got a strange
    disallow: /next/

    I wonder why...(???)
    Why is there anything behind 'disallow:'???
    Anyone?

  13. justjennifer
    Member

    It appears that the directory /next/ on our WP.com sites contains whatever is necessary for the Next button in the bar at the top of your screen when you are surfing wp.com. It shouldn't be crawled and therefore is blocked.

    Just for general robots.txt info
    http://www.robotstxt.org/wc/exclusion-admin.html

  14. miguelhoeltje
    Member

    OK, I'm very new to all this, so this will probably be a no-brainer for you guys. My file http://eppe.wordpress.com/robots.txt says:

    User-agent: IRLbot
    Crawl-delay: 3600
    User-agent: *
    Disallow: /next/
    User-agent: *
    Disallow:

    Since my site is invisible to google even though its been up for more than 6 weeks, I thought this may be the source of the problem... Anyone? Oh, and thanks in advance for helping a poor newbie out!

    EDIT: Ok, should have had a look at some documentation BEFORE posting... If I understood correctly, my robot.txt is fine, not disallowing anything. Right?

  15. justjennifer
    Member

    I am not staff and am only making an educated answer based on past experience. From what it says in the robots.txt file, the "next" directory is blocked to all Bots. The IRLbot apparently crawls sites too frequently, so it's been told that it can't crawl more than once every 3600 seconds.

    There's really not much we can do in changing our robots.txt file other than setting up our preferences to allow our blogs to be searched by search engines, etc. under the Options/Privacy tab in our Dashboard. Make sure yours is set to allow it. All the rest is under the hood and not accessible to users.

    Forum volunteers have provided some much needed information on how to promote your blog, including submitting to search engines. Check out these threads:
    http://en.forums.wordpress.com/topic.php?id=15804&page&replies=4
    http://en.forums.wordpress.com/topic.php?id=16349&page&replies=5

  16. scopettg
    Member

    Can we trust the volunteers, ie.

  17. cjwriter
    Member

    It's not a matter of trusting anyone. You post asking a question, we do our best to answer. We're volunteers; we don't claim to know everything but most of us have been using WP for a while and like to help newer users. And generally we don't post in a thread unless we have a good idea of what we're talking about. If you're not sure about something we say, wait to hear back from staff instead. That's your choice.

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags