Need help? Check out our Support site, then

robots.txt file is stopping google from accessing my blog - how do I delete it?

  1. This file is on my wordpress blog: It's stopping google from indexing the blog, and I presume other search engines too. Any idea how I get rid of, or change, this file.

    My privacy options are set so that the blog should be indexable by search engines.

  2. HTTP/1.1 404 Not Found

  3. sorry - the file's at

    The full stop I put in broke the link. One of those days...

  4. I can't beleive it. any idea about how this file has been put out there?

  5. Fixed! Just had to toggle the privacy options in Options --> Discussion

  6. thanks barry - google doesn't seem to have realised there's a new robots.txt file yet :( but when it does it should work OK?

  7. he-he...

    everything will be ok, john. don't worry. next time when bot visit your blog it'll crawl whole site and reindex it.

  8. I receive this same file-type name and have this message at my google account -

    User-agent: IRLbot
    Crawl-delay: 3600

    User-agent: *

    is that not normal? I didn't ever set up the -

    I've noticed some "not found'" responses in response to some of my posts in the google account when viewing the stats -

    Thanks much -

  9. @johnw

    Google will read the changes soon and it should be fine then


    Yes, that is the default robots.txt file on all blogs. Since I don't have access to your Google webmaster account (which is I assume what you are talking about), I can't really comment on the "not found" messages.

  10. thanks all for the help - very much appreciated. I was tearing my hair out trying to figure out why the robots.txt file was there...

  11. Mine's got a strange
    disallow: /next/

    I wonder why...(???)
    Why is there anything behind 'disallow:'???

  12. It appears that the directory /next/ on our sites contains whatever is necessary for the Next button in the bar at the top of your screen when you are surfing It shouldn't be crawled and therefore is blocked.

    Just for general robots.txt info

  13. OK, I'm very new to all this, so this will probably be a no-brainer for you guys. My file says:

    User-agent: IRLbot
    Crawl-delay: 3600
    User-agent: *
    Disallow: /next/
    User-agent: *

    Since my site is invisible to google even though its been up for more than 6 weeks, I thought this may be the source of the problem... Anyone? Oh, and thanks in advance for helping a poor newbie out!

    EDIT: Ok, should have had a look at some documentation BEFORE posting... If I understood correctly, my robot.txt is fine, not disallowing anything. Right?

  14. I am not staff and am only making an educated answer based on past experience. From what it says in the robots.txt file, the "next" directory is blocked to all Bots. The IRLbot apparently crawls sites too frequently, so it's been told that it can't crawl more than once every 3600 seconds.

    There's really not much we can do in changing our robots.txt file other than setting up our preferences to allow our blogs to be searched by search engines, etc. under the Options/Privacy tab in our Dashboard. Make sure yours is set to allow it. All the rest is under the hood and not accessible to users.

    Forum volunteers have provided some much needed information on how to promote your blog, including submitting to search engines. Check out these threads:

  15. Can we trust the volunteers, ie.

  16. It's not a matter of trusting anyone. You post asking a question, we do our best to answer. We're volunteers; we don't claim to know everything but most of us have been using WP for a while and like to help newer users. And generally we don't post in a thread unless we have a good idea of what we're talking about. If you're not sure about something we say, wait to hear back from staff instead. That's your choice.

Topic Closed

This topic has been closed to new replies.

About this Topic