robots.txt file is stopping google from accessing my blog – how do I delete it?

  • Author
    Posts
  • #123730

    jonhw
    Member

    This file is on my wordpress blog: http://holfordwatch.info/robots.txt. It’s stopping google from indexing the blog, and I presume other search engines too. Any idea how I get rid of, or change, this file.

    My privacy options are set so that the blog should be indexable by search engines.

    #123896

    options
    Member

    HTTP/1.1 404 Not Found

    #123897

    jonhw
    Member

    sorry – the file’s at http://holfordwatch.info/robots.txt

    The full stop I put in broke the link. One of those days…

    #123898

    options
    Member

    I can’t beleive it. any idea about how this file has been put out there?

    #123900

    barry
    Staff

    Fixed! Just had to toggle the privacy options in Options –> Discussion

    #123902

    jonhw
    Member

    thanks barry – google doesn’t seem to have realised there’s a new robots.txt file yet :( but when it does it should work OK?

    #123904

    options
    Member

    he-he…

    PS
    everything will be ok, john. don’t worry. next time when bot visit your blog it’ll crawl whole site and reindex it.

    #123908

    #123909

    I receive this same file-type name and have this message at my google account –

    User-agent: IRLbot
    Crawl-delay: 3600

    User-agent: *
    Disallow:

    is that not normal? I didn’t ever set up the – http://todayyesterdayandtomorrow.wordpress.com/robots.txt

    I’ve noticed some “not found'” responses in response to some of my posts in the google account when viewing the stats –

    Thanks much –

    #123910

    barry
    Staff

    @johnw

    Google will read the changes soon and it should be fine then

    @todayyesterdayandtomorrow

    Yes, that is the default robots.txt file on all WordPress.com blogs. Since I don’t have access to your Google webmaster account (which is I assume what you are talking about), I can’t really comment on the “not found” messages.

    #123913

    jonhw
    Member

    thanks all for the help – very much appreciated. I was tearing my hair out trying to figure out why the robots.txt file was there…

    #124330

    scopettg
    Member

    Mine’s got a strange
    disallow: /next/

    I wonder why…(???)
    Why is there anything behind ‘disallow:’???
    Anyone?

    #124331

    justjennifer
    Moderator

    It appears that the directory /next/ on our WP.com sites contains whatever is necessary for the Next button in the bar at the top of your screen when you are surfing wp.com. It shouldn’t be crawled and therefore is blocked.

    Just for general robots.txt info
    http://www.robotstxt.org/wc/exclusion-admin.html

    #124332

    OK, I’m very new to all this, so this will probably be a no-brainer for you guys. My file http://eppe.wordpress.com/robots.txt says:

    User-agent: IRLbot
    Crawl-delay: 3600
    User-agent: *
    Disallow: /next/
    User-agent: *
    Disallow:

    Since my site is invisible to google even though its been up for more than 6 weeks, I thought this may be the source of the problem… Anyone? Oh, and thanks in advance for helping a poor newbie out!

    EDIT: Ok, should have had a look at some documentation BEFORE posting… If I understood correctly, my robot.txt is fine, not disallowing anything. Right?

    #124333

    justjennifer
    Moderator

    I am not staff and am only making an educated answer based on past experience. From what it says in the robots.txt file, the “next” directory is blocked to all Bots. The IRLbot apparently crawls sites too frequently, so it’s been told that it can’t crawl more than once every 3600 seconds.

    There’s really not much we can do in changing our robots.txt file other than setting up our preferences to allow our blogs to be searched by search engines, etc. under the Options/Privacy tab in our Dashboard. Make sure yours is set to allow it. All the rest is under the hood and not accessible to users.

    Forum volunteers have provided some much needed information on how to promote your blog, including submitting to search engines. Check out these threads:
    https://en.forums.wordpress.com/topic.php?id=15804&page&replies=4
    https://en.forums.wordpress.com/topic.php?id=16349&page&replies=5

    #124334

    scopettg
    Member

    Can we trust the volunteers, ie.

    #124335

    cjwriter
    Member

    It’s not a matter of trusting anyone. You post asking a question, we do our best to answer. We’re volunteers; we don’t claim to know everything but most of us have been using WP for a while and like to help newer users. And generally we don’t post in a thread unless we have a good idea of what we’re talking about. If you’re not sure about something we say, wait to hear back from staff instead. That’s your choice.

The topic ‘robots.txt file is stopping google from accessing my blog – how do I delete it?’ is closed to new replies.