robots.txt file is stopping google from accessing my blog – how do I delete it?
This file is on my wordpress blog: http://holfordwatch.info/robots.txt. It’s stopping google from indexing the blog, and I presume other search engines too. Any idea how I get rid of, or change, this file.
My privacy options are set so that the blog should be indexable by search engines.
HTTP/1.1 404 Not Found
sorry – the file’s at http://holfordwatch.info/robots.txt
The full stop I put in broke the link. One of those days…
I can’t beleive it. any idea about how this file has been put out there?
Fixed! Just had to toggle the privacy options in Options –> Discussion
thanks barry – google doesn’t seem to have realised there’s a new robots.txt file yet :( but when it does it should work OK?
everything will be ok, john. don’t worry. next time when bot visit your blog it’ll crawl whole site and reindex it.
I receive this same file-type name and have this message at my google account –
is that not normal? I didn’t ever set up the – http://todayyesterdayandtomorrow.wordpress.com/robots.txt
I’ve noticed some “not found'” responses in response to some of my posts in the google account when viewing the stats –
Thanks much –
Google will read the changes soon and it should be fine then
Yes, that is the default robots.txt file on all WordPress.com blogs. Since I don’t have access to your Google webmaster account (which is I assume what you are talking about), I can’t really comment on the “not found” messages.
thanks all for the help – very much appreciated. I was tearing my hair out trying to figure out why the robots.txt file was there…
Mine’s got a strange
I wonder why…(???)
Why is there anything behind ‘disallow:’???
It appears that the directory /next/ on our WP.com sites contains whatever is necessary for the Next button in the bar at the top of your screen when you are surfing wp.com. It shouldn’t be crawled and therefore is blocked.
Just for general robots.txt info
OK, I’m very new to all this, so this will probably be a no-brainer for you guys. My file http://eppe.wordpress.com/robots.txt says:
Since my site is invisible to google even though its been up for more than 6 weeks, I thought this may be the source of the problem… Anyone? Oh, and thanks in advance for helping a poor newbie out!
EDIT: Ok, should have had a look at some documentation BEFORE posting… If I understood correctly, my robot.txt is fine, not disallowing anything. Right?
I am not staff and am only making an educated answer based on past experience. From what it says in the robots.txt file, the “next” directory is blocked to all Bots. The IRLbot apparently crawls sites too frequently, so it’s been told that it can’t crawl more than once every 3600 seconds.
There’s really not much we can do in changing our robots.txt file other than setting up our preferences to allow our blogs to be searched by search engines, etc. under the Options/Privacy tab in our Dashboard. Make sure yours is set to allow it. All the rest is under the hood and not accessible to users.
Forum volunteers have provided some much needed information on how to promote your blog, including submitting to search engines. Check out these threads:
Can we trust the volunteers, ie.
It’s not a matter of trusting anyone. You post asking a question, we do our best to answer. We’re volunteers; we don’t claim to know everything but most of us have been using WP for a while and like to help newer users. And generally we don’t post in a thread unless we have a good idea of what we’re talking about. If you’re not sure about something we say, wait to hear back from staff instead. That’s your choice.
The topic ‘robots.txt file is stopping google from accessing my blog – how do I delete it?’ is closed to new replies.