Google Blogroll is crawling my blog at a very slow rate

  • Author
  • #391137

    I’ve had my blog for a couple of months and am high in the Google search and my entries always hit the Google blogsearch within minutes and appear in Google email Alerts for blog searches for that topic.

    I’ve noticed the last few days that my entries are not showing up for about 24 to 36 hours in the blogroll and they no longer appear in Google Alerts in emails.

    The Google WEB search is fine. The little bit of content that appears under the name of my blog in Google search results changes regularly, so it is being indexed for the general Google search.

    I’m more concerned with blogroll results and I’ve noticed in my Webmaster tools that the crawl stats are almost at zero, where it used to be much higher. Could this be the reason the blog search is taking so long?

    Can this be related to WordPress or maybe just a Google problem the last few days? Has anyone else noticed that blog indexing has slowed down? Or is my blog being penalized by google for something?

    And is there anything I should be looking for in my webmaster results as a clue to what’s ahppening? I’m pretty clueless when it comes to interpreting these results.

    Here’s my blog:

    The blog I need help with is



    I think the blog search uses its own php script, I’m 99% sure they wouldn’t use google.
    As for the Alerts, it’ll display and index different things.

    I don’t see anything wrong, so I would contact Google and see what happens.

    There’s not really much you can do about it unless you want to head over to and get a bunch of SEO Plugins. Which, bad SEO could be the case here.


    SEO at wordpress.COM is very strong and will typically be better than a wordpress.ORG blog with any of the SEO plugins do to the extra boost bloggers here get from the global tags pages.

    Google has been redoing how they search and index blogs, and they are also going through all the search algorithms and tweaking those to compensate for the “SEO gamers” (those that try to scam the search engines).


    That should be “…due to the extra…”

    I haven’t yet had my coffee.


    Thanks sacredpath. I had read that google was testing out a new search engine. And my crawl stats have increased a bit. Maybe it’s just a blip. I noticed that other bloggers aren’t getting blog alerts either.


    There definitely seems to be a problem with my blog specifically.I did a test of the robot.txt at Google Webmaster and it came up with a couple of errors. I don’t know if these errors are always there, but here they are:

    These are the test results:

    Line 4: Crawl-delay: 3600 Rule ignored by Googlebot

    Allowed by line 20: Disallow:
    Detected as a directory; specific files may have different restrictions

    If anyone has any advice on how I can correct any of this I’d appreciate it.


    I’m just checking again if anyone can help me this issue of Google Blog Search not indexing my blog. The only advice I got from the Google Webmaster forum was to make sure I “acted like a blog!”


    I actually used the robots.txt file here as a starting point on my self-hosted blog and have had no issue with google crawling my site. I’ve used the same basic configuration in probably 25 blogs that I’ve set up for people i the page six months, and they are all getting plenty of attention from Google. In fact Google comes to those blogs and to mine far more often that I would have actually expected.

    The crawl delay is to keep search engines from completely hogging your blog and bandwidth for long periods of time, which some lesser search engines and bots will do. I’ve had to specifically ban certain bots because a few of them increased my bandwidth usage by 40% one month and I had to pay for extra bandwidth to keep my blog up just because of those search engines. As far as the crawl delay, Google just chose to ignore it.

    Contact staff at .


    According to this search at google, , they last crawled your blog on August 16, 2009, which is the date of your latest post.


    Thanks sacredpath.

    What I’m concerned with is the Google Blog Search. I usually index very quickly in the blog search and my updates go out quickly in google email alerts. But my last email alert was August 3 and my posts don’t show up in the blog search by keyword, only if you do a blogurl:site search. That’s not too helpful.

    Google seems to have changed how they index my blog, namely they don’t. And I don’t know if because I’m being penalized, since I don’t see any similar blogs having a similar problem.

    I think I will contact support since I seem to be having a problem with my sitemap as well.


    Google is screwed up right now for everyone.

    They are redoing everything search related.

    Everyone is affected. It ain’t just you.

    They don’t hate Jeffrey Dean Morgan I’m sure.



    Who could??

    I always thank you for your support, sacredpath. Now thanks for the laugh!


    You’re welcome for the help and the laugh.




    I have to concur: Google Blogsearch is hooped at the moment. I’m coming up higher in regular Google searches than in Google Blogsearches for the same terms.

    They hate everyone equally. Yay socialism?



    I wanted to let you know that I was having very similar problems with the slow/no indexing of new posts by Google blog search. Also the red X next to the sitemap submission on Google. A few weeks ago when it started I even put my blog “on vacation” for a few days to try and resolve the matter.

    I have also come to the conclusion that all the trouble is on Google’s end. After a new post, I check the sitemap on Google. If the red X is there, I have had to resubmit my sitemap up to 20 times before it takes! Usually it just takes 1-5 tries..

    Re the changes at Google, I found this url which allows one to see how the new “caffeinated” Google will change one’s results page. This is web search, not blog search. But interesting nonetheless….



    After a new post, I check the sitemap on Google.

    You mean immediately after? Sorry, but Google doesn’t work THAT fast.

    And I’m not sure why you’re having issues with the sitemap. I’ve never experienced Google Webmater Tools not reading my



    Hi husdal,

    I’m not sure what’s going on either. I posted to reassure “rhymeswithmagic” that he’s not the only one experiencing these issues and that it is probably due to the quirks of Google changing the way it crawls the web.

    I had no problems whatsoever until a few weeks ago. Like the OP, my posts used to almost immediately appear in Google blog search under “recent” and then shoot to the top under “relevance.” Then all of a sudden the latest posts weren’t being indexed and there were various errors on my Webmaster tools page. I contacted Google to resubmit my site and sitemap. Also corresponded with WordPress support, who saw nothing amiss with my sitemap, and advised me to resubmit it.

    After repeated–and I do mean repeated–resubmission of sitemap everything righted itself. New posts appeared in Google blog search and were at or near the top of the search page. However, I have noticed that the time ti takes them to appear in Blog search can vary widely. From just a few hours to 20 hours.



    However, I have noticed that the time ti takes them to appear in Blog search can vary widely. From just a few hours to 20 hours.

    Interesting. In my experience, all my blog posts appear almost immediately, and my sitemap is crawled every day, sometimes several times per day.

    I have errors appearing in my Webmaster Tools as well, but that’s usually deleted posts or pages that are no longer found.



    I have had problems with Google sitemap not reading my sitemap, so it does appear to be a common problem.


    thanks for the reassurance:) Do your posts appear in Google email Alerts for your keywords? And they now appear in Google blog search as they did before?


    I wouldn’t wonder about it if it was at least a few blogs in the same genre, but all the usual blogs are appearing in blog search. I’m the only one that appears to be knocked out of google blog search. I’m not as concerned with web search. I’m more concerned with getting my blog back in google email alerts.



    I am very glad to have found such a relevant and recent topic for my similar problem:

    I just started using the google webmaster tools, my domain is verified and I want to start making the SEO changes.

    However, the diagnostic sector reveals no data whatsoever. It says ‘no data available’ and when I click the link ‘why not’ it gives the unlikely reason that the preferred domain is not explored by crawlbots. I verified that but crawlbots do visit my 1,5 years old blog on a regular basis.

    I have no idea what causes that problem, any suggestions?

The topic ‘Google Blogroll is crawling my blog at a very slow rate’ is closed to new replies.