Need help? Check out our Support site, then


Google Blogroll is crawling my blog at a very slow rate

  1. rhymeswithmagic
    Member

    I've had my blog for a couple of months and am high in the Google search and my entries always hit the Google blogsearch within minutes and appear in Google email Alerts for blog searches for that topic.

    I've noticed the last few days that my entries are not showing up for about 24 to 36 hours in the blogroll and they no longer appear in Google Alerts in emails.

    The Google WEB search is fine. The little bit of content that appears under the name of my blog in Google search results changes regularly, so it is being indexed for the general Google search.

    I'm more concerned with blogroll results and I've noticed in my Webmaster tools that the crawl stats are almost at zero, where it used to be much higher. Could this be the reason the blog search is taking so long?

    Can this be related to WordPress or maybe just a Google problem the last few days? Has anyone else noticed that blog indexing has slowed down? Or is my blog being penalized by google for something?

    And is there anything I should be looking for in my webmaster results as a clue to what's ahppening? I'm pretty clueless when it comes to interpreting these results.

    Here's my blog: http://jeffreydeanmorganunlimited.com

    The blog I need help with is jeffreydeanmorganunlimited.com.

  2. I think the blog search uses its own php script, I'm 99% sure they wouldn't use google.
    As for the Alerts, it'll display and index different things.

    I don't see anything wrong, so I would contact Google and see what happens.

    There's not really much you can do about it unless you want to head over to WP.org and get a bunch of SEO Plugins. Which, bad SEO could be the case here.

  3. SEO at wordpress.COM is very strong and will typically be better than a wordpress.ORG blog with any of the SEO plugins do to the extra boost bloggers here get from the global tags pages.

    Google has been redoing how they search and index blogs, and they are also going through all the search algorithms and tweaking those to compensate for the "SEO gamers" (those that try to scam the search engines).

  4. That should be "...due to the extra..."

    I haven't yet had my coffee.

  5. rhymeswithmagic
    Member

    Thanks sacredpath. I had read that google was testing out a new search engine. And my crawl stats have increased a bit. Maybe it's just a blip. I noticed that other bloggers aren't getting blog alerts either.

  6. rhymeswithmagic
    Member

    There definitely seems to be a problem with my blog specifically.I did a test of the robot.txt at Google Webmaster and it came up with a couple of errors. I don't know if these errors are always there, but here they are:

    These are the test results:

    Line 4: Crawl-delay: 3600 Rule ignored by Googlebot

    Allowed by line 20: Disallow:
    Detected as a directory; specific files may have different restrictions

    If anyone has any advice on how I can correct any of this I'd appreciate it.

  7. rhymeswithmagic
    Member

    I'm just checking again if anyone can help me this issue of Google Blog Search not indexing my blog. The only advice I got from the Google Webmaster forum was to make sure I "acted like a blog!"

  8. I actually used the robots.txt file here as a starting point on my self-hosted blog and have had no issue with google crawling my site. I've used the same basic configuration in probably 25 blogs that I've set up for people i the page six months, and they are all getting plenty of attention from Google. In fact Google comes to those blogs and to mine far more often that I would have actually expected.

    The crawl delay is to keep search engines from completely hogging your blog and bandwidth for long periods of time, which some lesser search engines and bots will do. I've had to specifically ban certain bots because a few of them increased my bandwidth usage by 40% one month and I had to pay for extra bandwidth to keep my blog up just because of those search engines. As far as the crawl delay, Google just chose to ignore it.

    Contact staff at http://support.wordpress.com/contact/ .

  9. According to this search at google, site:jeffreydeanmorganunlimited.com , they last crawled your blog on August 16, 2009, which is the date of your latest post.

  10. rhymeswithmagic
    Member

    Thanks sacredpath.

    What I'm concerned with is the Google Blog Search. I usually index very quickly in the blog search and my updates go out quickly in google email alerts. But my last email alert was August 3 and my posts don't show up in the blog search by keyword, only if you do a blogurl:site search. That's not too helpful.

    Google seems to have changed how they index my blog, namely they don't. And I don't know if because I'm being penalized, since I don't see any similar blogs having a similar problem.

    I think I will contact support since I seem to be having a problem with my sitemap as well.

  11. Google is screwed up right now for everyone.

    They are redoing everything search related.

    Everyone is affected. It ain't just you.

    They don't hate Jeffrey Dean Morgan I'm sure.

  12. rhymeswithmagic
    Member

    LOL!!!

    Who could??

    I always thank you for your support, sacredpath. Now thanks for the laugh!

  13. You're welcome for the help and the laugh.

    :-)

  14. I have to concur: Google Blogsearch is hooped at the moment. I'm coming up higher in regular Google searches than in Google Blogsearches for the same terms.

    They hate everyone equally. Yay socialism?

  15. rhymeswithmagic,
    I wanted to let you know that I was having very similar problems with the slow/no indexing of new posts by Google blog search. Also the red X next to the sitemap submission on Google. A few weeks ago when it started I even put my blog "on vacation" for a few days to try and resolve the matter.

    I have also come to the conclusion that all the trouble is on Google's end. After a new post, I check the sitemap on Google. If the red X is there, I have had to resubmit my sitemap up to 20 times before it takes! Usually it just takes 1-5 tries..

    Re the changes at Google, I found this url which allows one to see how the new "caffeinated" Google will change one's results page. This is web search, not blog search. But interesting nonetheless....
    http://www.comparegoogle.com/

  16. After a new post, I check the sitemap on Google.

    You mean immediately after? Sorry, but Google doesn't work THAT fast.

    And I'm not sure why you're having issues with the sitemap. I've never experienced Google Webmater Tools not reading my blogname.wordpress.com/sitemap.xml

  17. Hi husdal,

    I'm not sure what's going on either. I posted to reassure "rhymeswithmagic" that he's not the only one experiencing these issues and that it is probably due to the quirks of Google changing the way it crawls the web.

    I had no problems whatsoever until a few weeks ago. Like the OP, my posts used to almost immediately appear in Google blog search under "recent" and then shoot to the top under "relevance." Then all of a sudden the latest posts weren't being indexed and there were various errors on my Webmaster tools page. I contacted Google to resubmit my site and sitemap. Also corresponded with WordPress support, who saw nothing amiss with my sitemap, and advised me to resubmit it.

    After repeated--and I do mean repeated--resubmission of sitemap everything righted itself. New posts appeared in Google blog search and were at or near the top of the search page. However, I have noticed that the time ti takes them to appear in Blog search can vary widely. From just a few hours to 20 hours.

  18. However, I have noticed that the time ti takes them to appear in Blog search can vary widely. From just a few hours to 20 hours.

    Interesting. In my experience, all my blog posts appear almost immediately, and my sitemap is crawled every day, sometimes several times per day.

    I have errors appearing in my Webmaster Tools as well, but that's usually deleted posts or pages that are no longer found.

  19. rhymeswithmagic
    Member

    hudsal,

    I have had problems with Google sitemap not reading my sitemap, so it does appear to be a common problem.

    Electricia,

    thanks for the reassurance:) Do your posts appear in Google email Alerts for your keywords? And they now appear in Google blog search as they did before?

    raincoaster,

    I wouldn't wonder about it if it was at least a few blogs in the same genre, but all the usual blogs are appearing in blog search. I'm the only one that appears to be knocked out of google blog search. I'm not as concerned with web search. I'm more concerned with getting my blog back in google email alerts.

  20. I am very glad to have found such a relevant and recent topic for my similar problem:

    I just started using the google webmaster tools, my domain is verified and I want to start making the SEO changes.

    However, the diagnostic sector reveals no data whatsoever. It says 'no data available' and when I click the link 'why not' it gives the unlikely reason that the preferred domain is not explored by crawlbots. I verified that but crawlbots do visit my 1,5 years old blog on a regular basis.

    I have no idea what causes that problem, any suggestions?

  21. And I have no idea what your problem is.

    However, the diagnostic sector reveals no data whatsoever. It says 'no data available.

    Mind you, this is the technical support forum for blogs on wordpress.com. It is not a support forum for Google Webmaster Tools.

    That said, if you 'just started' using Google Webmaster Tools it will take some time before there is any data available, usually several weeks, sometimes even months.

  22. Really, that long? I have no patience for that ;)

    But if you have any other tips to increase traffic please share. My blog has everything a blog needs to get high readership: high quality content, easy layout, regular postings, effective use of keywords, shared on social bookmark sites. In other words, most of the topics covered in this article: http://onecoolsite.wordpress.com/2009/01/15/twenty-five-steps-to-increase-blog-traffic/

    All the essentials are there, yet I don't get the traffic I deserve for my hours and hours of work. The last thing I thought of was SEO, but wordpress.com is already highly optimized.

    I really don't get it, am I missing something?

  23. I really don't get it, am I missing something?

    Yes. Patience. Ask yourself, is 'traffic' all you want? Or do you want regular readers that come back time and again and who link back to you and help you build a fan base, slowly but surely.

    All the essentials are there, yet I don't get the traffic I deserve for my hours and hours of work.

    There is no such as the traffic I deserve. You can spend hours and hours on optimizing your blog, but if you have nothing to offer or nothing that people are likely to search for than you will not get any traffic. You need to offer unique and sought after content to get traffic. Since my Dutch is rather lousy it's hard for me to judge your content specifically, but large parts of your posts appear to be taken word for word from newspapers or other websites. Because most of these sites rank a lot higher than you your posts are considered duplicate and will never make it into the search results.

  24. hello ,
    i need help !
    my blog is NOT visual by google and yahoo and also when i wrote some keywords or the name of my blog the resultat is 0 !not in the toolbar
    what must to change to work
    please for you help step by step !
    http://www.martinair1.blogspot.com
    thank you !

  25. my email is [email redacted] thank you for attention !

  26. rhymeswithmagic
    Member

    Your blog is a blogger blog. You should go there for support.

    http://www.google.com/support/blogger/

  27. Hi, can anyone help me out w/ a similar problem? Google had been crawling our site,
    http://www.agreatertown.com every few days, then it increased to once a day or once every two days. Now this week it hasn't come at all. Anyone have any ideas on this subject?
    Thanks,
    Drew

  28. Your blog is not a WordPress.COM blog, so our answers won't apply to you. Have you read the sticky threads at the top of this forum?

Topic Closed

This topic has been closed to new replies.

About this Topic