Need help? Check out our Support site, then


Posts taking HOURS to be picked up by feed readers

  1. My posts on my WordPress.com blog are taking quite a few hours these days to show up in feed readers like those of Yahoo or Google. Does the default crawl delay in robots.txt have anything to do with it?

    Also, Google has only indexed a few pages of my blog, and I am wishing there was a way it would add every new post to its index automatically. Is there a way to make this happen?

    Any suggestion/help by members or moderators will be appreciated because many of my posts are time-sensitive and news-oriented.

    thanks in advance,
    farrukh
    copywriter, journalist, advertising blogger
    http://farrukh.wordpress.com

  2. I am not sure if I understand if you are talking about Search Engines or Feed Readers. Readers pick up your feed based on their own refresh rate. Your feed is not delayed and you can see that for yourself by at anytime viewing your own feed at:

    http://farrukh.wordpress.com/feed/

    As for search engine indexing, this might be a topic that has been discussed before, but I have never seen my site indexed any differently than other hosting. What do you mean by:

    Does the default crawl delay in robots.txt have anything to do with it?

    Not sure what you mean by that.

    Trent

  3. robots.txt:

    User-agent: IRLbot
    Crawl-delay: 3600

    User-agent: *
    Disallow:

    Has no effect whatsoever on bots apart from IRLbot

    When you post, your feed is updated and pingomatic is pinged. From there whatever happens is out of our control. We cannot make anything happen.

  4. Thanks for your reply, trent.

    By Reader delay - I meant that that my latest post does not show in my Google feed reader (in google.com/reader) or Yahoo feed reader (in My Yahoo) for quite some time.

    Once a post is made, I see that it is quick to come up in Google's blog search (perhaps because it shows the latest posts on top), but not in Google's 'Web' search which takes ages or sometimes doesn't show at all. The criteria/algorithms for these two separate services by Google must be different in some sense I guess.

    Sometimes, the posts do get indexed by Google but exist in the 'Supplemental pages' which just don't show up in major searches.

    Ideally, one would like a new post to show up in people's feed readers right after it's posted and in search results too.

    Mark - thanks for setting me at ease about the robots thing - I think Googlebot ignores it. I'm trying to figure out if there is something I am doing that is not very friendly with search engine crawlers... or is this normal to have posts showing up in blog searches but not in main 'web' searches?

    farrukh

  5. I mentioned that I have a WordPress.com blog not because I am comparing with other hosts but because in the context of pages getting indexed and showing up in search engines because some people recommend sitemaps for better indexing. These require some complicated things to be done to WordPress pages (XML, etc) which I think I can't do using the free hosting option I currently use on WordPress. If there is a relevant topic to this that I might have missed - kindly point me in the right direction. Your help is much appreciated.

Topic Closed

This topic has been closed to new replies.

About this Topic