WordPress.com

1. farrukh
   Member
   Mar 16, 2007, 9:14 PM

   My posts on my WordPress.com blog are taking quite a few hours these days to show up in feed readers like those of Yahoo or Google. Does the default crawl delay in robots.txt have anything to do with it?

   Also, Google has only indexed a few pages of my blog, and I am wishing there was a way it would add every new post to its index automatically. Is there a way to make this happen?

   Any suggestion/help by members or moderators will be appreciated because many of my posts are time-sensitive and news-oriented.

   thanks in advance,
   farrukh
   copywriter, journalist, advertising blogger
   http://farrukh.wordpress.com

2. trent
   Member
   Mar 16, 2007, 10:02 PM

   I am not sure if I understand if you are talking about Search Engines or Feed Readers. Readers pick up your feed based on their own refresh rate. Your feed is not delayed and you can see that for yourself by at anytime viewing your own feed at:

   http://farrukh.wordpress.com/feed/

   As for search engine indexing, this might be a topic that has been discussed before, but I have never seen my site indexed any differently than other hosting. What do you mean by:

   Does the default crawl delay in robots.txt have anything to do with it?

   Not sure what you mean by that.

   Trent

3. mark
   Support
   Mar 16, 2007, 10:52 PM

   robots.txt:

   User-agent: IRLbot
   Crawl-delay: 3600

   User-agent: *
   Disallow:

   Has no effect whatsoever on bots apart from IRLbot

   When you post, your feed is updated and pingomatic is pinged. From there whatever happens is out of our control. We cannot make anything happen.

4. farrukh
   Member
   Mar 18, 2007, 11:31 PM

   Thanks for your reply, trent.

   By Reader delay - I meant that that my latest post does not show in my Google feed reader (in google.com/reader) or Yahoo feed reader (in My Yahoo) for quite some time.

   Once a post is made, I see that it is quick to come up in Google's blog search (perhaps because it shows the latest posts on top), but not in Google's 'Web' search which takes ages or sometimes doesn't show at all. The criteria/algorithms for these two separate services by Google must be different in some sense I guess.

   Sometimes, the posts do get indexed by Google but exist in the 'Supplemental pages' which just don't show up in major searches.

   Ideally, one would like a new post to show up in people's feed readers right after it's posted and in search results too.

   Mark - thanks for setting me at ease about the robots thing - I think Googlebot ignores it. I'm trying to figure out if there is something I am doing that is not very friendly with search engine crawlers... or is this normal to have posts showing up in blog searches but not in main 'web' searches?

   farrukh

5. farrukh
   Member
   Mar 18, 2007, 11:48 PM

   I mentioned that I have a WordPress.com blog not because I am comparing with other hosts but because in the context of pages getting indexed and showing up in search engines because some people recommend sitemaps for better indexing. These require some complicated things to be done to WordPress pages (XML, etc) which I think I can't do using the free hosting option I currently use on WordPress. If there is a relevant topic to this that I might have missed - kindly point me in the right direction. Your help is much appreciated.

Topic Closed

This topic has been closed to new replies.