robot.txt

  • Author
    Posts
  • #1466501

    luxmagkop
    Member

    Where do I edit this file through WordPress account? Google used to cache but it stopped a few months ago.

    The blog I need help with is lux-mag.com.

    #1466592

    justjennifer
    Moderator

    The robot.txt file for your WordPress.com site is determined by what you have checked under Dashboard>Settings>Reading>Privacy.

    You cannot edit it any other way.

    #1466595

    luxmagkop
    Member

    Thanks. I can only get to Dashboard>Settings>Reading. Somehow Privacy is not shown as an option.

    However for Site Visibility, it is set to “Allow search engines to index this site”

    How do I view the Privacy Setting?

    #1466599

    justjennifer
    Moderator

    Sorry, incorrect term on my part. It is “Site Visibility”. I’m still thinking old dashboard!

    If it is set to “Allow search engines to index this site” then search engines should be indexing your site.

    Is there a specific problem you are running into that you want to change your robot.txt?

    #1466600

    luxmagkop
    Member

    No problem.

    Google has not cache the pages since March 2013. No change to the setting since the dashboard was upgraded. I wonder what is wrong.

    #1466603

    justjennifer
    Moderator

    Hm, looking at the cache of your site on Google, I see last cached on September 15th. Have you done the cookie dance (ie deleted your browser’s cookies and cache)?

    While not directly related to this discussion, have you connected your site to Google’s Webmaster Tools? You might want to consider doing so if you haven’t. The instructions can be found on WordPress.com’s Support site.

    #1466607

    luxmagkop
    Member

    It does cache the front page but not the rest of the site. Google seems to have stopped caching for articles added after March 2013.
    I am checking this using Google Toolbar, Page Rank > Cache Snapshot of a Page. I have also added it to Webmaster Tools a week ago.

    Saw this from lux-mag.com/robots.txt on my blog site. Can this be the cause?

    User-agent: *
    Disallow: /next/

    #1466612

    justjennifer
    Moderator

    If you go to yoursite.wordpress.com/next/ you’ll see that this directory provides a way of navigating through various WordPress.com sites, so I do not believe it should affect how your site is being crawled.

    Your robot.txt allows crawling of your site by search engines. How or why Google is not caching earlier than March of this year is probably a question better suited for Google.

    #1466615

    luxmagkop
    Member

    Thank you.

The topic ‘robot.txt’ is closed to new replies.