Need help? Check out our Support site, then


robot.txt

  1. Where do I edit this file through WordPress account? Google used to cache but it stopped a few months ago.

    The blog I need help with is lux-mag.com.

  2. The robot.txt file for your WordPress.com site is determined by what you have checked under Dashboard>Settings>Reading>Privacy.

    You cannot edit it any other way.

  3. Thanks. I can only get to Dashboard>Settings>Reading. Somehow Privacy is not shown as an option.

    However for Site Visibility, it is set to "Allow search engines to index this site"

    How do I view the Privacy Setting?

  4. Sorry, incorrect term on my part. It is "Site Visibility". I'm still thinking old dashboard!

    If it is set to "Allow search engines to index this site" then search engines should be indexing your site.

    Is there a specific problem you are running into that you want to change your robot.txt?

  5. No problem.

    Google has not cache the pages since March 2013. No change to the setting since the dashboard was upgraded. I wonder what is wrong.

  6. Hm, looking at the cache of your site on Google, I see last cached on September 15th. Have you done the cookie dance (ie deleted your browser's cookies and cache)?

    While not directly related to this discussion, have you connected your site to Google's Webmaster Tools? You might want to consider doing so if you haven't. The instructions can be found on WordPress.com's Support site.

  7. It does cache the front page but not the rest of the site. Google seems to have stopped caching for articles added after March 2013.
    I am checking this using Google Toolbar, Page Rank > Cache Snapshot of a Page. I have also added it to Webmaster Tools a week ago.

    Saw this from lux-mag.com/robots.txt on my blog site. Can this be the cause?

    User-agent: *
    Disallow: /next/

  8. If you go to yoursite.wordpress.com/next/ you'll see that this directory provides a way of navigating through various WordPress.com sites, so I do not believe it should affect how your site is being crawled.

    Your robot.txt allows crawling of your site by search engines. How or why Google is not caching earlier than March of this year is probably a question better suited for Google.

  9. Thank you.

Topic Closed

This topic has been closed to new replies.

About this Topic