Need help? Check out our Support site, then


WordPress.com Forums » Ideas

Search engine related features

  1. derekblog
    Member

    Some ideas for controlling search engines:

    Deleted blogs' robots.txt
    All deleted blogs should have their robots.txt set to Disallow: /

    Wp-login index
    Stop wp-login.php from being indexed by adding <meta name="robots" content="noindex" />

    Disable archiving
    An option to disable archiving by search engines by adding <meta name="robots" content="noarchive" /> and Disallow the ia_archiver robot.

    Disable pings
    An option to disable pings to ping servers (such as weblogs.com.)

    Disable feeds
    An option to disable feeds should help to keep out some bots, (although it's quite a sacrifice.)

  2. nosysnoop
    Member

    Controlling search engines for a better pagerank?

  3. derekblog
    Member

    No, it won't boost your Google PageRank. It's about protecting privacy.

  4. timethief
    Member

    @derekblog
    Please don't forget to send staff a feedback including these ideas in it.

  5. drmike
    Member

    Um, we already have this feature. Dashboard -> Options -> privacy -> Don't allow search engines. Turns off all of that.

  6. timethief
    Member

    Thanks drmike :)

  7. derekblog
    Member

    @ drmike, people forget to disallow search engines before they delete a blog.
    @ timethief, ok, I've send a feedback, thanks.

  8. drmike
    Member

    Actually I believe the header for a deleted blog page is a 404 report that the search engines should be picking up on.

    Would it really matter though? Once the blog's content is gone and the same message is repeated over and over again to teh spiders, the site would be dropped fairly quickly.

  9. derekblog
    Member

    The internet archive doesn't drop it, unless the robots.txt is set to disallow.

  10. drmike
    Member

    If IA isn't obeying 'noindex,nofollow,' that's an issue you may want to bring up with them. When you choose the "Do not let search engines in" option, the following is placed within the header:

    <meta name='robots' content='noindex,nofollow' />

    You can also opt out via email at info at archive dot org.

    You can also set your blog to be private with the third option on that page. Gotta admit that even with the privacy setting set to 2 or 3, some search engines will index your site. That'll happen even with a robots.txt file. If you want privacy, that's probably going to be your bestoption. Either that or finding a host, installing the software yourself, and password protecting the directory teh blog sits in.

  11. derekblog
    Member

    IA obeys those tags. The problem is when you delete a blog, IA doesn't remove it.

  12. derekblog
    Member

    An easy way to solve problem 1 and 3 is to disallow ia_archiver from wordpress.com.

  13. timethief
    Member

    Thanks for contributing.

  14. drmike
    Member

    There's not contributing if they're posting here in the forums. They need to send this in via feedback on Monday.

    Again, if IA isn't obeying 'noindex,nofollow' then teh issue is with them. If they're not obeying internet standards, then they are the cause of their own issue.

  15. timethief
    Member

    Sorry -- I assumed that the blogger would follow the instructions I gave him in the third post above and send in a feedback to staff.

  16. derekblog
    Member

    @ drmike, IA copies everything from everywhere and keeps every copy forever and publicly. They do obey meta-tags, but the meta-tags are not retroactive in IA. So the robots.txt is the only option here.
    @ timethief, I'll send in an extra feedback to staff tomorrow.

  17. drmike
    Member

    You can also opt out as noted up above. Says that on their website.

  18. timethief
    Member

    Thanks for replying and letting us know you have sent in a feedback and will send in another one.

  19. derekblog
    Member

    The Internet Archive also uses archive.org_bot and ia_archiver-web.archive.org

  20. sohamdas
    Member

    Hi is it possible to get a list of all search engine terms used to come to my blog? If yes then how? I would like to know about it...

  21. cjwriter
    Member

    If you go to your blog stats in the WordPress dashboard (Dashboard > Blog Stats) there's an option to view your search engine terms. If you click it it will show you the search engine terms for the last 7 days; if you change the line in your browser's address bar from 7&blog to something like 100&blog, then you can see the stats back for as far as you like.

    Btw, in future if you have a question which isn't exactly the same as the existing topic, you're better off starting a new thread; if we see 20 or more replies we tend to think the question has already been resolved, so we might not see it.

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags