Here's a guide to not getting indexed in major search engines, and to remove yourself from these sites.

Why would you want this?
Some people make personal sites for a small group of people, and do not want their site being indexed, and their valuable work all over the internet.

Okay, so let's say you do not want any search engine to index any part of your site. The first part is to create a file called

Quote:robots.txt


Then put the following in that file:

Code:
User-agent: *
Disallow: /

All pages won't get indexed. Now, let's say you only want the search engine to not index only one page. Then you must put the following in the robots.txt page.

Code:
User-agent: *
Disallow: /directory1/
Disallow: /directory2/
Disallow: /directory3/

This means those 3 directories will not be indexed by any search engine, good for work you want to show off to people, but don't want it lying around on the web.


Now let's say you do not want to have only a particular search engine to index your site, like Google, or Yahoo. Then use the following, and substitute "bot" with the search engine of your choice.

Code:
User-agent: Bot
Disallow: /


Now, here's the opposite, if you want to only have one search engine index your site, use the following and substitute "bot"

Code:
User-agent: bot
Disallow:

User-agent: *
Disallow: /

That's all, hope that helps :thumb1:

TY ^^.

good tut :)

Thanks might be useful for a personal blog thingy :)

good share thanks!!

I wonder, would banning the bots IP from cPanel help?

Probably would, but there are quite a few bots out there, can't ban them all.

thanks

Maybe we can ban the HTTP Referrer?

There are many bots that don't obey the robots.txt file though. Sure the big ones like google and yahoo abide by them but yea not all.
I have heard of people that specifically search robots.txt files for Disallow: /whatever.
Now if someone does not want the bots near it then more than likely its good gear.
Why not just allow bots to view forum or site but protect areas with .htaccess and password it.
Or if its a forum just make in certain user groups.
But i do both to be sure.