.htaccess - Will this robots.txt only allow googlebot to index my site? -


Will this robots.txt file only allow googlebot to index the index.php file of my site? CAVEAT, I have an htaccess redirect that people type

http://www.example.com/index.php

http://www.example.com/

Therefore, this is my robots.txt file contents ...

< Pre> User-agent: Googlebot Allow: /index.php Reject: / User-agent: * Disclaimer: /

Thanks in advance!

Not really.

Good bot
Only "good" bots robots.txt instructions (all robots and spiders / code / robots.txt Do not bother to read). It can not include the bots of all the main search engines, but of course it means that some web crawlers will ignore your requests completely (if you actually prevent bots / crawlers from seeing some parts You should use .htaccess or password protection. Your site).

Second Investigation
This second visit, including appearing as a browsing user, visiting your website several times on your website, robots.txt Ignore the file. The second trip is not actually an index (if you are worried), but it is checked to ensure that you are not trying to fool the indexing bot (for SEO etc.).

This is your syntax ... If you are asking all, yes, it will work, hope can not be done along with the bus.


Comments