Will this robots.txt file only allow googlebot to index the index.php file of my site? CAVEAT, I have an htaccess redirect that people type
http://www.example.com/index.php
http://www.example.com/
Therefore, this is my robots.txt file contents ...
< Pre> User-agent: Googlebot Allow: /index.php Reject: / User-agent: * Disclaimer: /
Thanks in advance!
Not really.
Good bot
Only "good" bots robots.txt
instructions (all robots and spiders / code / robots.txt Do not bother to read). It can not include the bots of all the main search engines, but of course it means that some web crawlers will ignore your requests completely (if you actually prevent bots / crawlers from seeing some parts You should use .htaccess or password protection. Your site).
Second Investigation
This second visit, including appearing as a browsing user, visiting your website several times on your website, robots.txt
Ignore the file. The second trip is not actually an index (if you are worried), but it is checked to ensure that you are not trying to fool the indexing bot (for SEO etc.).
This is your syntax ... If you are asking all, yes, it will work, hope can not be done along with the bus.
Comments
Post a Comment