I might probabbly add a post for SEO beginners about robots.txt and how to create them in a later post..
So basically this is for someone who already know a bit about web crawlers and robots.txt .
Do all the web bots (crawlers) refer to robots.txt before they enter the site?
Not necessarily yes.. I was trying to build a miniature web robot and as it was the initial stage, I wasn’t following the robots.txt instructions..
So its not a filter you are keeping in your site by using robots.txt . Its a rule for an ethical search engine robot to check robots.txt for excluded files. Never take this for granted that you can hide any file from any robot by keeping a robots.txt instruction.
This post doesn’t mean that I am into unethical robot development, I will try to include functionality to check robots.txt instructions.