Assume you have a website, you type yourdomain.com on the search box and hit enter, it is not showing your site in search results, and you are wondering why, the above link will answer few of your questions.

• Search engines are not programmed to find each and every website online and show up results from them
• It is close to impossible for any search engine to display results for any page on any website

The above are few facts one need to know about search engines.

How a search engine finds your website?

There are two ways as I explained earlier,

• You let the search engine know about your website URL (use the add url option provided by them)
• Invite search engine crawler automatically by getting few hyperlinks pointed to your website URL from others sites which are already in search engine’s database (index).

Second method is best as far as I can say, you can even get your new website in search engine’s index in 24 hours (this method is what SEO agencies do, when they publicise that they can get your site indexed in 24 hours)

Getting into the SERPs (Search Engine Result Pages)
This depends on the keyword relevance and quality of the content on the webpage and the link popularity of your site. These are basics for any search engines, the weightages do vary from search engine to search engine. Google give more weightage to the incoming authoritative votes (links) from other websites.

Search Engine Optimization is understanding these search engine algorithms and planning the OnPage and Offpage strategies for developing the site for better placement in the search engine result pages.

So basically this is for someone who already know a bit about web crawlers and robots.txt .

Do all the web bots (crawlers) refer to robots.txt before they enter the site?

Not necessarily yes.. I was trying to build a miniature web robot and  as it was the initial stage, I wasn’t following the robots.txt instructions..

So its not a filter you are keeping in your site by using robots.txt . Its a rule for an ethical search engine robot to check robots.txt for excluded files. Never take this for granted that you can hide any file from any robot by keeping a robots.txt instruction.

This post doesn’t mean that I am into unethical robot development, I will try to include functionality to check robots.txt instructions.

Well my experience says it does.. We had an adserver which used to serve ads in an iframe into our websites. One fine day our System admin noticed that a page from our internal ip in google’s index.

Digging on to the above, we found that it was the page which was shown in other sites within iframes. There was no other way google could have indexed our page !!

Any other arguments? Shout here.. We will get into a strong conclusion !!

To me if you ask the same question, I would say it doesn’t matter at all

Why

Search Engine crawlers are like a normal user, they cant see what is the code written at the server level to output a page. So what a crawler/regular user will be getting is a page coded in HTML. (I am using HTML as a generic term, leave behind xhtml , wml etc).

Again another argument is that having all the files with .htm or .html is advantageous in SEO. I cant comment on this strongly, but still as of now there seem no algorithm exist to give rating based on the file extension.

Stick to basics

Having .htm or .html extensions/files in a site is advantageous too. If they are plain html pages, they load faster as it doesn’t go through the parsers such as ASP parser, PHP parsers etc. When a file is requested by client, the web-server just need to respond with the file without processing the server sided code in it, because they don’t have any..

But nowadays the web-servers are quite capable to send out processed server sided scrips without much delay, so you cant really differentiate the response times of plain html and pages like ASP,PHP etc.

Another disadvantage of websites using server sided scripts is the over usage of query parameters in URL. Its very easy to use a single server sided page and show different page outputs depending on the query parameters. This should be avoided as far as possible.

One can make use of URL rewriting to hide such lengthy URL’s having some many special characters such as ‘&’, ‘#’ , ‘?’ and show a very friendly URL with an ending file extension .htm or .html !!