What is the Structure of Crawler-based Top Search Engines?
Published March 11, 2010 | 
The structure of top search engines is similar and consists of three main components, with the first is known as a Crawler or Spider, and it surfs the Web looking for changes such as pages that were recently created and any updates of already indexed pages. Spiders crawl through links on the Web, both inside the individual site and between websites, and if there is no site with a links to yours, the search engine spider will not visit your site unless you suggest the site yourself, which the majority top search engines allow.
The second component of crawler-based search engines is known as an Index, which gets the information gathered by search engine spiders. 
Think of the Index of crawler-based top search engines as a huge container that selectively concentrates the copies of all the pages found by spiders or crawlers that are identified and treated as relevant and worthy of being posted by top search engines, but even if crawlers or spiders visit your page, it doesn’t mean it will be indexed right away, and it could take for several weeks or months while waiting for you to resort to a “paid submission.”
The third top search engine’s component is Software, which scans the Index and then presents the consecutive list of the most relevant pages gathered on your search query. It is important to understand that every top search engine has its own algorithm, and the tuning of all search engine crawlers and spiders differs. If you decide to ultimately rank high at a top search engine, do not forget that the value will be given not by a person, but by a non-discerning search engine robot, which will not take into consideration and value the sophistication of the site design.