Search Engine Robots: An Essential Beginning Guide.
An essential part of ensuring your site is listed in search engines is to entertain the search engine robots on your site. Being aware of how often each robot comes, what pages each robot has sampled and knowing what search engines they represent is a good first step in understand the strengths of each search engine in listing your site and identifying which search engines are serious about your site.