Crawlers (or bots) are used to assemble info out there on the internet. By using web site navigation menus, and studying inner and external hyperlinks, the bots start to know the context of a page. Of course, the words, photographs, and other knowledge on pages also assist search engines like google