A bot or robot is any type of programmed application that is able to run automated tasks. Search engines bots, (also known as spiders or crawlers for example, Googlebot) are tasked with scouring the web and scanning websites’ pages. From there on, a massive database or index is generated of all sites that have been crawled. These catalogs are then consulted when compiling search engine results.
