CRAWLER

is a program that supports the INDEXING of content in SEARCH ENGINE (automatic). In short, it is about analyzing the CONTENT of available websites and placing them in a specific order / hierarchy, in response to a specific query (in the form of a KEYWORD) that you enter into the SEARCH ENGINE. Technically, an indexing robot is a specific type of bot (check the term BOT).