Technological Web optimization: Technical Search engine marketing refers to Web page and server optimization that assists the crawler in crawling, indexing and ranking functions to rank your internet site far better. txt file is then parsed and will instruct the robot as to which web pages aren't to get crawled. https://www.youtube.com/watch?v=V7lEomNLUOE&t=8s