10. Gzip Compression: Help Gzip compression on the server to reduce the measurement of information transferred between the server and also the person’s browser. txt file is then parsed and will instruct the robot regarding which pages are certainly not to be crawled. Like a internet search engine crawler may https://www.youtube.com/watch?v=0xsCkQ6bbjk