Generate effective robots.txt files generator free tool that helps ensure Google and other search engines are crawling and indexing your site properly https://kingofseotools.com/robots-txt-file-generator