A robots.txt file is an important and easy way to improve your websites fortunes. This tiny, simple file is attached to every website on the Internet, but most people don’t even know about it and if they do, they are unaware of its potential. Designed to work with search engines, it is an excellent way of unlocking some of those SEO signals that your website needs to perform at its very best.

Source: https://www.semrush.com/blog/beginners-guide-robots-txt/