Robots.txt Is a file that exists on the root Listing of each Site and can be used to instruct search engines like google on which directories/files of the website they will crawl and contain in their index. Irrespective of whether you're employed on Search engine optimization oneself, delegate it to https://news.rafeeg.ae