I need Google to look at my pages. If Google doesn't look at my pages, then my pages won't be in its search results. Therefore, I am ready to do a lot of things that will help my pages to be scanned. In order to understand what Google needs, of course, I will go to the Google Search Console and register my site on it.
File robots.txt you need to create so that some pages are not indexed by Google. To be honest, if I don't want the pages not to be indexed, I'd rather restrict access to them through authorization. I consider robots.txt as unreliable protection.
Therefore, do not leave the following information for free access:
- personal data of customers,
- their account numbers,
- PIN codes,
- mobile phone number, etc.
This data is too confidential to trust their to file robots.txt, under no circumstances should they fall into the wrong hands or be made public, I wouldn't even trust them with a hosting provider, in other words, they would be encrypted in the database. And there will be a separate article about this.
However, if the page is not ready yet, it will not be very good if Google indexes it - if the quality of the unfinished page does not appeal to Google's artificial intelligence, this may negatively affect the ranking of the site itself. Here is for the unprepared pages robots.txt - it fits perfectly.
Although, I prefer just not to put the pages that are not ready on an external site. It's easier than constantly changing the file robots.txt.
Well, now a simple and unambiguous answer to the question of the article itself: is the file robots.txt really necessaryon my website?
The unequivocal answer is NO.
If web site works fine, confidential data is protected through a password or AES-256, there is not a single faulty web page, or I simply do not know that the pages are faulty (and when I find out, I will fix everything quickly), then there is no point in adding the robots.txt file.