google: do not use robots.txt
https://support.google.com/webmasters/answer/9689846?hl=en > Do not use robots.txt as a blocking mechanism.
https://developers.google.com/search/blog/2020/09/new-open-source-robotstxt-projects
https://support.google.com/webmasters/answer/7489871
https://developers.google.com/search/blog/2019/07/a-note-on-unsupported-rules-in-robotstxt
