bharath06/robotstxt
Robots.txt for the top ~1k sites
2 best points · 0 comments · 1 Hacker News post · 6 GitHub stars · View on GitHub
Hacker News discussions
- Easter Eggs in Robots.txt Files (2 points, 0 comments, by bolasanibk, 2022-08-26)
Robots.txt for the top ~1k sites
2 best points · 0 comments · 1 Hacker News post · 6 GitHub stars · View on GitHub