Where robots.txt has failed for me in the past, I have added dummy paths to it (and other similar paths hidden in html or in JS variables) which, upon being visited, cause the offending IP to be blocked.Eg, I'll add a /blockmeplease/ reference in robots.txt, and when anything visits that path, its IP, User-Agent, etc get recorded and it gets its IP blocked automatically.
- Posts
- 0
- Comments
- 3
- Joined
- 3 yr. ago
- Posts
- 0
- Comments
- 3
- Joined
- 3 yr. ago
What's funny is that when he first started moving production outside of the UK, he said it was because the pound was too strong and the UK wouldn't join the Euro.