https://www.jwz.org/robots.txt is a useful template that results in fail2ban "running full tilt", but apparently the AI companies will slurp that shut up enough to cause more technical problems - that site seems to have some practical advice on handling this sort of thing without giving control to cloudflare
- Posts
- 8
- Comments
- 372
- Joined
- 3 yr. ago
- Posts
- 8
- Comments
- 372
- Joined
- 3 yr. ago
Lemmy Shitpost @lemmy.world Construction of the Sarlacc pit
Work Reform @lemmy.world "Labour has been sucked into the WFH culture war. It should know better"
Mildly Infuriating @lemmy.world This seat reservation doesn't reserve any seats
InsanePeopleFacebook @lemmy.world SovCit visits a presidential rally
Fuck Cars @lemmy.world Urban Designer Answers City Planning Questions
Fediverse @lemmy.world Labour MPs begin quitting X over ‘hate and disinformation’
Privacy @lemmy.ml YouTube downloader is looking for a maintainer
micromobility - Bikes, scooters, boards: Whatever floats your goat, this is micromobility @lemmy.world Custom camper towed with an e-bike




"A criminal is trying to block these requests from accessing their website!"