I was scouring the indie-web earlier, and found a pretty useful list of bots to add to your robots.txt. But, since I’m not convinced that this is enough to keep them away, I also figured out a simple way to at least potentially completely block them from accessing your websites.
I suppose it comes down to being offensive or defensive. I don’t think being defensive is worth my time. I’m not paying for bandwidth and compute-time is so cheap it’s irrelevant so I’m on the offensive. You can do both if you want. There’s definitely more ready-to-go defensive solutions than there are offensive (your own article, for example), but I think tinkering and adapting my own solution is fun. It’s like a game of cat and mouse but they have money to lose and I don’t.