I think it's akin to the sentiment that we see on here sometimes where people wish that Lemmy had bigger communities.
We've been conditioned to think more views/subscribers/upvotes is always a positive thing and you should always be aiming for more.
In reality (and imo), communities that have less members more successfully function as actual communities. Toxic conversations are a lot less likely to break out among people who've had a bunch of other conversations before and users who exhibit toxic behavior are socially shunned and shamed if not banned. (I'm massively generalizing here, but I don't want to write a book).
The way I've managed with my group is to just not have the other accounts and tell them that if they want to message me then they can install Signal. It's a bit of a filter as well; If they're not willing to do something as simple as install an app to talk to you, then they were not likely very good friends to begin with (imo).
Roger, I'm convoluting your vector if you know what I mean