You don’t have to actively engage with your neighbors, or just do so sparingly. You are in control of your life and (generally) what happens with it.
It does sound like a lot of people you associate with just hang out with you to make themselves feel better by being ‘above’ you in their eyes. I would try to cut these people out of your life and try to find other like-minded people who may understand you and your situation better.
Think of all the classified information the bugs on it can capture!