It's always a case of the oppressor using fighting words or direct actions of harassment to get us to lash out and them crying and hiding as the victims while we get brutalized.

Yes.  And some argue that this means "nonviolence" is the only way of beat them at their own game.  Such people are fools who believe that fawning will being them safety even though that never has and never will work.  And in my experience, speaking up about such abuse gets me crucified anyway.  If whatever I do will result in them still doing harm then I figure I may as well harm them back.