Brass brush on a drill would probably melt into plastic fast. But a handheld one would work ok, or steel if you need more persuasion and aren’t worried about scratches.

What’s under the caulk? If it’s ceramic tile, you can probably get away with a brass cup brush in a drill. Test in an inconspicuous area first obviously, but it should be less likely to scuff up the surface as something like scotchbrite.