I agree with you that words meaning is a thing of context, and what matters the most is what people imply when they use it, generalization usually takes a person to the wrong path; the only real guide for almost everything is the context.
IMO, anti-Semitism is more used to refer to anti-muslim sentiment than anti-jew sentiment, but I guess the location on earth that I walk matters a lot.
A region at the north of south America where Muslim is broadly practiced, not so many Jews here, and the word Islam-phobia isn't popular.