I think it’s at least partly related to patriarchy. Same reason bisexual men and bisexual women are generally treated differently. Women are seen as much less of a “threat” to masculinity and generally more socially accepted (albeit more objectified & marginalized in different ways) in the patriarchal system. I think a lot of men use their attraction to women as “proof” to themselves that they’re not gay (and to them gay = feminine = bad), and the existence of bi men undermines their masculine foundation to an extent.

I think the underlying idea with trans people is “of course women would want to be men” whereas “a man who wants to be a woman” is disrupting the patriarchal logic of maleness being more desirable. The idea that a man could actually be a woman (and even take radical steps to affirm that) is more transgressive in a system that highly values male dominance.