Is your trivial pleasure indulgence more important than the lives and freedom taken away from animals? It's one thing to think your life is more important than theirs (speciesism), but it's quite another to believe your pleasure matters more than their lives. This is an inbalanced consideration of interests - their interests to not have those things done to them by humans are more important to them than your addiction/habit and greedy desire to have very specific, privileged and unnecessary products for which you have plenty of alternatives. I'm presuming you don't think pleasure is a justification for victimizing humans or dogs, so there's still a double standard of discrimination happening based on species - despite them all being sentient beings with similar foundational interests (e.g. to live, be free, be happy, be with loved ones, avoid danger, not be harmed or exploited or killed, etc). Heck, they want pleasure too. Why does their pleasure not matter? And what you take from them is much more fundamental and valued than any kind of pleasure.

Moreover, do you think you can't experience pleasure without consuming animals & their secretions? Beyond all the other non-food-related ways of entertaining yourself, there are plenty of tasty plant based foods, much of animal flesh is seasoned with plants anyway (which we can use to season other plant based foods), and there are products designed to replicate the same exact experiences of consuming animal products. It's simply a lie to think that being vegan means giving up pleasure - many vegans can attest that vegan food is not only tasty, it's tastier (and makes you feel better health wise, and in terms of happiness and lifting the burden of guilt toward animals and the environment off your conscience), and it inspires you to expand your culinary horizons and experience many more foods than you did before, since there are so many more plants and plant based foods/recipes than the few dead animal species and their secretions that most humans eat. "It takes nothing away from a human to be kind to an animal." - Joaquin Phoenix, actor and vegan / animal rights and environmental advocate. However it should be noted that pleasure never justifies victimization anyway, and also that veganism is not a diet but an ethical stance against animal exploitation (beyond animal-derived food, including avoiding supporting/contributing to/participating in animal-derived clothing, any other commodity made from animals, or other industries, services or activities that involve animal exploitation).

Anti vegans are aggressively against the idea of veganism in general, not just opposed to going vegan themself (which is probably most people, who aren't necessarily "anti vegan" - though perhaps anti-animal rights, if they knew what it means). So, not personally wanting to go vegan doesn't really explain being anti vegan, unless it's an extreme reaction and psychological coping mechanism to feel the need to reactionarily lash out at the people who make you feel guilty over exploiting animals (the "meat paradox" in psychology).

Vegans are not trying to take away anyone's pleasure - we aren't puritans or asceticists (inherently, of course a vegan could be separately). Veganism as a philosophy is in favor of respecting animals, not against experiencing pleasure - generally, even the same kinds of pleasure that come from consuming or using animals, which is why we often recommend plant-based replication/substitution products that aim to provide the same experience as animal products (for people who desire them, as an alternative means of persuading them to stop using real animals - they of course aren't necessary and many vegans don't have them).

Also, some vegans don't even do activism or advocate for animals/veganism, they're just vegan, so it would make more sense if anything to be anti-animal rights activism, but even then, activists aren't trying to (and have no ability to) take anything away from you, including the animal products you consume - only attempting to reason with you and inspire you to stop doing it. Even if someone screamed in your face and told you that you were a murderer and rapist and enslaver of animals (referencing contributing indirectly to animal slaughter, exploitation, and forceful impregnation/artificial insemination) and that you need to stop (while much of activism is less confrontational), let me know when they're actually stealing animal products away from you. They don't and they can't, and they almost never are violent either, so it's just words and emotion directed at the issue you partake in. It's valid freedom of speech, expression and protest, and if anything anti-vegans are trying to take away those rights when they argue people like Tash Peterson should be prosecuted or jailed just for yelling in McDonalds stores dressed as a cow, or agree with the massive fine she received for criticizing a veterinarian for being a hypocrite by eating animals (which has been contested under free speech protections).