We have vastly increased our ability to produce food.

And it has been largely the (petro)chemical industry responsible for this. The Haber-Bosch process transformed agriculture, but accounts for percent-level quantities of global energy consumption and carbon emissions. And it requires raw materials that are typically produced from hydrocarbons (although admittedly there are renewable options). And other nutrients typically come from mining (even organic options) - which displaces many species of all sorts. And this does not account for pesticides, etc., that others have mentioned.

Prior to the development of modern chemistry, our best sources of fertilizer were often animal manures - which require breeding, raising, and ultimately usually killing animals.

Sure, there is a lot we can do to minimize harm, and generally we should, and I try to myself as much as possible. But I'm not fooling myself into thinking that eating vegan or growing my food organically means nothing or no one suffered. Until we all go back to pre-agrarian societies, we will continue to cause large-scale destruction in some way. But of course this in itself would cause massive population decline and resultant suffering in humans.