Out-breeding depression primarily results in a decrease of fertility and infant mortality. So although it is occasionally observed in surviving offspring, in general it is much lower probability.
Also the neandertal crossing was deleterious it would be much lower percentage in modern humans. It also would not have come from multiple crossing events.
Just to preface, I am a plant breeder/geneticist with extensive experience in interspecific crosses in multiple species - all practical. This discussion should definitely be finished over some a few drinks.