"Invasive species" is a technical ecology term used to describe plants and animals that face no predation in a given environment and have a negative effect on the other species native to that environment. Like many other scientific terms, it was misused by people without the technical understanding.

The crux of your argument is species drift compared to forced mass introduction, but there's a huge difference between them. It's similar to the difference in outcomes that can occur when somebody surfaces after deep diving. Go slow enough, and the systems have an opportunity to acclimate to the changes and respond. Go quickly enough, and things break and start impacting systems around them in a cascade. That's the thing that gets missed every time I see this sentiment.