How so you get useful government issued ID cards there? The only argument that makes sense to me here is that making an government issued ID mandatory hurts the lower/working classes who can't afford to take a day off work to get their photo taken and info recorded for the ID to be issued.
Not to mention they have expiration dates as people change over time (being issued one at birth won't necessarily be accurate later if it has a photo or address or anything that, say, would be required for voting. I mean, we've got Birth Certificates. And we all get government issued Social Security numbers and cards. The latter isn't supposed to be used for ID as it's just a 9 digit number that will be reused. (but places use it anyway)
People also don't really want everything tied to one 9 digit number. (Why Drivers licenses are better, IDK) tracking individuals is kinda anti-freedom. That said, all of our info has been leaked so many times by so many state sponsored, public and private businesses at this point that it's all just theater anyway.
Honestly the only phobia to passports I've heard is that they’re not cheap and require forms/work to obtain. I've got one, but if you're just scraping by, you're not going to want to drop what equates to like a week and a half of federal minimum wage earnings to get a document you might never use.