I dunno what country you're in, but in my country you are required by law to have a valid reason to reject a job candidate. That reason can be pretty simple, such as "your application was not as strong as other candidates" but you need to be able to back that claim up if you're challenged (and you can be challenged on it).
The recommended approach is to have a list of selection criteria, and carefully consider each one then write it down and keep a record of the decision for a while, incase you end up on the wrong end of a discrimination lawsuit. Candidates have the right to ask why they were unsuccessful (and they should ask - to find out what they can do better to improve their chances next time. As a hiring manager I would note down anyone who asks and consider offering them a job in the future, bypassing the normal recruitment process).
I rank each criteria from one to ten, then disregard the worst scoring candidates until I have a short list that I can compare directly (at that point, I wouldn't worry too much about numbers. You are allowed to say "you were a great candidate, but we had multiple great candidates and had to pick one. Sorry".
If your selection criteria includes "they need to wear nice clothes" then you're treading on very dangerous territory and could be breaking the law. The damages here are commonly six months pay at the salary of the position they applied for, and can also include a court order for you not to be involved in the hiring process going forward.
It's perfectly reasonable to require someone to dress well if they have a customer facing role... but that requirement should be implemented at work and not during the job interview. I'm well aware that a lot of hiring managers rely heavily on these things to make their decision but they should not be doing that. It's not as bad as picking someone because they're a straight white male candidate (which is also very common), but it's still a bad policy.
Australia. It's not clearly illegal but it's dangerous territory. Candidates have a general right to be treated as equals and you need to reject someone for reasons that are relevant to the job position.
Something that can easily be changed, like a shirt, might not be OK. ANZ bank (a massive bank with several hundred billion dollars in assets they manage), for example, requires customer facing staff to wear a branded uniform but back at the office? You can wear whatever you want. When they changed their dress code years ago to no-longer require a suit/tie the CEO deliberately wore ugly clothes for a while to set an example.
Obviously no candidates are expected to turn up to an interview in their uniform - they don't have a uniform yet. And if someone can wear a Marilyn Manson shirt in the office, then why not also at the interview?
The bank I'm with is even more relaxed - even customer facing staff can wear anything they want. Sure, if it's offensive they'll be told to wear something else, but that's a conversation I'd be having with the candidate rather than a reason to reject their application. I might reject them if I don't like their response.