Yes, power supplies can absolutely fail due to age. Namely, the electrolytic capacitors that smooth the incoming AC (and in other places smooth the switched secondary voltages) are only rated for a certain number of hours of operation before the electrolyte starts to break down, and when they fail, the electrolyte begins to boil and can build up pressure and then rupture with a bang exactly like you describe.

Temperature and operating time are both factors in how quickly capacitors fail, so the fact that the computer was left on most of the time for 12 years means that the caps were probably running at elevated temperature nearly all the time.

In terms of what makes a "good" power supply, the short answer is that, unless you have an electrical engineering degree, the reputation of the brand and the efficiency rating of the power supply are your best indicators. Big OEMs like Dell are likely trying to cut costs in their desktops, so the power supplies in OEM desktops may actually use cheaper components and worse build quality than the power supplies that enthusiast PC builders use in their rigs.

Many modern power supplies are rated on the "80 plus" rating system-- meaning the power supply is more than 80% efficient, although nowadays many power supplies are more like 90% efficient and may be rated "80+ gold" or "80+ platinum". A more efficient power supply is losing less power as waste heat, so the components run cooler and last longer.

That looks like it is a standard ATX power supply, so almost any power supply on the market should fit. Unfortunately, most 80+ gold PC power supplies would be much more expensive than you probably want, but something like a humble Thermaltake 500W power supply would be 80+ rated, made by a brand people have heard of, and costs around $40. For 5 bucks more, it may be more reliable than some no-name OEM power supply replacement.