I went through a handful of devices, and none of them ticked off all the boxes for me.
Now I use a Unihertz Jelly Star. It's tiny, it has Bluetooth, Wifi, expandable memory, headphone jack, and in a pinch it's also a phone with a camera and flashlight and so forth. The battery charges very fast, and it's got a halfway decent external speaker. More dedicated audiophiles than me would have to weigh in on if it's pumping out audio signal in all the right bass and treble frequencies at appropriate levels, but it does everything I wanted from a digital audio player.
You can't bring the time travel souvenir back with you in the time machine. You've got to store it in a safe cave or storage unit or something so that it'll be at least plausibly aged by the time it reaches the future, even if surprisingly well-preserved.