Restricting access to files within a user is why sandboxing is useful. It in theory limits the scope of a vulnerability in an app to only the files it can read (unless there is a sandbox escape). Android instead prevents apps from accessing other apps' files by having each app run as a separate user.

One way to keep the encryption keys encrypted at rest is to require the login password (or another password) to open the app, and use it to encrypt the keys. That said, if an adversary can read Signal's data, they can almost certainly just replace Signal with a password-stealing version.