I feel like the difference is not that big, though.
But the difference is massive. Telegram, because E2EE does not work for the majority of its use cases, is hoarding tons of CSAM and other illegal content. This isn't just about the "criminals" who are adding illegal content, its about Telegram's access and hoarding of this data.
On the other hand, Signal is simply a transport vehicle for data. No illegal content is stored or accessible by Signal, its developers or anyone who may gain access to their infrastructure - the complete opposite of the situation over at Telegram. Signal cannot be implied to be storing illegal content because they simply don't store any content. Law enforcement can ask Signal to provide all the data they have on specific users, and they have, but the only data they have is when you created your account and the last day (not time) a client pinged their servers.
That's outdated information:
Go forth and contribute, fork, or create your own.
This link has existed forever at this point if we count in internet years: https://signal.org/android/apk/ - getting an app directly from the developer with no middleman is about as distant as you can get from Google's app store.