Because of the CORS settings on Google's servers would tell your browser to not go forward with the request.
There are two ways it could eventually be possible:
By opening the video in a new page/tab that only contains the video, with the YouTube player, which defeats the purpose a bit.
Fair enough, that's interesting. I assume this only applies to the non-web clients. On the web, it would not be possible.
You can verify by looking at the outgoing network requests on this random video for example: https://invidious.privacyredirect.com/watch?v=qKMcKQCQxxI
Invidious and YouTube piped (and LibreTube) by default load the videos server-side, as opposed to GrayJay, NewPipe or Smarttube.
It has advantages (mostly that your IP address is not shared with YouTube, and it allows users from countries where YouTube is blocked to still access it) and inconvenients (much harder to keep up when YouTube actively seeks to block them).
Smarttube next doesn't require rooting the device, it can be sideloaded.
Sideloading is not very complicated. Google is not trying to block any sideloading (at the moment, at least).
Probably around 370 in total, but at any given moment, 3.