I agree with alot of what you are saying and I think making something like this while gray ethically is miles more ethical than some of the current research going into brain organoid based computation or other crazy immoral stuff.

With regards to agency I disagree, we are reactive creatures that react to the environment, our construction is setup in a way where our senses are constantly prompting us with the environment as the prompt along with our evolutionary programing, and our desire to predict and follow through with actions that are favourable

I think it would be fairly easy to setup a deep learning/ llm/ sae or LCM based model that has the prompt be a constant flow of sensory data from many different customizable sources along with our own programing that would dictate the desired actions and have them be implanted in an implicit manner.

And thus agency would be achieved , I do work in the field and I've been thinking of doing a home experiment to achieve something like this with the use of RAG + designed heuristics that can be expanded by the model based on needs during inference time + local inference time scalability.

Also I recently saw that some of the winners of arc used similar approaches.

For now I'm still trying to get a better gfx card to run it locally 😅

Also wanted to note that most models that are good are multimodal and don't work on text prediction alone..