This post was also posted yesterday, but yeah, with ChatGPT you can execute random code. It is however in a VM of some sorts, so just trying to delete things won’t do that much.
It doesn’t kill the LLM instance, internally it just calls an API to run the generated code on a machine if you ask for it.
But getting out of the VM will most likely be protected. So you’ll have to find exploits for that as well. (Eg can you get further into the network from that point etc)