So far as I know, if your device uses their Gemini Nano LLM, it doesn't reach back to their servers at all unless you OPT IN to the 'Help service inprove'.
This feature though and a few other calling features has made me switch from iPhone single handedly, I was receiving 6-10 spam calls a day, now I see none because they're screened in the background. It's fantastic. I'm hooked on these Pixel features and only hope more move to becoming on device features with the ability to opt in to sending certain things off device.
Home Assistant is such an awesome tool. I use it every day and shamefully have it linked to my Google Home so Gemini can turn on and off devices when prompted. Aside from that, I could just go the route of setting up a local LLM on my server and having Home Assistant be my new assistant on device so it doesn't use Google at all.
I definitely recommend Home Assistant though, between the iPhone users and now myself on Android in our home, it makes everything appear native to the end user. Now I just use Zigbee and Zwave devices for everything since they're more reliable and much cheaper.