If tomorrow we banned non-self-driving (NSD) cars, sure. But in most countries, grandfathering in old cars is going to happen for a while. Which means that self-driving and non-self-driving cars will have to share the road.
I could see some transitions possibly. For example, on a 4-lane highway: “In 2027, lane 1 will be separated by a barrier and only allow SD cars. Lanes 2-4 will be for NSD cars only. In 2029, lanes 1-2 for SD. By 2033, NSD cars will be banned on this highway.”
But then they might be kidnapped by a bulging, beastly man in a blue leisure suit!