It's ridiculous to compare this to cars. The size category is .5 to 55 pounds. A 55 pound car is called a 'toy' and given to five year olds. You should have just as much right to fly a tiny drone as you have to fly a kite.
If it is ridiculous to compare drones to cars, it is also ridiculous to compare drones to kites. All three have different risk profiles. To have something that could weigh the same as a bowling ball fall on you is different than a kite on the end of a string. 55 pounds isn't tiny in this context.
That being said, cars are clearly the most dangerous of the three. But they are what we are most dependent on as well.
If you want to keep people out of the places cars are actually supposed to go, but let them be on sidewalks and such, then it's fine to have some restrictions. And likewise let drones fly under 400 feet.
But you shouldn't be banning people from the roads either, that's been a big problem making cities worse over the last century.