This issue seems split between two camps, the "I can wonder around and film/take pictures of anything as long as you dont suffer any undue harm", and the absolute "get off my lawn!" folks. I think there's validity to both, but I'm siding with the home owners.
1. The property is marked private.
2. Street view drivers may not have seen anything posted, but being reasonably smart individuals (to operate the equipment and drive at the same time), they should have known not to enter an unmarked driveway because usually those lead to private residences.
3. Yes satellite imagery does take away a level of privacy, but those were and mostly still are government run installations who sub license to private industry. There is still a zoom cap in place. Street view breeches that distance limitation.
And the twist, "who owns the images". If the drive is marked private and Google illegally entered, and the took the intellectual property of the home owners, they can and should pay damages. Think of paparazzi and famous people. If the papz get a zoom lens at take pictures of that actor/actress in their private homes, they get the bejesus sued out of them. They regularly lose. But the funding from the images negates the cost of litigation. So they keep doing it. Google may make the same strategic decision, but they still should pay.
And, why fight this? Are they trying to tarnish the Google do "no" evil/Don't be evil image. They should admit to this being a mistake, compensate the home owners in the same range as images of non-celeb/z-listers, and call it a day. The images were already removed so the home owners cant claim they are still being harmed.