Do people legally have privacy in an uncovered yard? I don't think they do. I'm talk about legal, not rudeness.
In the book The Law (In Plain English) For Photographers (ISBN 978-1-58115-712-3), attorney Tad Williams discusses the right of privacy as it applies to photographers. Two cases in point are mentioned: Dietemann v. Time, Inc. (284 F. Supp. 925, 1968) and Galella v. Onassis (487 F.2d 986, 1973). Those are the two cases most often cited as examples of the tort of "intrusion on one's seclusion", and are the basis of the doctrine of "reasonable expectation of privacy on one's own property". (I leave the review of those cases as an exercise for the student.)
The general rule of thumb for photographers is that if it can be seen from a public place, it can be photographed from a public place, UNLESS the subject being photographed is on their private property. Keeping in mind that anyone can be sued for anything at any time, it's best that a quadcopter operator err on the side of caution and make sure to NOT fly their aircraft in a manner that could be construed as attempting to make photographs of persons on private property without consent.
Of course, it may require a few people having their expensive quadcopters blown out of the air by a well-placed shotgun round to get that message across.
DISCLAIMER: I am not an attorney and am not qualified to give legal advice. Consult a licensed attorney with experience in the subject matter for definitive legal advice regarding a particular situation. I am, however, a photographer, and make it a point to keep up with laws and ordinances that affect my enjoyment of the hobby of photography.