In my particular case there can only be one reason, realistically.
True; as you've been saying, you live in a fairly secluded part of Maine. If someone's hovering near your home with a huge bunch of land around you, there's probably something a little fishy going on....and that's another aspect that needs to be considered (location, that is).
And hundreds of feet?
Two or two point five hundred is technically "hundreds", although the wording generally implies more. I was thinking in orders of magnitude.
There was someone else that I was discussing something similar with the other day. His perspective was that someone flying near where he lived (a multi-story apartment building adjacent to a busy city street and sidewalk) would be a safety problem. I think that most people imagine a specific place that the theoretical drone is flying and make up their opinion partially based on what they visualize.