I don't agree that my presence within a boundary drawn by other people constitutes consent. I'm in a spot that your government claims, but the claim is not legitimate and does not rest on any legitimate principle that makes it "theirs" and not "mine." I don't consent. I agree with you as far as raising children, but your analogy fails because I am not in your house. I am in my house. You and I did not partner together to buy two houses and form an association where we each have some say in what goes on in each other's houses.
Refusing to vote is just as much a part of the system as casting a ballot
I'm glad to hear that. Then my decision ought to be praised, as well as my efforts to persuade others to also withdraw their consent.
it just lowers your stress
I wish. If you think being blue in a red state, or red in a blue state is hard, try being the guy who doesn't raise his hand at the office when the VP says "So, who voted today?"
until it's too late to act.
It essentially already is. But there's some faint hope if we can get a critical mass of people to believe in everybody having the freedom to withdraw consent, and nobody having the right to force others to say pledges, fund wars they don't believe in, adhere to religious principles they don't believe in, etc.