You just put yourself at risk for losing your house over a few percentage points. You may have great resources but you just never know what may happen that may prevent you from paying in the future. I mean, look around - people who were making good money a few years back are losing their homes left and right.
Personally, I think you should have bought the house outright, then taken the projected cost of the mortgage and deposit it into a range of mutual funds. That way, you would have eliminated the risk. And hell - with a paid for home - you won't feel pressured to stay in a job that sucks (if it ever goes south).