Even with 3 kids, you do not need 5000 square feet of house.
That actually depends a lot on the people involved. As a single person with no kids, I'm finding ~1,800 square feet to be woefully insufficient for my needs. When I do woodworking projects, I have to either lose my master bathroom or my kitchen, and sometimes both, because I don't have a proper wood shop. My drum kit, weight bench, and grand piano take up almost the entire living room, so I don't have a proper living room. My filing cabinet, sewing table, printer table (for a printer the size of an office refrigerator), and Christmas tree (too big for any of my closets) ensure that I don't have a proper dining room, either—just the breakfast nook in the kitchen. My bedroom has essentially no available floor space, between the bed, dresser, two bedside tables, and a treadmill, with the exception of the main walking path into the room. My guest bedroom is completely filled with the bed, one tall dresser, and a small bedside table. I'd like to add another bookcase, but there's no space left in my house unless I put it on the front porch.
Basically, I've done the math, and to be comfortable, I need a minimum of 400 square feet for a wood shop, 200 square feet of closet space for the tree, at least 200 square feet of dedicated space for exercise equipment, and ideally a music room with another 400 square feet or so, for a total of 1,200 additional square feet, or 3,000 square feet total. If I got married, and if my wife had similarly space-demanding hobbies, and if we had three kids, even after you deduplicate the kitchen, bathrooms, and shared living space, we would still need well over 6,000 square feet. So 5,000 square feet for a family of 5 is not excessive. If anything, I'd find it kind of cramped.
Get a smaller, comfortable place to live and put you money somewhere more useful. If you're married, obviously your spouse needs to agree with this...
Or move to a location where you can afford a decent-sized house. Five thousand square feet is not a mansion. I believe that the current lower bound is 8,000 square feet.