Italy and Japan have shrinking populations. We would too, if it weren't for immigration. However our population growth rate is still low, and if it were any lower we'd be facing serious economic and social challenges. Sure, a shrinking population would drop housing prices, but we are far from having so many people there isn't space to fit them. Our real problem is seventy years of public policy aimed at the elimination of "slums" and the prevention of their reemergence.
If you think about it, "slum" is just a derogatory word for a neighborhood with a high concentration of very affordable housing. Basically policy has by design eliminated the most affordable tier of housing, which eliminates downward price pressure on higher tiers of housing. Today in my city a median studio apartment cost $2800; by the old 1/5 of income rule that means you'd need an income of $168k. Of course the rule now is 30% of income, so to afford a studio apartment you need "only" 112k of income. So essentially there is no affordable housing at all in the city, even for young middle class workers. There is, however a glut of *luxury* housing.
In a way, this is what we set out to accomplish: a city where the only concentrations of people allowed are wealthy people. We didn't really think it through; we acted as if poor to middle income people would just disappear. In reality two things happened. First they got pushed further and further into the suburbs, sparking backlash by residents concerned with property values. And a lot of people, even middle-class young people, end up in illegal off-the-book apartments in spaces like old warehouses and industrial spaces.