The real reason is probably a lot simpler: Cost.
Before you break ground on a single-family home in Pleasanton CA, you must cough up in excess of $125,000 (yes, 125 grand) in fees and permits.
I expect said fees and permits are even more expensive in San Francisco proper.
[For comparison, in Los Angeles County a home building permit is $38,000. Here in Montana it's from $50 to $2000 depending where you are.]