Back for the content. I largely agree with and find your arguments about the problem quite compelling.
Here is a novel (I think) though to providing impetus for solving it.
We have a tragedy of the commons here. SF residents are harmed by crazy high housing prices, but locally residents don't want high density housing "in my back yard"
My solution is a revenue neutral tax incentive program: Essentially the city decides the rate it wants to drive new housing starts, and then for each qualifing high density housing project that starts it adjusts the local taxes. It has a model for the impact on surounding foot traffic, road traffic, etc. And it has a model for the region of the city (maybe the whole city) that benefits from the additional housing.
All of this goes into one big equation and some peoples property taxes go up to pay those who agree do have high density in their back yard... their property taxes go down.
I think we would find some pretty creative local business owners and residences who suddenly found it in their interest to figure out how to adjust all the regulations etc. to GET that housing into their back yard.
And if your property taxes are going up because of this, well you need to figure out how to get some housing by you too.
I think certain parts of the city would specialize in becoming super dense... (which means they would have to somehow solve the issues around getting services for the area up to required standards) and doing this would solve our housing crisis.
What I like about it, is it aligns folks around rules and policies that are globally good, not just good for them.
.... and it doesn't cost a cent! (except the overhead of building the traffic models etc. which hopefully the city already has)
--dan