Tax payers wouldn't have to pay for anything, the only impact would be potentially on credit ratings of the government and possibly also certain interest rates in a minimal way. You can create market rate housing with government subsidized loans that don't come from tax coffers. Hell, the US federal reserve is technically the backer of hundreds of billions worth of mortgages.