I don't think it's the controversy of his guests so much as many of their unqualified ramblings that get treated as expertise. It's really obnoxious that it all gets put into political controversy when it's just often facially stupid BS.
This is like Gell-Man Amnesia, but for the YouTube age. If I can recognize a source is unreliable for my area of expertise, it's more strange to expect that I would trust it in other areas just because I don't have the expertise in that area to directly evaluate all of the claims