For a fraction of the price of 96GB vram, I built a desktop based on a supermicro server mobo and EPYC 9 series CPU, with just under 400GB rdimm ram (approx $4500 all in but this was before the ram price hike). Works really well for serving larger local modals at a decent enough speed (I consider anything more than 10 tokens/second usable and value accuracy over speed).
Yes, if your trust model allows you to use API providers or the big 3, you 100% should. They have better util than anything you self host, so they can be more efficient. On top of that, they're shoveling cash into the fire to try to capture marketshare, so they're offering inference for well below break-even costs.
The main reasons to use local models are:
1. Self-sovereignty & control
2. Data security
3. Offline availability
If none of those apply to you, then you should just use OpenAI or Anthropic.
