I don't doubt that locally hostable models will be important for agentic tasks but I suspect they are still going to be bigger than most can comfortably host for time being. I still think there will be a place for the super large models for more complex reasoning although how much will be due to the intrinsic knowledge in the weights and how much due to the plumbing around them remains too be seen.

And if course LLM's are not going to be the end point of the search for AGI. Whatever their architecture they will still need copious amounts of compute.

but I suspect they are still going to be bigger than most can comfortably host for time being

ARM and CXMT both have the potential to change this situation rather dramatically.

ARM: The RAM/VRAM divide is kind of a legacy thing at this point, doing more harm than good in the x86 (desktop OS) space, but ARM doesn’t have the same baggage.

CXMT: We know the DRAM cartel have previously engaged in price-fixing, and the current shortage looks suspiciously similar to their old behavior. When confronted by a new challenger, they might be forced to actually compete.

SLM's are still trained on massive datacenters as large models and then quantized down. But yes for inference there's hope in the future

Training happens once per model, but inference is an ongoing process. So, there's going to be a huge amount of energy saving if we move to using local models.

Alibaba just announced a chip specifically for running local models. We'll see what it ends up going for. https://wccftech.com/alibabas-tsmc-built-5nm-risc-v-chip-xuantie-c950-now-runs-qwen-3-8-27b-model-natively-unlocking-massive-vertical-integration-tailwinds/

he RAM/VRAM divide is kind of a legacy thing at this point, doing more harm than good in the x86

That has nothing to do with x86 though...

Hence “x86 (desktop OS) space”. It’s not intrinsically part of x86, but it has settled in as a conventional piece of x86 desktop OSes. x86 consoles and ARM desktops don’t assume the same principle, and they can get a lot more mileage out of SoCs as a result.

I think this all hinges on whether or not progress will slow down. For example I've heard of some companies etching AI models onto silicon, but nobody will buy those if the model is obsolete in a year. But if the models stop improving, then these chips might be worth the investment since they will be way more efficient for inference.

So that's one of the biggest questions in AI right now. Are we going to hit a wall? It does kind of seem like the big models aren't improving as much anymore, and the small models are catching up. But at the same time, Moore's law has been going for way longer than people expected, maybe AI will be the same.

Edit: wording on last sentence of first paragraph

I wouldn't say it's totally legacy. (v)ram bandwidth does matter and while the Apple M-series chips do well with their unified memory architecture don't forget it's fixed because it's part of the CPU chip.

Aren't there arm workstations that still take pcie cards and have a similar architecture to x86?

One of my first posts here was asking how to make use of local LLM the best. People said to buy the most beefy laptop within my budget.

Is that really the most reasonable take?

Yes. The issue is not that ARM systems don’t/can’t communicate with a dGPU over PCIe. It’s that x86 systems (edit: running conventional x86 desktop OSes) have to basically pretend they’re using PCIe to communicate with an iGPU. (Not literally, but a lot of the primitives they use were inherited from AGP/PCI and don’t make a lot of sense for an SoC.)

I expect doing ASICs for models will work even if they keep improving. It'll be like regular chips getting new versions. You buy a chip with a specific model etched into it, and if it does what you need great. Next year, a new version comes out. So, it's actually a feature since it allows companies to keep selling new chips.

It does look like we are entering diminishing returns territory though. The biggest evidence for this is that Chinese companies have now basically caught up to Anthropic and OpenAI. If the progress at the frontier was still happening at the same rate, then the gap wouldn't be closing so quickly. There's also a lot less noticeable difference between stuff like Claude 4.6 and Claude 5. When they went from 3.x to 4.x it was very noticeable. And at least for agentic coding, most of the improvement seems to come from the harness now. I expect improvements will continue, but at a much more gradual pace. It's also possible people will figure out a new architecture that's superior to LLMs, or works with them. World models are one promising area already being explored.

If you absolutely need one now, a mac is probably the cheapest way to run them because of the unified memory. With any x86 solution you have to get a separate video card with at least 32gb vram to run a decent local model. However, if you wait a bit then you can probably get a dedicated chip a lot cheaper in the near future https://wccftech.com/alibabas-tsmc-built-5nm-risc-v-chip-xuantie-c950-now-runs-qwen-3-8-27b-model-natively-unlocking-massive-vertical-integration-tailwinds/

What about those small prebuilt workstations with Strix/Gorgon Halo chips and big ram pools? Chinese OEMs like Beeline and GMKTec are offering some sleek little boxes. I guess it still technically not unified on a system level, but you allot most of it to VRAM in the BIOS regardless. You can probably get comparable performance with a fraction of the cost.

Possibly, I haven't looked at how easy it is to get your hands on one of those.