I'll give LM studio a go, thanks.

Sure they are. Especially with moores law dead. And especially since 2020; the more million dollars behind it, the more they run like one from a inexperienced single-dev on itch.

How can you run a local model for better coding support than fable/opus?

Not sure what Moore's law has to do with anything here to be honest. The models you can run locally on a consumer desktop can do real work, and their resource usage is no different from any other software like games that you'd run.

I run Qwen 3.6 27B quantized down to Q4_K_M or even a variant Q5_K_S on my 8gb VRAM entry level AMD GPU, with support of my CPU and 32gb system RAM. Yes, I also limit the Context Length heavily to something like 18k. It's slow. But the point is, you don't need necessarily 16gb VRAM.

But it's better to use faster models for this type of hardware anyway. The MoE type of models (such as 26B A4B, or 35B A3B) are vastly, vastly faster for normal usage, but they are a bit worse in some cases. Also Googles QAT trained model versions also have less RAM requirements without losing much quality.

I'm just saying that, so others are not discouraged too much. It's not the same experience as the full version off course, but you can use them with some tricks on weak hardware too.

I'm fairly optimistic that people will figure out how to optimize the models a lot further going forward. One obvious path is to try and separate the reasoning network from the trivia that gets baked into the model, and some work is being done in this area. If you could have a context free reasoning engine and then feed the facts it needs to know on the fly based on the context you're running it in, then you could likely have a much smaller model that's very capable.

You can't right now because there is no dataset available for fine tuning. I'm just saying that it's possible to fine tune 8B-35B parameter model in int4 that will outperform those models ex. for single programming language and developer specific problems. You can read example of fine tuned 8B model here https://fermisense.com/when-machines-take-the-wheel/

This seems like a great time to mention what I built and hosted months ago. https://masland.tech/ai-efficiency-index/

The AI Efficiency Index tracks the cost/intelligence mix. Low intelligence is useless. Capability scales exponentially. Expensive ≠ better value. MiMo-V2.5 is the leader.

If you have any questions feel free to ask. This uses the Artificial Analysis leaderboard. They have "cost per task" which I think is weird.

according to performance on standard benchmark. somewhat covered by the controversy surrounding the term: benchmaxing.
if you see all benefit as a linear one dimensional height on a bar graph..
its almost like you assume that the previous model gave the same exact answer(same style) and the new mode gave the same exact answer PLUS additional useful information.
it might be convenient if measuring progress was so simple. but unfortunately/fortunately , it is not so simple .
the most important benchmark are the comparison of outcomes on the problems that YOU have & prompts that YOU can(will) write. nothing else matters for YOU.

• i admit benchmarks are well designed to objectively measure competence on challenging problems that require skill and really need only ONE correct answer.

Sure, a benchmark doesn't capture all the subtleties and different use cases, but it does give a general idea of the capabilities of a model. Obviously, you have to run the model and see if it does what you need. But the chart isn't really about the nuance, it's showing how drastically the efficiency of the models has improved in just a year. The fact that we can even reasonably compare a model you can run on a desktop to one that needed a data center just a year ago is phenomenal.

How do you get into this? Any articles you can share? OP says you can run on your desktop... How? Doesn't this stuff require huge data centers?

Running model that is good at everything require huge amount of energy and huge data center. Those models are mixture of experts. Latest Kimi K3 have 896 experts. Imagine you have company with 896 employees. Each question involves 16 employees to figure out what to do in what area of your business. Like a brainstorm to solve problem. Now if you know exactly what you want and in which area you actually need only 1-5 people. Like an agile team instead of all those people that you have. So you can hire just couple Kimi K3 experts. 16 experts are 100B parameters so roughly 1 expert in frontier open source model is 6B parameters. 5 experts is 30B parameters. You can run 27B Qwen 3.6 quantized into int4 on your computer like other people are doing right now.

I posted link below to example where they fine tuned model ( take it like a employee training ) for specific task.

A P40 with 24GB is ~150€, a V100 (32GB) is ~600€. Both of these fit Qwen3.6 27B (The P40 is about 3x slower though). The V100 even fits 400k context with a Q4 KV-Cache , which means you can have two slots for parallel processing (llama-cpp). You don't even have to use system memory.
One of my inference servers is running with 8gb of DDR3 and a 2nd Gen i7, so my old hardware has a good use again.

thanks, good stuff! sadly, got none of these locally available.

I had to get mine from ebay and wait a couple weeks as they came from China too. Also I recommend having a 3D-Printer and some Blower-Fans on hand as you will either have to buy or print your own fan shroud for these server cards.

That's why you couple it with your own, self-hosted yacy or searxng instance. Embedded world knowledge does not help a model if it becomes outdated. I just let my agent research, embed that knowledge to a little Qdrant server, so other servers are not bothered again and pull the information from there when needed again. With a little RAG you can have GPT at home.

i see what your saying. i didnt mean to discredit standard benchmarks entirely.
i guess its obvious that it measures capability regardless of imprecision.
2 major proposed changes:
**first, i dont really know. aside from saying "benchmark your own prompt+usecase"
a proposed plan:

• approach one: pay attention and credit new or improved architecture designs and research.
• approach two: spend more attention on benchmarks. especially specific benchmarks ( that are not focused with industrial domain tasks.) **domain task pursuit, is useful!.. but it depends on if your interest align to popular domains.
• approach three: if willing to utilize remotely hosted models. rating should also take in consideration.. tools and everything else: websearch performance, RAG performance, smooth interface, pref/balance between speed vs comprehensiveness, cost (if relevant), etc.. .

Honestly, I think the most reasonable approach is just to see what other people's experience is like and which models are well regarded, then try them out and see which one is the best fit for what you're doing. You might not even need the top performing one necessarily, and speed or lower resource usage might be a bigger factor.

An important caveat to that explanation is that the mixture of experts gets re-evaluated for every single token in the input sequence and not for high level tasks as in the example. There will be certain "experts" for looking at indentation tokens, or ones that look at specific word beginnings/prefixes etc.

Is it still telling people to put glue on their pizza and convincing teenagers to kill themselves?