Python coding is practically the only usecase for local for me.
Cloud llm are able to run 1 trillion parameters and have all of python knowledge in a transparent rag that's 100gbit or faster. Of course they'll be the bestest on the block.
But when the new GPT coding benchmarks only barely behind grok 4 or gpt5 with high reasoning.
>Model(s) & size: exact name/version, and quantization (e.g., Q4_K_M).
My most reliable setup is Devstral + openhands. unsloth Q6_K_XL, 85,000 context, flash attention, kcache and vcache quant at Q8.
Second most reliable. GPT-OSS-20B + opencode. Default MXFP4, I can only load up 31,000 context or it fails?(still plenty but hoping this bug gets fixed), you cant use flash attention or kv or v quantization or it becomes dumb as rocks. This harmony stuff is annoying.
Still preliminary, just got working today, but testing is really good. Qwen3-30b-a3b-thinking-2507 + roo code or qwencode, 80,000 context, unsloth q4_k_xl, flash attention, kcache and vcache quant at Q8.
>Runtime/tooling: e.g., Ollama, LM studio, etc.
LM studio. I need vulkan for my setup. rocm is just a pain in the ass. They need to support way more linux distros.
Super useful config dump—thanks. Do you have wall‑clock numbers for prefill/gen tokens/sec and power draw on the 24GB card for those three setups? Also curious where quality starts to degrade vs. context length in your tests.
LLMs in general are going to make C/C++ development viable again for the masses. A big reason everything went to web frameworks was memory safety - even big frameworks like Electron and React Native are memory safe overall. Writing low level code by hand was a migraine experience, even for experienced devs. AI changes this relationship entirely, even if you just use it as a pair programmer. I've had even small 8b models correctly call out memory unsafe code and suggest fixes. Larger models are even better.
Local LLMs? A new renaissance. All that power without having to pinky swear with a cloud provider that they won't just take your generated code and use it for themselves.
Expect to see some awesome Windows and Mac apps being developed in the coming months and years. 100% on device, memory safe, and with a thin resource footprint. The 1990s/2000s are coming back.
I hope so too. I’ve been using claude in some medium sized c projects and tbh its still not excellent. YMMV. Bolting it onto existing projects has been difficult. It doesn’t do a great job of understanding the build graph, nor the dependency relationships between the host system’s libraries (even with coaching, particularly in mac/arm space). It does pretty good at crushing impl code, but kind of the infrastructure and setup pieces are still lagging . I have a counter example on a smaller project, for which I was programming an E ink display, and it was able to understand all of the dependencies quite well, but that’s mainly cause they were localized and not heavily reliant on any sort of system libraries, and the build system was much simpler
The Chinese models like Deepseek R1 and the Qwen Coder series have been way better at coding than any of the western models. And they're open source, too.
32 billion sized R1 and Qwen models blow GPT, Gemini, and Claude out of the water for coding tasks usually. And they can be made to run on a standard gaming PC off the shelf from somewhere like Best Buy or Costco. They will run really really slowly, though, but they will run.
Thanks-this is genuinely encouraging; I'd assumed AI help was strongest on front-end work(web apps/SwiftUI), so this is my first concrete example of an LLM catching memory‑unsafe C/C++-could you share your toolchain (CLI/IDE integration) and model details (name/quant/runtime), and what "awesome" 100% on‑device Windows/Mac apps you most want to see?
I've been using a lot of the Chinese open source models like R1 and the Qwen coder series. I've also been trying some community mods of them from HuggingFace.
I was using a combo of Ollama + Roo Code for the front/back end but Ollama is kind of dumb when it comes to memory overload protections. I've since switched to LM Studio and it has a very annoying hard timeout of 2-3 minutes on its API server. Local isn't perfect right now, but when it does run you can see the potential as clear as day.
Currently I am using llama.cpp for an interactive repl chat. I was previously using Alpaca (a GTK GUI), but was annoyed with how slow it was and some random crashes. I am transitioning some of this to self hosted in the cloud for things that can't run on my laptop.
I am looking to get away from my current interface, and write my own. Mostly for experience of deeply integrating agents into a program. If anyone knows a good library for interacting with a local model that doesn't involve standing up a webserver I am interested :)
My daily driver is gemma3n. Its been a nice balance between speed and performance without spinning up my laptop fans.
I am super interested in local models, partially because there is no friction from managed services, but also because I think as small models become more viable we will see an explosion of apps incorporating them.
Gemma3n as a daily driver sounds nice—4b or 8b? and rough tokens/sec on your laptop? And have you A/B‑tested code generation quality across local models (e.g., Gemma3n vs others)?
I am using the smaller one, specifically the e2b-it flavor.
I get ~20-30 tok/sec. It's fast enough that its not frustrating, but if it were faster you could more easily skim as it generates.
I haven't done any serious testing. My process is typically learning about new models on HN or elsewhere, and trying to give them a real shake. I have some goto code generation prompts that I try on all of them. None succeed but they are getting close. I also do a lot of just feeling it out. The more I can use solutions unedited the better it feels.
Tried running a local LLM and it felt like adopting a pet dragon. Fun at first, but then it keeps eating all my GPU and still refuses to clean up its own context window.
Haha, a cute pet dragon. Two knobs that helped me tame VRAM: KV‑cache quant/eviction and sliding‑window attention (if your runtime supports them). What model/runtime and context are you running when it tips over? Are you using Ollama?
Also log into your Claude/OpenAI dashboard and read the logs. Now they log every damn thing that goes through the API and keep it there for a minimum of 30 days without any option to delete (unless you're enterprise). No anonymization or anything. Just raw audit logs.
Thanks for raising the privacy angle. Do you have a source or plan details for the 30‑day retention and the lack of deletion options (non‑enterprise)? It would help to know account tier and where that policy is documented.
Beyond policy, how are you actually using local LLMs—what tasks do you run locally vs. in the cloud—and which scenarios feel most privacy‑sensitive to you (e.g., proprietary code, contracts, health notes)?
Python coding is practically the only usecase for local for me.
Cloud llm are able to run 1 trillion parameters and have all of python knowledge in a transparent rag that's 100gbit or faster. Of course they'll be the bestest on the block.
But when the new GPT coding benchmarks only barely behind grok 4 or gpt5 with high reasoning.
>Model(s) & size: exact name/version, and quantization (e.g., Q4_K_M).
My most reliable setup is Devstral + openhands. unsloth Q6_K_XL, 85,000 context, flash attention, kcache and vcache quant at Q8.
Second most reliable. GPT-OSS-20B + opencode. Default MXFP4, I can only load up 31,000 context or it fails?(still plenty but hoping this bug gets fixed), you cant use flash attention or kv or v quantization or it becomes dumb as rocks. This harmony stuff is annoying.
Still preliminary, just got working today, but testing is really good. Qwen3-30b-a3b-thinking-2507 + roo code or qwencode, 80,000 context, unsloth q4_k_xl, flash attention, kcache and vcache quant at Q8.
>Runtime/tooling: e.g., Ollama, LM studio, etc.
LM studio. I need vulkan for my setup. rocm is just a pain in the ass. They need to support way more linux distros.
24gb vram.
Super useful config dump—thanks. Do you have wall‑clock numbers for prefill/gen tokens/sec and power draw on the 24GB card for those three setups? Also curious where quality starts to degrade vs. context length in your tests.
LLMs in general are going to make C/C++ development viable again for the masses. A big reason everything went to web frameworks was memory safety - even big frameworks like Electron and React Native are memory safe overall. Writing low level code by hand was a migraine experience, even for experienced devs. AI changes this relationship entirely, even if you just use it as a pair programmer. I've had even small 8b models correctly call out memory unsafe code and suggest fixes. Larger models are even better.
Local LLMs? A new renaissance. All that power without having to pinky swear with a cloud provider that they won't just take your generated code and use it for themselves.
Expect to see some awesome Windows and Mac apps being developed in the coming months and years. 100% on device, memory safe, and with a thin resource footprint. The 1990s/2000s are coming back.
I hope so too. I’ve been using claude in some medium sized c projects and tbh its still not excellent. YMMV. Bolting it onto existing projects has been difficult. It doesn’t do a great job of understanding the build graph, nor the dependency relationships between the host system’s libraries (even with coaching, particularly in mac/arm space). It does pretty good at crushing impl code, but kind of the infrastructure and setup pieces are still lagging . I have a counter example on a smaller project, for which I was programming an E ink display, and it was able to understand all of the dependencies quite well, but that’s mainly cause they were localized and not heavily reliant on any sort of system libraries, and the build system was much simpler
The Chinese models like Deepseek R1 and the Qwen Coder series have been way better at coding than any of the western models. And they're open source, too.
32 billion sized R1 and Qwen models blow GPT, Gemini, and Claude out of the water for coding tasks usually. And they can be made to run on a standard gaming PC off the shelf from somewhere like Best Buy or Costco. They will run really really slowly, though, but they will run.
Thanks-this is genuinely encouraging; I'd assumed AI help was strongest on front-end work(web apps/SwiftUI), so this is my first concrete example of an LLM catching memory‑unsafe C/C++-could you share your toolchain (CLI/IDE integration) and model details (name/quant/runtime), and what "awesome" 100% on‑device Windows/Mac apps you most want to see?
I've been using a lot of the Chinese open source models like R1 and the Qwen coder series. I've also been trying some community mods of them from HuggingFace.
I was using a combo of Ollama + Roo Code for the front/back end but Ollama is kind of dumb when it comes to memory overload protections. I've since switched to LM Studio and it has a very annoying hard timeout of 2-3 minutes on its API server. Local isn't perfect right now, but when it does run you can see the potential as clear as day.
I almost exclusively self host the models I use.
Currently I am using llama.cpp for an interactive repl chat. I was previously using Alpaca (a GTK GUI), but was annoyed with how slow it was and some random crashes. I am transitioning some of this to self hosted in the cloud for things that can't run on my laptop.
I am looking to get away from my current interface, and write my own. Mostly for experience of deeply integrating agents into a program. If anyone knows a good library for interacting with a local model that doesn't involve standing up a webserver I am interested :)
My daily driver is gemma3n. Its been a nice balance between speed and performance without spinning up my laptop fans.
I am super interested in local models, partially because there is no friction from managed services, but also because I think as small models become more viable we will see an explosion of apps incorporating them.
Gemma3n as a daily driver sounds nice—4b or 8b? and rough tokens/sec on your laptop? And have you A/B‑tested code generation quality across local models (e.g., Gemma3n vs others)?
I am using the smaller one, specifically the e2b-it flavor.
I get ~20-30 tok/sec. It's fast enough that its not frustrating, but if it were faster you could more easily skim as it generates.
I haven't done any serious testing. My process is typically learning about new models on HN or elsewhere, and trying to give them a real shake. I have some goto code generation prompts that I try on all of them. None succeed but they are getting close. I also do a lot of just feeling it out. The more I can use solutions unedited the better it feels.
Tried running a local LLM and it felt like adopting a pet dragon. Fun at first, but then it keeps eating all my GPU and still refuses to clean up its own context window.
Haha, a cute pet dragon. Two knobs that helped me tame VRAM: KV‑cache quant/eviction and sliding‑window attention (if your runtime supports them). What model/runtime and context are you running when it tips over? Are you using Ollama?
Restrictive privacy laws.
Also log into your Claude/OpenAI dashboard and read the logs. Now they log every damn thing that goes through the API and keep it there for a minimum of 30 days without any option to delete (unless you're enterprise). No anonymization or anything. Just raw audit logs.
Thanks for raising the privacy angle. Do you have a source or plan details for the 30‑day retention and the lack of deletion options (non‑enterprise)? It would help to know account tier and where that policy is documented.
Beyond policy, how are you actually using local LLMs—what tasks do you run locally vs. in the cloud—and which scenarios feel most privacy‑sensitive to you (e.g., proprietary code, contracts, health notes)?
qwen3-coder for programming help as I'm not very good at css nor flask. sometimes I test other models just for fun, but only for coding or sysadmin.
[dead]