So there are a zillion of these, including ones that do what yours does. But that being said, I'm going to always upvote every single one of them because this is better and more important that what the big companies are doing across the board.
Thank you! Would love to hear which ones you've used. Working with local files without anything going to the cloud was the main thing I wanted. I tried Jan, OpenWork, Chatbox and Lobe, and couldn't get that working well in any of them. If there's one I missed it would be great to check it out!!
Have you tried Open WebUI? It doesn't determine which model works best for your hardware, but it otherwise has a very large feature set with a large community, and runs offline by default. Geared more for technical users, however.
It feels like the last mile is so long in this space. I'm a SWE with a ton of experience and I still haven't gotten over the decision paralysis of figuring out which model to run locally, and which inference platform should I use, and will my M1 Macbook Pro actually run anything well enough to be useful, and on and on.
It's kind of embarrassing to admit, but having something Just Work(tm) makes getting started so much easier.
It’s a huge new subject area and it’s completely exhausting to learn. I still feel like a total novice.
I have an M1 Max (more memory bandwidth) so I am not sure how much advice I can give about models, but I would suggest maybe trying the new LFM2.5-24B-A2B. It’s surprisingly good. If you want to mess about with multimodal work, Gemma 4 12B is really impressive. (I mostly use Muse Glimmer but it might be rather slow on an M1 Pro)
If you want to have the space to learn the terminology, tinker with what you can control etc., then you have three easy options: LM Studio, Unsloth Desktop or the llama.cpp native Mac app. But don’t get put off by the llama.cpp command line; most of these tools are running it behind the scenes, and most entries on Hugging Face have recommended parameters.
MLX is an option but so far I haven’t found it to have an edge over simple GGUF on Metal on the M1 at least; it seems it might have more advantages on the later M series chips, but it is usually slower for me.
I have explored running OpenCode inside a VM (you can “remote into it” from the OpenCode GUi desktop app). You might want to look at Paseo, which can do a similar thing but with many more coding agents.
I am quite intrigued by Osaurus; on macOS 26 and up it looks like it will manage a container for you.
You might also want to look at Orchard. Not done this myself yet but I can see the appeal.
I am an experienced web dev in my fifties. Six months ago everything I have written above would have been total gibberish to me. Learning some of the open weights stuff has made me feel much less helpless than I think I might if I just developed a Claude dependency. Nothing matters more to me than grasping the fundamentals.
I still don’t really use LLMs for hands-off coding. But I have found that a local LLM can be a useful sidekick; even at the most basic level it’s a better Google that remembers the context of queries.
Thanks for taking the time to write all this up, I'm checking out the projects you mentioned now.
I feel you on the tinkering-to-feel-less-overwhelmed front. I've been through a few ground shifts in my career so far, and even a little bit of "Oh, so that's how it works, ok..." can make it feel so much less daunting.
Mostly I think my own approach has been to try to combat my anxiety and exasperation about all of this by taking this measure of control.
While no really small local model comes super-close to a frontier model, the shape of the technology is settling and I think one can learn so much by approaching it the conventional nerdy way: what's a practical setup I can run on kit I already have? What can I build that helps me explore the problems?
I am quite cynical about AI still, including for generative coding, but I am not cynical about what I can run locally. There is no doubt that they are already helpful to me even in a minimal form.
And the really cool thing about being able to run them locally is to understand the tradeoffs.
Fwiw, I'm liking the koboldcpp "ecosystem", including koboldlite; basically it's letting me try a little of everything, though there may be something of a "jack of all trades master of none" thing going on. But it has these nice little presets for, e.g. llms, voice, music, images, a "chatgpt"-esque interface, etc.
Haven't tried it yet, but I like that you shared your process in such an honest way, and kept it no frills, as in no LLM wall-of-text readme in the repo. Wish we could see that more often these days.
Looks great. One thing I am missing is the ability to change the installation directory for the models. For space reasons, I want them to be on a different partition.
Linux. I saw you support HOB_USER_DATA, which is great. But I want my config to remain in .config/Hob and only move the models (or whatever else needs lots of space) to another disk/directory.
Edit: Actually I think HOB_USER_DATA is enough for my use case.
I did not enjoy that there was no way for it to clean up the 18GB of models it installed after I determined it was not really usable for any real work.
It's still pretty tricky to do this as a solo dev to keep feature parity in my experience (fun fact, even Claude's own Desktop app is made in Electron).
That being said, majority of the heavy LLM lifting happens natively with llama.cpp :)
Here's roughly what it picks today from my testing:
- 8 GB Mac (M1/M2): Gemma 4 E2B
- 16 to 18 GB Mac: Gemma 4 E4B
- 24 to 64 GB Mac: Gemma 4 26B A4B
- 128 GB Mac Studio: gpt-oss 120B
- Windows/Linux with an RTX 3060 or 4070 (12 GB) and 32 GB RAM: Gemma 4 26B, split between the GPU and system memory
- RTX 4090: Gemma 4 26B fully on the GPU
- No GPU, 16 GB RAM: a compressed Gemma 4 E4B on the CPU
High level, it looks at your memory and memory bandwidth and then picks the smartest one that stays above about 15 tokens a second with room for a decent context.
Interesting, thanks for sharing this. I have some questions? I have a Macbook M5 with 16GB unified memory. What Model would you suggest I use with this. I want something that would replace dot or grokbot , do posting, some marketing work, and email handling. Not coding. What would you suggest?
With the raise of AI, I believe: open-source and local software's will be on the rise, but with the open-source environment there is a real security issues that rise.
Never the less, really appreciate your contribution
I'm actually building something similar and thank you very much for sharing you gave me a few inspirations. I want mine to integrate seemlesly with an animated Ai companion (it's harder than it sounds haha). Anyways great stuff! Gave it a star.
“Garth is the most significant artist that I've worked with and I've worked with Lulu and four other people, so we’re talking crème de la crème.” — Dean Learner
So there are a zillion of these, including ones that do what yours does. But that being said, I'm going to always upvote every single one of them because this is better and more important that what the big companies are doing across the board.
Thank you! Would love to hear which ones you've used. Working with local files without anything going to the cloud was the main thing I wanted. I tried Jan, OpenWork, Chatbox and Lobe, and couldn't get that working well in any of them. If there's one I missed it would be great to check it out!!
Have you tried Open WebUI? It doesn't determine which model works best for your hardware, but it otherwise has a very large feature set with a large community, and runs offline by default. Geared more for technical users, however.
It feels like the last mile is so long in this space. I'm a SWE with a ton of experience and I still haven't gotten over the decision paralysis of figuring out which model to run locally, and which inference platform should I use, and will my M1 Macbook Pro actually run anything well enough to be useful, and on and on.
It's kind of embarrassing to admit, but having something Just Work(tm) makes getting started so much easier.
Shouldn’t be embarrassing to admit.
It’s a huge new subject area and it’s completely exhausting to learn. I still feel like a total novice.
I have an M1 Max (more memory bandwidth) so I am not sure how much advice I can give about models, but I would suggest maybe trying the new LFM2.5-24B-A2B. It’s surprisingly good. If you want to mess about with multimodal work, Gemma 4 12B is really impressive. (I mostly use Muse Glimmer but it might be rather slow on an M1 Pro)
If you want to have the space to learn the terminology, tinker with what you can control etc., then you have three easy options: LM Studio, Unsloth Desktop or the llama.cpp native Mac app. But don’t get put off by the llama.cpp command line; most of these tools are running it behind the scenes, and most entries on Hugging Face have recommended parameters.
MLX is an option but so far I haven’t found it to have an edge over simple GGUF on Metal on the M1 at least; it seems it might have more advantages on the later M series chips, but it is usually slower for me.
I have explored running OpenCode inside a VM (you can “remote into it” from the OpenCode GUi desktop app). You might want to look at Paseo, which can do a similar thing but with many more coding agents.
I am quite intrigued by Osaurus; on macOS 26 and up it looks like it will manage a container for you.
You might also want to look at Orchard. Not done this myself yet but I can see the appeal.
https://orchard.andon.dev/ai.html
I am an experienced web dev in my fifties. Six months ago everything I have written above would have been total gibberish to me. Learning some of the open weights stuff has made me feel much less helpless than I think I might if I just developed a Claude dependency. Nothing matters more to me than grasping the fundamentals.
I still don’t really use LLMs for hands-off coding. But I have found that a local LLM can be a useful sidekick; even at the most basic level it’s a better Google that remembers the context of queries.
Thanks for taking the time to write all this up, I'm checking out the projects you mentioned now.
I feel you on the tinkering-to-feel-less-overwhelmed front. I've been through a few ground shifts in my career so far, and even a little bit of "Oh, so that's how it works, ok..." can make it feel so much less daunting.
No worries. And yes, absolutely. Even the tiniest opportunity to poke around with stuff changes my perspective dramatically.
I've picked up lots of stuff this way — just being able to watch the server logs is great!
P.S. I prodded myself into playing with Osaurus; it's actually rather good.
Excellent, concise overview of stuff to explore—
Thanks :-)
Mostly I think my own approach has been to try to combat my anxiety and exasperation about all of this by taking this measure of control.
While no really small local model comes super-close to a frontier model, the shape of the technology is settling and I think one can learn so much by approaching it the conventional nerdy way: what's a practical setup I can run on kit I already have? What can I build that helps me explore the problems?
I am quite cynical about AI still, including for generative coding, but I am not cynical about what I can run locally. There is no doubt that they are already helpful to me even in a minimal form.
And the really cool thing about being able to run them locally is to understand the tradeoffs.
Definitely check out Osaurus for macOS and Windows. Heavily privacy focused with e2ee iOS and iPadOS tethers.
In PasClaw, I specifically built it to do local embedding and other local only functionality. Whether it works well or not yet is another story.
I am using Hermes and I did not try charts, but the rest it can do with ease. And I use DeepSeek flash API as a driver most of the time.
Orca works great.
Fwiw, I'm liking the koboldcpp "ecosystem", including koboldlite; basically it's letting me try a little of everything, though there may be something of a "jack of all trades master of none" thing going on. But it has these nice little presets for, e.g. llms, voice, music, images, a "chatgpt"-esque interface, etc.
Haven't tried it yet, but I like that you shared your process in such an honest way, and kept it no frills, as in no LLM wall-of-text readme in the repo. Wish we could see that more often these days.
thank you! Haha I always make it a point these days do handwrite these things. I think everyone is tired of overly verbose perfectly written AI text.
Looks great. One thing I am missing is the ability to change the installation directory for the models. For space reasons, I want them to be on a different partition.
Good call. Are you on PC/Mac/Linux?
Linux. I saw you support HOB_USER_DATA, which is great. But I want my config to remain in .config/Hob and only move the models (or whatever else needs lots of space) to another disk/directory.
Edit: Actually I think HOB_USER_DATA is enough for my use case.
I did not enjoy that there was no way for it to clean up the 18GB of models it installed after I determined it was not really usable for any real work.
Sorry about that. I’ll make sure it cleans up after itself in the next update. What was the main issue you ran into?
The models are really dumb, mainly.
Why does it have to be JS? With all this AI, why can't we ship native apps.
It's still pretty tricky to do this as a solo dev to keep feature parity in my experience (fun fact, even Claude's own Desktop app is made in Electron).
That being said, majority of the heavy LLM lifting happens natively with llama.cpp :)
> Automatically finds the best open-weight model for your hardware on first install
Do you have examples of what hardware runs what models?
Here's roughly what it picks today from my testing:
- 8 GB Mac (M1/M2): Gemma 4 E2B
- 16 to 18 GB Mac: Gemma 4 E4B
- 24 to 64 GB Mac: Gemma 4 26B A4B
- 128 GB Mac Studio: gpt-oss 120B
- Windows/Linux with an RTX 3060 or 4070 (12 GB) and 32 GB RAM: Gemma 4 26B, split between the GPU and system memory
- RTX 4090: Gemma 4 26B fully on the GPU
- No GPU, 16 GB RAM: a compressed Gemma 4 E4B on the CPU
High level, it looks at your memory and memory bandwidth and then picks the smartest one that stays above about 15 tokens a second with room for a decent context.
Interesting, thanks for sharing this. I have some questions? I have a Macbook M5 with 16GB unified memory. What Model would you suggest I use with this. I want something that would replace dot or grokbot , do posting, some marketing work, and email handling. Not coding. What would you suggest?
It will automatically pick the best one for your HW on install
Thank you so much, that's helpful. Do you offer any benchmarks comparing with others?
Not yet, but that sounds like a useful overview to add to the landing page!
With the raise of AI, I believe: open-source and local software's will be on the rise, but with the open-source environment there is a real security issues that rise.
Never the less, really appreciate your contribution
I'm actually building something similar and thank you very much for sharing you gave me a few inspirations. I want mine to integrate seemlesly with an animated Ai companion (it's harder than it sounds haha). Anyways great stuff! Gave it a star.
Thank you. Yes please fork that's exactly why I put it out on MIT license :)
Did you happen to try bionic?
https://lmstudio.ai/
This doesn't seem to be open source? They link a GitHub account, but I don't see the repo for this tool.
Correct, they’re closed source and you can’t opt out of analytics
What made you change it's name from 'Garth'?
I went back and forth with my coworkers on the naming and we essentially voted on Hob vs Garth. Vote came to Hob. Do you prefer Garth?
“Garth is the most significant artist that I've worked with and I've worked with Lulu and four other people, so we’re talking crème de la crème.” — Dean Learner
Can we configure it to use local models but in LAN? Like Ollama or Strata.
So basically allow you to point the LLM to a hosted model on your local network if I understand correctly?
Yes.