Old news by now, but you might not be aware that llama-server can do multi-model for a while now,
Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI)
Then, any api client can just select a model and the system does the right thing.
It's great software. It just works.
__
You just need to ignore the cargo culting commandline options on social media.
But you should be listening to the devs.
Have you already enabled ngram-mod (or rather just spec-default)? It is practically free.
Caveat: this kind-of-works. There are plenty of subtle bugs around this. For example, it's hard to have a default model, and clients need to select the model explicitly in every API request, which makes it difficult to work with several models and multiple clients.
There are tools like llama-swap that make this pretty bullet proof and offer an OAI style API and compatibility for local Agents. I've been running flavors of OpenClaw and Hermes exclusively on local models hosted this way.
If you can get away with it, you really should stick to llama-server alone. When they added support the ability to load multiple models + an idle timeout to unload them, it finally did it for me.
Adding llama-swap adds a fair amount of complexity that is unpleasant to debug and its documentation isn't very good (the only real doc is the sample config file).
The native router matrix for llama.cpp is still pretty immature (though evolving quickly) and I plan to switch once it reaches feature parity.
Personally, I don't see the value in idle timeouts or time based eviction. It doesn't improve model load performance but does mean you're more likely to incur a load penalty for any request.
I generally use the same two models for my agentic workloads and they sit comfortably next to one another in the 32GB of VRAM on my GPU. If I need to load a larger model, llama-swap can easily eject both models if necessary.
Is that with vanilla llama.cpp or with the third-party llama-swap manager? Last time I checked llama-swap was still the go-to solution, although I admit I haven't looked into it further.
The router mode and matrix routing in llama.cpp is still early days and it can't easily juggle multiple models as easily as llama-swap so there's still benefits if you're using 24-32GB cards that can run multiple models simultaneously.
The gulf is quickly shrinking though and it seems like the need for llama-swap will disappear soon.
If you revisit my comment and pay attention to the opener:
> but you might not be aware that llama-server can do multi-model for a while now
you will see that the sentence structure clearly implies both a change compared with a prior state and also lack of any third-party thing.
So the answer to the question has already been encoded as text available.
_
I can see the desire for explicit validation though. For that, I would propose a sentence structure like
> Oh cool! That means that llama-swap is now superseded/no longer needed?
That shows that you've read and understand the message, gives you the double-check and might on top spark a conversation about how these solutions compare.
Plus that if the guy you're commenting too has spoken nonsense, they need to backpedal.
btw, llama-swap provides a nice UI for monitoring performance and logs, and even the ability to stop an infinite session that consumes GPU resources (sometimes that happens).
Does the llama.cpp UI provide the same? If not, it is too early to say that llama-swap is “superseded/no longer needed.”
> any api client can just select a model and the system does the right thing
My machine can only load one model at a time. The loading/unloading times just don't seem worth the switch. I tend to use qwen3.6 for anything and that's it. Then again, I am a simple coder.
This is all quantifiable. I regularly have my model run benchmarks against all the config permutations and then choose the best based on my criteria, which typically boil down to trading prefill and decode times
I'm running on Strix Halo so memory bandwidth is my constraint. In that example I'm describing the choice between using ROCm or Vulkan. I have a llama-swap config that can call different instances of llama-server running a toolbox with either runtime.
Memory bandwidth is my constraint too (M4 Max) but prefill and single-token decode don't run at the same time. It's best to use batched prefill so you can benefit from processing multiple tokens with a single pass through the model weights.
Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement.
ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.
At this point the options are llama-server or vLLM if you're serious about running things at your desk in the under 256GB RAM size class (70B, 120B size models). In addition to, of course, 27B to 35B size things. With of course a ton of compile time build customization options for whatever specific hardware platform you want to run either llama or vllm on.
> At this point the options are llama-server or vLLM
Which last time I checked, both use different formats of the weights, the former GGUF while the latter .safetensors. I mostly end up using vLLM these days and I'm a bit more performance sensitive than what I used to be. Just a shame it's a hassle to share the weights between them with conversion and what not, either batched or on-startup.
I think it is due to the new website? it now looks like every other vibe coded site,the only upside is that is looks more saleable for people unfamiliar with it, e.g., explaining OSPO,IT the stack you are using. they should also add a pricing page for eenterprise where they promise 99.9% uptime for local models*.
Wow it’s aggressively vibe coded. Nothing inherently wrong with that, but it looks a bit amateurish which is funny.
I’m still waiting on 98.css to become the standard for vibe coded sites. You don’t have to read docs anyway if you’re just using LLMs! All you have to do is say “use 98.css” and you have a 10/10 site
possibly hitting front page because this website is fairly new? For me, it's certainly the first time I've seen a one-liner curl|bash installer for llama.cpp, which was basically the only reason to use ollama.
The gap was MUCH larger in the past, but in my tests, oMLX and llama.cpp are now very similar (within 10%) in both prompt processing and generation speed. GGUF ecosystem provides a better selection of quants, in my experience Unsloth ones are excellent.
I thought the main advantage of oMLX is it's less likely to invalidate the KV cache when working with coding agents, which is key when working on a Mac because of the slower prompt processing.
true, switched from ollama to llama.cpp these days and it's good. wonder if this is also the best option for edge ai deployment(currently use it on desktop)
Vanilla llama.cpp leaves a lot of performance on the table. I'm reaching 120 t/s with a custom inference engine for a model that llama.cpp can barely run at 70 t/s. Theoretical maximum on this hardware is around 147 t/s according to measured memory bandwidth.
WRT speculative decode, basically zero finetunes keep it. I'm testing with some ridiculous abliterated amalgamation so spec decode has been gone for most of its ancestry.
Fable recommended n-gram speculation so I'm working on that now.
Optimize startup params for llama-server for your hardware (not llama.cpp itself), on my 2x 4090 I got ~20% speedup after maybe 40 mins.
ps. ngram didn't work for me very well, but dedicated speculative model works very well
ps. 2. in my case I'm just maintaining Makefile that does everything from update/upgrade (git pull/recompile) to starting server with different models, stuff like:
I'm on M4 Max, so I don't have split parameters to tune. I can fit full context, I can fit the full model in unified memory, my only concern has been the very meager memory bandwidth of these chips. The only thing that can seem to feed the GPU is very large prefill batches, on the order of many thousands of tokens per batch, and that's with only a 9B model at 3 bpw.
llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit.
Two examples:
- https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost a month. Fortunately there's workaround (switching to Vulkan rather than ROCm devices), but both the way the bug was introduced and the way it is not fixed just doesn't give much confidencen
- LM Studio is using llama.cpp internally for GGUF, they ship their own build with their closed source system as "runtimes". Their ROCm runtime does not enable the the AMD GPU inside the Framework, even thought the llama.cpp version would support it. So their runtime keeps telling me that there's no supported AMD GPU -- again, the solution is to use the GPU with the Vulkan devices. Not fixed since Jan at least https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1...
I guess overall it's the worst runtime I've seen so far, except for all the other runtimes out there... I'm a fan, though in some cases I don't have enough knowledge, or I don't have access to fix things, and that feels like a bummer...
A team from AMD maintains Lemonade, another all-in-one setp with convenient installers for setting everything up: https://lemonade-server.ai/
These are probably better than running against llama.cpp ROCm directly as there are frequent/constant regressions on the main branch, especially for gfx1151 (Strix Halo), but RDNA in general.
There are number of AMD-focused llama.cpp forks (nathanw1014, charlie12345, ciru-ai, justinappler, etc) - as well as a few alternatives like hipfire or my hipEngine. While ROCm has gotten a lot better, one of the things I've found after writing an inference engine that has completely custom tuned/fused C++/HIP kernels, is that while it's been pretty straightforward to match/beat llama.cpp ROCm performance, that Vulkan RADV has been a lot harder since RDNA3 support for ROCm has a few issues that make it underperform ACO on some common operations on both gfx1100 and gfx1151 (see: https://github.com/ROCm/ROCm/issues/6409 )
In general, for anyone just looking to run LLM models on an AMD card, I'd just recommend going with llama.cpp Vulkan and skipping ROCm completely.
I think ROCm is just a total second class citizen in the space TBH.
It's a shame coz it's not even really what we want, we would obviously all be better served if we could use Vulkan or something. But I guess it's inevitable that a generic framework lags behind here.
If I was AMD I'd hire a whole ecosystem team to sit next to the ROCm people and just support big users like llama.cpp to work better on their HW, e.g. giving OSS maintainers access to their board farms. Maybe they have already done that, in which case I guess I should say I'd double the size of that team.
You probably can just use Vulkan. Why don't you think Vulkan is an alternative? Can it not handle BLAS and memory access? What is a graphics card supposed to be doing that isn't efficient in Vulkan?
Sometimes it’s just that a particular backend is poorly optimized or has a regression on a particular platform as compared to the “mainstream” backends. For example, whisper.cpp’s Vulkan backend performs 2-3x worse than CPU on my Snapdragon X2 laptop when using the ggerganov v3 turbo model. It’s probably a simple fix, but it does need to be fixed.
I just switched to Vulkan, and be done with it. :)
As much as I can tell, the ROCm version of llama.cpp would be a bit faster on prompt processing, but about the same on the token generation as Vulkan. Real life benchmarks don't seem to give any "ROCm or nothing" sort of vibes. And the difference between the performance of different models are way bigger than the difference between the llama.cpp versions (and versus different runtimes like the llama.cpp/GGUF and the MLX runtimes on Mac for the same models)...
I've tinkered enough with the serving, that I'd rather do something with them with, say 10% slower speed, than spending hours on seting things up again... YMMV
Hipfire works on my 7900xtx best of all. Biggest surprise - Qwen3.6-27B dense does not grind to a halt with context depths all the way up to 250K! Measured at 1K, 8K, 32K, 64K, 128K, 192K, 250K - Hipfire speed holds close to 40 tok/s. Finished 5 days run of gpu 100% inferencing, Hipfire did not crash even once afaics. I assumed speed dropping like a stone on Qwen models was a feature/bug of the model, and "nothing can be done about it". Hipfire showed me wrong - pleasantly surprised there.
Atm OMP (oh-my-pi) with opencode-go subscription using deepseek-v4-flash-0731 (and mimo-v2.5 as advisor). Used pi before (still good, but too minimalist for all general use; works on Android under Termux btw) and OpenCode (which is ok too). I've used Codex the longest, but the $20 sub doesn't last that long. Before that Claude (and Claude on the web - about Nov'25), but nowadays with the new models not possible to do anything on the cheap sub. Got Zai (GLM-5.2) legacy sub to supplement too for variety. :-) Run local jobs on the 7900xtx (24gb vram - so dense models) batch bulk jobs. Non-dense models like nemotron-cascade-2-30b-a3b MoE run at 100 tok/s, looking forward to try the new lightning-3.5-30b-a3b. And got old MBP M2 Max 96gb uram (unified ram) for bigger sparse MoE models. Testing DS4F there under ds4 server/harness (too slow, 12-15 tok/s even with REAP25 reduced model), and (got agents) porting Ling-3.0-flash atm from llama.cpp -> ds4. Hoping only A5B (5B active, versus 13B of DS4F) will result in better tok/s. TBS. Back in Feb-Apr'26 looked local will be the only way to run llm jobs (plus 300w gpu job works out for office heating when cold), otherwise the ex-quota API payg costs would bankrupt me. But now with the oss weights models and providers like opencode - actually it's not only faster but also cheaper for me to run llm jobs on the api. Engines for local - llama.cpp for all, hipfire (mentioned) for the 7900xtx, and mlx on the apple ASI (that never quite work out for me). And ds4 trying forever when off-line :-) - e.g. on a flight - either ds4-agent, or ds4-server + agent pi. And I use Hermes as a general agent, non-coding agent, for everything that's not strictly coding.
Lemonade-server works pretty well (most of the time). It wraps llama.cpp and other runtimes - it downloads the official binaries as far as I could see, and you can set alternative versions if needed. Works nicely with Strix Halo for a while now.
The first one multiple contributors highlighted the PR as urgent andits had lots of review but it appears to be waiting for another review and/or someone that owns the affected hardware to test that the PR fixes the issue, it wpuld be easy for you to test and report whether or not it does, and the second thing is not related to llama.cpp at all
Yes ideally there would be testing every hardware + software combo but this costs engineering time and $$$ money, and you are running on master branch, no master branch of any software is stable, inherently, if you run into issues, just stick to the old hash where stuff worked, why are you insistent on both being at the bleeding edge and experience 0 breakage!
The second I didn't say it's any of llama.cpp's "fault", but it is _related_ to llama.cpp since it's being shipped in another system, aye?
Can't stick to the old hash either, because older version have different bugs. E.g. on older versions the same Qwen3.6 model reliably fails to call specific tools due to template issues, while just having the newer llama.cpp version has that fixed. So different versions - different bugs, rather than no bugs.
In some cases it may be that llama.cpp causes a bug or lack of functionality downstream but lack of ROCM support appears to be entirely LM Studio’s doing and unrelated to llama.cpp
Sorry if I was too harsh, it’s just that my perception watching the repo has been that the llama.cpp devs are by far the most cautious and slow moving of all the inference implementations, so I found your perspective a bit surprising, I do think that the desire for stable software that never break, and software that supports the latest models and devices/device API’s are conflicting, nothing will do both, and I think that llama.cpp devs do a good job of balancing between shipping features and not breaking users.
Anything that suggests curl into bash just plain sketches me out. (edit: I know, this isn't totally rational, it just seems weird to me. We download and trust a lot of software and run code from a bunch of package repositories as a regular activity...).
Package management provides cryptographic attestation over the entire process, including the scripts run. Nothing is arbitrary nor mutable in the default state. The files touched will be in a predictable place and the scope of privilege escalation is clearly defined.
It's more about installation location for me. For these types of projects, I don't even install it. I just clone, build and run from there. I have no idea where the bash script is going to put things.
I mean it's probably not, I just haven't got used to it yet. It's about the same level of security as installing a windows app on win2000 25 years ago and blindly downloading a .exe off the internet and running it to get into the install wizard. But indeed I also kind of blindly trust that whatever I'm getting from the debian trixie officially gpg-signed packages isn't backdoored.
One thing I do not do as a matter of practice is install things with a ridiculous number of recursive npm dependencies.
I do not blindly trust anything, and come to the conclusion that downloading binaries from Debian trixie is a lot more safe. There is a world of difference between "curl | sh" and downloading from a curated package repository maintained by a respected community with well-defined processes.
Absolutely, and it's even worse when you are asked to blindly download and run software like the Brave browser, which is open source but exclusively packaged and distributed by an adware company with a questionable history. The software itself is open source, but independent builds seem to be discouraged and no Linux distro packages Brave as a result. I built it myself in the past, but it takes close to two hours on a workstation and isn't worth the effort to keep it updated.
so instead of downloading a script with curl and running it with bash, you should download it with git and run it with cmake? if you aren't reviewing the code between download and execution, what's the difference?
You're not wrong about the appearance of sketchiness of that, but.... dude, come on. "It's not hard" is only true because you already know how to do it.
Exactly. That way of assuming everyone is a seasoned dev or sysadmin always baffles me. There are a ton of tinkerers out there, we are legion, and we are often reluctant to ask, especially here (as suggested by another parallel comment) because we don't want to receive an answer like this. Full circle.
If you’re unsure how to do it, you can ask others for help or Claude will tell you.
Point Claude Code at a repository and ask how to install it safely. You don’t have to know about make or cryptography of HTTPS or anything, really. It will walk you through the options and risk.
If you have questions about any part of it—i.e. you don’t recognize an acronym or deeply understand why something works—you can ask.
It's literally three steps, assuming you have the equivalent of the debian "build-essential" dependencies installed on your system for cmake, compiler. It's the exact three rows of my post, pasted one at a time into the CLI. The llama-server build guide page is actually pretty good.
I mean, sure, if there's people who can't figure that out, they're probably better off using a GUI that is a wrapper on top of somebody else's precompiled llama-server, like unsloth studio or lm studio. There's a good sized market for that and I wish them well.
You're right about all of that up until the point any one of those commands doesn't work and you get a garglblargl amount of incomprehensible error messages. You've been in the sauce for so long that you've forgotten what it's like to start fresh in these things.
Please give it a try before you assume it won't work? Tell me what platform you're on and follow the three sequential cli commands I provided, and if it doesn't compile I'll be glad to look at your error message and get it going in the right direction.
You're missing the trees for the forest, friend. I know how to troubleshoot this stuff. I wouldn't expect some random schmo who's just learning ubuntu for the first time to be able to.
This, "just run these three commands" attitude is exactly the reason why these curl|sh "installers" have become popular.
GP is saying "it's not hard", yet you seem to be hearing "you should already know how to do this without ever looking it up".
Those aren't the same assertions.
It's not a sensible assumption that a process must necessarily be difficult or complex just because you don't already know how to do it. There are an unenumerable number of tasks each of us don't know how to do and have never done before which are not difficult at all.
>you don't need to know the weird esoterica of a 26 year old piece of software
You really don't need to know much at all in order to compile llama.cpp. You can literally ask any half decent AI agent to do it for you or give you a step by step guide, and troubleshoot any error you get.
> Anything that suggests curl into bash just plain sketches me out.
Yeah, 100% and it's becoming more and more of a thing, see rust install for example.
OTOH, if you're installing llama.cpp, you're more than likely planning to run an LLM on your Linux box with an agentic harness, so a curl into bash thing might be the least of your security concerns, :-)
One way I prevent possible catastrophic fuckups is that the 'doing code work' box that runs opencode or pi or whatever, is its entirely own separate VM and desktop environment (running as a xen or kvm guest and with its own LVM logical volume as boot/root and /home disk), than the machine running llama-server itself.
The harness gets the openai-compatible endpoint fed into it to talk to llama-server across the network, but the VM has no access whatsoever to my personal files, mail, backups/deep storage, fileserver, Documents folder, etc.
Yeah I recently tried the coding harness that's recommended here, Pi, in a bubble wrap sandbox and was horrified to learn that it spams multiple warnings at you if you don't give it write access to its own config/extension folder... Everyone else is rawdogging it I guess.
I think the best course is a docker container or a VM, i agree with the sentiment but trying to restrict the permissions of these things is fighting an uphill battle, it’s probably best to let them reign king in a disposable and isolated environment
You are horrified it needs access to its own stuff? I mean, any software that supports plugins needs access to them, no? How is that unreasonable? And with bwrap that’s an extra cli option.
It's okay-ish for Rust because you only need the curl|bash for Rustup, which is meant as a development environment. On production you can, for example, use APT to download it from Debian.
What would you win? Cmake is capable of executing shell commands and you end up with a binary that will execute on your machine anyway. At the end of the day it is just a matter of trust anyway, isn't it? I personally use bwrap if I'm not confident about the source.
Cloning a repo and building it is not _that_ hard, but easy installation is often the thing that makes or breaks a product. I believe Ollama proves that point in this context.
Mostly agree but at least with git clone you have a hash and the malicious code has to be served to everyone, and GitHub is less likely to get hijacked by a malicious third party.
But yes, still trusting the project with arbitrary code execution on your machine, including build formulas that pull stuff from the internet and suffer from all the above anyways
Yesterday I installed llama.cpp to test it with local AI Data Analyst that I'm building. I was also testing other open LLM providers: Ollama, Jan, vLLM, LM Studio. I had older NVIDIA card (RTX 3070) and llama.cpp instalation was smooth, contrary to vLLM which required me to reinstall CUDA drivers because by default it installed the latest one. I'm curious if there is a speed difference between the same open LLM model served with different runners.
I was able to fit only small models Qwen3.5-4B in RTX3070 which is not very useful for Python and SQL generation thought. When I wan to test larger open LLM models I often just use cloud resources.
If I said, “I want a setup that is usable for an agentic coding workflow, and it MUST be local”, what’s the smallest/cheapest option right now?
It’s _technically_ possible to get agents running on all kinds of setups but there seems to be an (undefined) floor for useful setups.
A lot of the stories people have about getting setups running on relatively low end hardware turn out to have huge compromises or run into issues on anything but trivial cases. I’ve found it hard to find a consensus.
Or maybe I just don’t like the multiple thousand dollar price tags people are suggesting…
I think the best option right now, since Apple has raised prices and Mac minis are basically impossible to get your hands on, is to build your own micro-itx machine. I actually built a mini-itx machine, but it does restrict your options a bit.
The Arc series Intel GPUs are what I think make this possible. I built a machine with an Arc b50 - it runs Gemma 26b a4b qat at around 30tok/s with their MTP head and prompt processing sits at around 500 tok/s. The really beautiful thing about this setup is the entire energy envelope of this machine sits at 120w at full load - when idle, it's at 40w and i've done some work in ubuntu to basically intelligently hibernate, which drops it to 0 watts when not in use. You can use a raspberry pi and Wake on Lan to wake the machine up for a overall draw of around 5 watts when not in use.
All in all this machine cost me 1.4k to build - but if you used micro-itx instead of mini-itx parts you could do it for under 1k - it has just 16gb of ddr5 but you don't really need more if you use models that can fit in vram.
I think it's pretty incredible that you can run an actually useful coding agent on a machine with a power envelope that is less than an incandescent light bulb. If you go up to micro-itx you can do even large cards like an intel b60 with 24gb or a b70 with 32gb and run even more powerful models. For all of these intel GPU's you'll want to compile the latest llama.cpp version with SYCL support - they are getting speedups every day, so worth staying on the edge.
straightforward solution is probably Qwen3.6-27B-Q4 running on a used RTX 3090. price on those is unfortunately high, in fact so high that getting a new Radeon AI PRO R9700 might be a better deal
a solid step up from there is anything that can run Deepseek V4 Flash but the hardware ask there is a bit higher
Is llama.cpp (and thus llama.app) really that much better than Ollama? I've Only ever played with Ollama, so geniously curious to hear other's real-world experiences.
I was playing with ollama because of name recognition. I was finding pretty quickly that the selection of models on the "ollama store" was pretty stale, and gguf models on huggingface are technically downloadable but don't work as well.
The "friends don't let friends use ollama" article linked in another comment convinced me to try llama-swap. I find it easier to directly deal with gguf files. Hard to quantify, but the outputs of the LLM seem better too. Asking the same gguf the same question with the same chat harness, I subjectively find llama.cpp does better. Might be some different defaults. I haven't dug deeply.
The dev behind ollama is adamant that ollama doesn't use llama.cpp (based on a technicality -- it uses ggml, which is created by the same people behind llama.cpp and is the backend of llama.cpp)
He made such a big fuss about ollama implementing their own kernels and felt slighted about the online comments saying ollama didn't properly credit llama.cpp and it kind of left a bad taste in the mouth among the local inference community.
Small tip, install llama.cpp with brew before llama.app, which will pick up the existing llama.cpp. That way it's easier to stay up to date with llama.cpp, since llama.app is on a slower release cadence.
Also, models installed with the hugging face CLI (hf) are picked up by llama.app automatically. The CLI will keep the model cache updated, e.g. when models get updated.
I think I can probably run Gemma 3 12B on my macbook M3 pro with 18GB. The question is, should I do it? This small model is probably not capable of doing a lot or advanced coding or reasoning. What else could it be used for, since it can run locally and privately?
A quantized Qwen3.6-35B-a3b can run in a similar footprint to gemma 12b, but is smarter. It can do coding tasks, if you specify them at a finer-grain than with bigger models.
pro tip: if u are building ur own harness with python (recommended) , use "llama-cpp-python"...
I started with "llama-server" and custom stuff around it, which is great for single model setups.. but for multi-model harness with quick switching, llama-cpp-python is peak
I tried to run in on my Arc A770, but all of the binary releases I could find were compiled without OpenVINO support enabled. I tried compiling it myself, but after two days of the compiler running it failed.
the full set of llama.cpp binaries builds in under 5 minutes with an unmodified build workflow straight from their github page on a literally ten year old dual xeon.
It also even easier to get working and integrate into your system in a sustainable manner with NixOS. Do it by hand or throw an LLM at it, it will get you a declarative patch for your NixOS config that brings llama-cpp into your config that you can review and add under version control (no random `make install` build artifacts contaminating your system, no wondering "what was it that I ran? what are all these files? how do I do the same with a newer version?" a couple months later). There's also likely some build cache where Nixoids have already build what you want.
I had a great experience with llama-cpp with Nvidia backend on NixOS.
Does the A770 use the Xe driver? If so then it might work with the scripts that I've been using to build llama.cpp with SYCL support for the Arc Pro B70.
It's designed so that you can re-run the scripts to pull the latest updates. When Muse Glimmer was released the other day I just ran the 02 script to build the latest version of llama.cpp with support for it.
llama.cpp is like the ffmepg of AI, and one of the reasons I so greatly dislike ollama is that the latter completely obfuscates that they're a rebrand of the former. Georgi Gerganov and team did all the hard work; ollama is langchain-like VC-bait with a HF download wrapper.
Hey folks, a bit of a hijack; but I've taken to using the Kobold gui, which I'm liking much more than Ollama -- but are there any major benefits to going straight up llama.cpp? Yes, this is somewhat of a question about laziness.
This is like complaining that the mariadb team has failed to make their tech easy to use. It's a server side piece of software. There are plenty of user friendly GUIs or wrappers for it. lm-studio or unsloth are to llama.cpp as something like phpmyadmin is to mysql/mariadb.
It's from https://github.com/ggml-org/llama.cpp -- not associated with Meta, it's been around for years, and surely they know about it -- so I would guess either it's not a trademark violation or they don't care.
It seems that llama.app is a direct competitor to ollama.com
I can understand the desire for the llama.cpp project to want to own the end user relationship, it is true that previous to this they were a tool provider and not really owning the end user experience.
Ollama uses the llama.cpp backend for inference. I find Ollama noticably slower. Llama.cpp has had a built-in webui (used as llama-server) for a long time now so have owned the user experience too.
Old news by now, but you might not be aware that llama-server can do multi-model for a while now,
Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI)
Then, any api client can just select a model and the system does the right thing.
It's great software. It just works.
__
You just need to ignore the cargo culting commandline options on social media. But you should be listening to the devs.
Have you already enabled ngram-mod (or rather just spec-default)? It is practically free.
Caveat: this kind-of-works. There are plenty of subtle bugs around this. For example, it's hard to have a default model, and clients need to select the model explicitly in every API request, which makes it difficult to work with several models and multiple clients.
There are tools like llama-swap that make this pretty bullet proof and offer an OAI style API and compatibility for local Agents. I've been running flavors of OpenClaw and Hermes exclusively on local models hosted this way.
https://github.com/mostlygeek/llama-swap
If you can get away with it, you really should stick to llama-server alone. When they added support the ability to load multiple models + an idle timeout to unload them, it finally did it for me.
Adding llama-swap adds a fair amount of complexity that is unpleasant to debug and its documentation isn't very good (the only real doc is the sample config file).
The native router matrix for llama.cpp is still pretty immature (though evolving quickly) and I plan to switch once it reaches feature parity.
Personally, I don't see the value in idle timeouts or time based eviction. It doesn't improve model load performance but does mean you're more likely to incur a load penalty for any request.
I generally use the same two models for my agentic workloads and they sit comfortably next to one another in the 32GB of VRAM on my GPU. If I need to load a larger model, llama-swap can easily eject both models if necessary.
Is that with vanilla llama.cpp or with the third-party llama-swap manager? Last time I checked llama-swap was still the go-to solution, although I admit I haven't looked into it further.
llama.cpp has router mode these days, swapping out models for you, you no longer need llama-swap.
I use it on the daily. Haven't exactly missed ollama.
The router mode and matrix routing in llama.cpp is still early days and it can't easily juggle multiple models as easily as llama-swap so there's still benefits if you're using 24-32GB cards that can run multiple models simultaneously.
The gulf is quickly shrinking though and it seems like the need for llama-swap will disappear soon.
If you revisit my comment and pay attention to the opener:
> but you might not be aware that llama-server can do multi-model for a while now
you will see that the sentence structure clearly implies both a change compared with a prior state and also lack of any third-party thing.
So the answer to the question has already been encoded as text available.
_
I can see the desire for explicit validation though. For that, I would propose a sentence structure like
> Oh cool! That means that llama-swap is now superseded/no longer needed?
That shows that you've read and understand the message, gives you the double-check and might on top spark a conversation about how these solutions compare. Plus that if the guy you're commenting too has spoken nonsense, they need to backpedal.
btw, llama-swap provides a nice UI for monitoring performance and logs, and even the ability to stop an infinite session that consumes GPU resources (sometimes that happens).
Does the llama.cpp UI provide the same? If not, it is too early to say that llama-swap is “superseded/no longer needed.”
It's also convenient if you want to run separate builds behind a single config, sometimes there's a PR I want to run before it's merged
> any api client can just select a model and the system does the right thing
My machine can only load one model at a time. The loading/unloading times just don't seem worth the switch. I tend to use qwen3.6 for anything and that's it. Then again, I am a simple coder.
> (Optimized by you through testing. Not that AI)
Why not optimized by AI through testing ? Give it a test set to work on and let it loose.
AI doesn't necessarily know what feels like a good tradeoff to you. I'm sure it could help guide you though.
^ This.
Intent is the answer and AI has none.
This is all quantifiable. I regularly have my model run benchmarks against all the config permutations and then choose the best based on my criteria, which typically boil down to trading prefill and decode times
What model has to trade between those? I have both. You just have different, independently-optimized forward passes for each.
I'm running on Strix Halo so memory bandwidth is my constraint. In that example I'm describing the choice between using ROCm or Vulkan. I have a llama-swap config that can call different instances of llama-server running a toolbox with either runtime.
Memory bandwidth is my constraint too (M4 Max) but prefill and single-token decode don't run at the same time. It's best to use batched prefill so you can benefit from processing multiple tokens with a single pass through the model weights.
Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement.
ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.
At this point the options are llama-server or vLLM if you're serious about running things at your desk in the under 256GB RAM size class (70B, 120B size models). In addition to, of course, 27B to 35B size things. With of course a ton of compile time build customization options for whatever specific hardware platform you want to run either llama or vllm on.
> At this point the options are llama-server or vLLM
Which last time I checked, both use different formats of the weights, the former GGUF while the latter .safetensors. I mostly end up using vLLM these days and I'm a bit more performance sensitive than what I used to be. Just a shame it's a hassle to share the weights between them with conversion and what not, either batched or on-startup.
I think it is due to the new website? it now looks like every other vibe coded site,the only upside is that is looks more saleable for people unfamiliar with it, e.g., explaining OSPO,IT the stack you are using. they should also add a pricing page for eenterprise where they promise 99.9% uptime for local models*.
Wow it’s aggressively vibe coded. Nothing inherently wrong with that, but it looks a bit amateurish which is funny.
I’m still waiting on 98.css to become the standard for vibe coded sites. You don’t have to read docs anyway if you’re just using LLMs! All you have to do is say “use 98.css” and you have a 10/10 site
https://jdan.github.io/98.css/
Using 98.css would still leave you with the AI slop text wording.
The core problem is that some people don't even seem to notice / care.
Wow, gonna use that. Thanks
possibly hitting front page because this website is fairly new? For me, it's certainly the first time I've seen a one-liner curl|bash installer for llama.cpp, which was basically the only reason to use ollama.
does your comment depend on the OS? I thought MLX has better performance on MacOS than llama.cpp
The gap was MUCH larger in the past, but in my tests, oMLX and llama.cpp are now very similar (within 10%) in both prompt processing and generation speed. GGUF ecosystem provides a better selection of quants, in my experience Unsloth ones are excellent.
I thought the main advantage of oMLX is it's less likely to invalidate the KV cache when working with coding agents, which is key when working on a Mac because of the slower prompt processing.
llama-server also supports saving the kv cache to SSD. I had no issues with cache invalidation using pi.
true, switched from ollama to llama.cpp these days and it's good. wonder if this is also the best option for edge ai deployment(currently use it on desktop)
Vanilla llama.cpp leaves a lot of performance on the table. I'm reaching 120 t/s with a custom inference engine for a model that llama.cpp can barely run at 70 t/s. Theoretical maximum on this hardware is around 147 t/s according to measured memory bandwidth.
Just run /goal to optimise it and you should be good in less than an hour. Also best to use models that support speculative decoding.
Optimize llama.cpp? Hmm.
WRT speculative decode, basically zero finetunes keep it. I'm testing with some ridiculous abliterated amalgamation so spec decode has been gone for most of its ancestry.
Fable recommended n-gram speculation so I'm working on that now.
Optimize startup params for llama-server for your hardware (not llama.cpp itself), on my 2x 4090 I got ~20% speedup after maybe 40 mins.
ps. ngram didn't work for me very well, but dedicated speculative model works very well
ps. 2. in my case I'm just maintaining Makefile that does everything from update/upgrade (git pull/recompile) to starting server with different models, stuff like:
I'm on M4 Max, so I don't have split parameters to tune. I can fit full context, I can fit the full model in unified memory, my only concern has been the very meager memory bandwidth of these chips. The only thing that can seem to feed the GPU is very large prefill batches, on the order of many thousands of tokens per batch, and that's with only a 9B model at 3 bpw.
Yeah for that dense models are not great better to pick some MoE to get benefits of large vram pool while keeping reasonable speed.
llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit.
Two examples:
- https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost a month. Fortunately there's workaround (switching to Vulkan rather than ROCm devices), but both the way the bug was introduced and the way it is not fixed just doesn't give much confidencen
- LM Studio is using llama.cpp internally for GGUF, they ship their own build with their closed source system as "runtimes". Their ROCm runtime does not enable the the AMD GPU inside the Framework, even thought the llama.cpp version would support it. So their runtime keeps telling me that there's no supported AMD GPU -- again, the solution is to use the GPU with the Vulkan devices. Not fixed since Jan at least https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1...
I guess overall it's the worst runtime I've seen so far, except for all the other runtimes out there... I'm a fan, though in some cases I don't have enough knowledge, or I don't have access to fix things, and that feels like a bummer...
For updated/validated updates, Donato Capitella maintains independent Strix Halo "toolboxes": https://strix-halo-toolboxes.com/
A team from AMD maintains Lemonade, another all-in-one setp with convenient installers for setting everything up: https://lemonade-server.ai/
These are probably better than running against llama.cpp ROCm directly as there are frequent/constant regressions on the main branch, especially for gfx1151 (Strix Halo), but RDNA in general.
There are number of AMD-focused llama.cpp forks (nathanw1014, charlie12345, ciru-ai, justinappler, etc) - as well as a few alternatives like hipfire or my hipEngine. While ROCm has gotten a lot better, one of the things I've found after writing an inference engine that has completely custom tuned/fused C++/HIP kernels, is that while it's been pretty straightforward to match/beat llama.cpp ROCm performance, that Vulkan RADV has been a lot harder since RDNA3 support for ROCm has a few issues that make it underperform ACO on some common operations on both gfx1100 and gfx1151 (see: https://github.com/ROCm/ROCm/issues/6409 )
In general, for anyone just looking to run LLM models on an AMD card, I'd just recommend going with llama.cpp Vulkan and skipping ROCm completely.
I think ROCm is just a total second class citizen in the space TBH.
It's a shame coz it's not even really what we want, we would obviously all be better served if we could use Vulkan or something. But I guess it's inevitable that a generic framework lags behind here.
If I was AMD I'd hire a whole ecosystem team to sit next to the ROCm people and just support big users like llama.cpp to work better on their HW, e.g. giving OSS maintainers access to their board farms. Maybe they have already done that, in which case I guess I should say I'd double the size of that team.
You probably can just use Vulkan. Why don't you think Vulkan is an alternative? Can it not handle BLAS and memory access? What is a graphics card supposed to be doing that isn't efficient in Vulkan?
Sometimes it’s just that a particular backend is poorly optimized or has a regression on a particular platform as compared to the “mainstream” backends. For example, whisper.cpp’s Vulkan backend performs 2-3x worse than CPU on my Snapdragon X2 laptop when using the ggerganov v3 turbo model. It’s probably a simple fix, but it does need to be fixed.
I have a framework 13, but I couldn't imagine running a local llm on it, how do you do it? Do you have a eGPU?
If you have AMD, you can benefit from the unified memory.
So are there any alternatives which do actually work well with ROCm OOTB?
I just switched to Vulkan, and be done with it. :)
As much as I can tell, the ROCm version of llama.cpp would be a bit faster on prompt processing, but about the same on the token generation as Vulkan. Real life benchmarks don't seem to give any "ROCm or nothing" sort of vibes. And the difference between the performance of different models are way bigger than the difference between the llama.cpp versions (and versus different runtimes like the llama.cpp/GGUF and the MLX runtimes on Mac for the same models)...
I've tinkered enough with the serving, that I'd rather do something with them with, say 10% slower speed, than spending hours on seting things up again... YMMV
Hipfire works on my 7900xtx best of all. Biggest surprise - Qwen3.6-27B dense does not grind to a halt with context depths all the way up to 250K! Measured at 1K, 8K, 32K, 64K, 128K, 192K, 250K - Hipfire speed holds close to 40 tok/s. Finished 5 days run of gpu 100% inferencing, Hipfire did not crash even once afaics. I assumed speed dropping like a stone on Qwen models was a feature/bug of the model, and "nothing can be done about it". Hipfire showed me wrong - pleasantly surprised there.
Thanks, sounds promising. Written in Rust, no PyTorch etc, sounds like my kinda tool. Surprised I haven't seen more folks mention it here.
Also if I may ask, what does the rest of your stack look like (agent, harness etc)?
Atm OMP (oh-my-pi) with opencode-go subscription using deepseek-v4-flash-0731 (and mimo-v2.5 as advisor). Used pi before (still good, but too minimalist for all general use; works on Android under Termux btw) and OpenCode (which is ok too). I've used Codex the longest, but the $20 sub doesn't last that long. Before that Claude (and Claude on the web - about Nov'25), but nowadays with the new models not possible to do anything on the cheap sub. Got Zai (GLM-5.2) legacy sub to supplement too for variety. :-) Run local jobs on the 7900xtx (24gb vram - so dense models) batch bulk jobs. Non-dense models like nemotron-cascade-2-30b-a3b MoE run at 100 tok/s, looking forward to try the new lightning-3.5-30b-a3b. And got old MBP M2 Max 96gb uram (unified ram) for bigger sparse MoE models. Testing DS4F there under ds4 server/harness (too slow, 12-15 tok/s even with REAP25 reduced model), and (got agents) porting Ling-3.0-flash atm from llama.cpp -> ds4. Hoping only A5B (5B active, versus 13B of DS4F) will result in better tok/s. TBS. Back in Feb-Apr'26 looked local will be the only way to run llm jobs (plus 300w gpu job works out for office heating when cold), otherwise the ex-quota API payg costs would bankrupt me. But now with the oss weights models and providers like opencode - actually it's not only faster but also cheaper for me to run llm jobs on the api. Engines for local - llama.cpp for all, hipfire (mentioned) for the 7900xtx, and mlx on the apple ASI (that never quite work out for me). And ds4 trying forever when off-line :-) - e.g. on a flight - either ds4-agent, or ds4-server + agent pi. And I use Hermes as a general agent, non-coding agent, for everything that's not strictly coding.
Lemonade-server works pretty well (most of the time). It wraps llama.cpp and other runtimes - it downloads the official binaries as far as I could see, and you can set alternative versions if needed. Works nicely with Strix Halo for a while now.
https://lemonade-server.ai
The first one multiple contributors highlighted the PR as urgent andits had lots of review but it appears to be waiting for another review and/or someone that owns the affected hardware to test that the PR fixes the issue, it wpuld be easy for you to test and report whether or not it does, and the second thing is not related to llama.cpp at all
Yes ideally there would be testing every hardware + software combo but this costs engineering time and $$$ money, and you are running on master branch, no master branch of any software is stable, inherently, if you run into issues, just stick to the old hash where stuff worked, why are you insistent on both being at the bleeding edge and experience 0 breakage!
I did report my test results on the first one. :)
The second I didn't say it's any of llama.cpp's "fault", but it is _related_ to llama.cpp since it's being shipped in another system, aye?
Can't stick to the old hash either, because older version have different bugs. E.g. on older versions the same Qwen3.6 model reliably fails to call specific tools due to template issues, while just having the newer llama.cpp version has that fixed. So different versions - different bugs, rather than no bugs.
Why the beating you are trying to gimme, mate? :)
In some cases it may be that llama.cpp causes a bug or lack of functionality downstream but lack of ROCM support appears to be entirely LM Studio’s doing and unrelated to llama.cpp
Sorry if I was too harsh, it’s just that my perception watching the repo has been that the llama.cpp devs are by far the most cautious and slow moving of all the inference implementations, so I found your perspective a bit surprising, I do think that the desire for stable software that never break, and software that supports the latest models and devices/device API’s are conflicting, nothing will do both, and I think that llama.cpp devs do a good job of balancing between shipping features and not breaking users.
Anything that suggests curl into bash just plain sketches me out. (edit: I know, this isn't totally rational, it just seems weird to me. We download and trust a lot of software and run code from a bunch of package repositories as a regular activity...).
Git clone llama.cpp and build it, it's not hard.
https://github.com/ggml-org/llama.cpp/blob/master/docs/build...
literally just a few steps for the basics:
git clone https://github.com/ggml-org/llama.cpp
cmake -B build
cmake --build build --config Release
>Anything that suggests curl into bash just plain sketches me out.
How is it different than trusting any other method of installation? If URL has https and is from an author you trust i dont see the difference.
Package management provides cryptographic attestation over the entire process, including the scripts run. Nothing is arbitrary nor mutable in the default state. The files touched will be in a predictable place and the scope of privilege escalation is clearly defined.
But he’s not using a package, he’s downloading from gir repo and build which exactly same getting a bash script from the same repo and executing it.
It’s not. Try pulling updates after a rebase. Very much not the same as piping arbitrary disembodied text into your shell.
Except the first time you do it
>Nothing is arbitrary nor mutable in the default state.
What do you mean with this?
It's more about installation location for me. For these types of projects, I don't even install it. I just clone, build and run from there. I have no idea where the bash script is going to put things.
Which also makes it not obvious how to uninstall it. pip/npm/cargo etc have well known mechanisms for that.
curl|sh is convenient for container images I guess.
I mean it's probably not, I just haven't got used to it yet. It's about the same level of security as installing a windows app on win2000 25 years ago and blindly downloading a .exe off the internet and running it to get into the install wizard. But indeed I also kind of blindly trust that whatever I'm getting from the debian trixie officially gpg-signed packages isn't backdoored.
One thing I do not do as a matter of practice is install things with a ridiculous number of recursive npm dependencies.
I do not blindly trust anything, and come to the conclusion that downloading binaries from Debian trixie is a lot more safe. There is a world of difference between "curl | sh" and downloading from a curated package repository maintained by a respected community with well-defined processes.
Rendered text cannot be assumed to equal the underlying text, unfortunately
How so? As i understand your point, this would mean we cannot trust GitHub enough to return the same content in git clone vs curl?
As an example, webfonts can make rendered text differ from the underlying text that ends up on your clipboard.
Sure, but doesn't this assume that you cannot the publisher anyway? So why would you not trust their homepage but trust their source-code
Download and inspect it.
There are also prebuilt binary archives for just about any distribution and inference backend for the latest github release:
https://github.com/ggml-org/llama.cpp/releases
No need to compile unless you really need to.
This. I use mise's github backend `mise use --global --pin github:ggml-org/llama.cpp` to grab the release binaries for Linux, Windows and macOS.
Absolutely, and it's even worse when you are asked to blindly download and run software like the Brave browser, which is open source but exclusively packaged and distributed by an adware company with a questionable history. The software itself is open source, but independent builds seem to be discouraged and no Linux distro packages Brave as a result. I built it myself in the past, but it takes close to two hours on a workstation and isn't worth the effort to keep it updated.
so instead of downloading a script with curl and running it with bash, you should download it with git and run it with cmake? if you aren't reviewing the code between download and execution, what's the difference?
You can also install it through homebrew https://formulae.brew.sh/formula/llama.cpp
doesnt look like it is the desktop app, its just cli
You're not wrong about the appearance of sketchiness of that, but.... dude, come on. "It's not hard" is only true because you already know how to do it.
Exactly. That way of assuming everyone is a seasoned dev or sysadmin always baffles me. There are a ton of tinkerers out there, we are legion, and we are often reluctant to ask, especially here (as suggested by another parallel comment) because we don't want to receive an answer like this. Full circle.
If you’re unsure how to do it, you can ask others for help or Claude will tell you.
Point Claude Code at a repository and ask how to install it safely. You don’t have to know about make or cryptography of HTTPS or anything, really. It will walk you through the options and risk.
If you have questions about any part of it—i.e. you don’t recognize an acronym or deeply understand why something works—you can ask.
Or ask here! HN is filled with smart humans.
It's literally three steps, assuming you have the equivalent of the debian "build-essential" dependencies installed on your system for cmake, compiler. It's the exact three rows of my post, pasted one at a time into the CLI. The llama-server build guide page is actually pretty good.
I mean, sure, if there's people who can't figure that out, they're probably better off using a GUI that is a wrapper on top of somebody else's precompiled llama-server, like unsloth studio or lm studio. There's a good sized market for that and I wish them well.
You're right about all of that up until the point any one of those commands doesn't work and you get a garglblargl amount of incomprehensible error messages. You've been in the sauce for so long that you've forgotten what it's like to start fresh in these things.
Please give it a try before you assume it won't work? Tell me what platform you're on and follow the three sequential cli commands I provided, and if it doesn't compile I'll be glad to look at your error message and get it going in the right direction.
You're missing the trees for the forest, friend. I know how to troubleshoot this stuff. I wouldn't expect some random schmo who's just learning ubuntu for the first time to be able to.
This, "just run these three commands" attitude is exactly the reason why these curl|sh "installers" have become popular.
GP is saying "it's not hard", yet you seem to be hearing "you should already know how to do this without ever looking it up".
Those aren't the same assertions.
It's not a sensible assumption that a process must necessarily be difficult or complex just because you don't already know how to do it. There are an unenumerable number of tasks each of us don't know how to do and have never done before which are not difficult at all.
Yes, and that's exactly why the curl|sh exists; you don't need to know the weird esoterica of a 26 year old piece of software. Accessibility matters.
>you don't need to know the weird esoterica of a 26 year old piece of software
You really don't need to know much at all in order to compile llama.cpp. You can literally ask any half decent AI agent to do it for you or give you a step by step guide, and troubleshoot any error you get.
> Anything that suggests curl into bash just plain sketches me out.
Yeah, 100% and it's becoming more and more of a thing, see rust install for example.
OTOH, if you're installing llama.cpp, you're more than likely planning to run an LLM on your Linux box with an agentic harness, so a curl into bash thing might be the least of your security concerns, :-)
One way I prevent possible catastrophic fuckups is that the 'doing code work' box that runs opencode or pi or whatever, is its entirely own separate VM and desktop environment (running as a xen or kvm guest and with its own LVM logical volume as boot/root and /home disk), than the machine running llama-server itself.
The harness gets the openai-compatible endpoint fed into it to talk to llama-server across the network, but the VM has no access whatsoever to my personal files, mail, backups/deep storage, fileserver, Documents folder, etc.
> security concerns
Yeah I recently tried the coding harness that's recommended here, Pi, in a bubble wrap sandbox and was horrified to learn that it spams multiple warnings at you if you don't give it write access to its own config/extension folder... Everyone else is rawdogging it I guess.
I think the best course is a docker container or a VM, i agree with the sentiment but trying to restrict the permissions of these things is fighting an uphill battle, it’s probably best to let them reign king in a disposable and isolated environment
You are horrified it needs access to its own stuff? I mean, any software that supports plugins needs access to them, no? How is that unreasonable? And with bwrap that’s an extra cli option.
It's okay-ish for Rust because you only need the curl|bash for Rustup, which is meant as a development environment. On production you can, for example, use APT to download it from Debian.
Backdoored developers are ok?
Yes, as long as their development environment is isolated and they can't push to prod without a review.
What would you win? Cmake is capable of executing shell commands and you end up with a binary that will execute on your machine anyway. At the end of the day it is just a matter of trust anyway, isn't it? I personally use bwrap if I'm not confident about the source.
Cloning a repo and building it is not _that_ hard, but easy installation is often the thing that makes or breaks a product. I believe Ollama proves that point in this context.
Mostly agree but at least with git clone you have a hash and the malicious code has to be served to everyone, and GitHub is less likely to get hijacked by a malicious third party.
But yes, still trusting the project with arbitrary code execution on your machine, including build formulas that pull stuff from the internet and suffer from all the above anyways
it's also on the arch repos
I was a bit suspicious of the url but it is also listed on llama.cpp github
https://github.com/ggml-org/llama.cpp
Same, but it looks legit enough to me. Here is the git repo for the site with a link back to llama.app: https://github.com/ggml-org/llama.pages
(I still deeply distrust curlpipes in general though.)
ty for digging this up!
Yesterday I installed llama.cpp to test it with local AI Data Analyst that I'm building. I was also testing other open LLM providers: Ollama, Jan, vLLM, LM Studio. I had older NVIDIA card (RTX 3070) and llama.cpp instalation was smooth, contrary to vLLM which required me to reinstall CUDA drivers because by default it installed the latest one. I'm curious if there is a speed difference between the same open LLM model served with different runners.
> I had older NVIDIA card (RTX 3070) and llama.cpp instalation was smooth
what model was it that you were able to run with the rtx 3070?
I was able to fit only small models Qwen3.5-4B in RTX3070 which is not very useful for Python and SQL generation thought. When I wan to test larger open LLM models I often just use cloud resources.
Just FYI lm-studio is a GUI wrapper on top of a copy of llama-server that the lm-studio developers compile and distribute
If I said, “I want a setup that is usable for an agentic coding workflow, and it MUST be local”, what’s the smallest/cheapest option right now?
It’s _technically_ possible to get agents running on all kinds of setups but there seems to be an (undefined) floor for useful setups.
A lot of the stories people have about getting setups running on relatively low end hardware turn out to have huge compromises or run into issues on anything but trivial cases. I’ve found it hard to find a consensus. Or maybe I just don’t like the multiple thousand dollar price tags people are suggesting…
I actually have been exploring this very thing!
I think the best option right now, since Apple has raised prices and Mac minis are basically impossible to get your hands on, is to build your own micro-itx machine. I actually built a mini-itx machine, but it does restrict your options a bit.
The Arc series Intel GPUs are what I think make this possible. I built a machine with an Arc b50 - it runs Gemma 26b a4b qat at around 30tok/s with their MTP head and prompt processing sits at around 500 tok/s. The really beautiful thing about this setup is the entire energy envelope of this machine sits at 120w at full load - when idle, it's at 40w and i've done some work in ubuntu to basically intelligently hibernate, which drops it to 0 watts when not in use. You can use a raspberry pi and Wake on Lan to wake the machine up for a overall draw of around 5 watts when not in use.
All in all this machine cost me 1.4k to build - but if you used micro-itx instead of mini-itx parts you could do it for under 1k - it has just 16gb of ddr5 but you don't really need more if you use models that can fit in vram.
I think it's pretty incredible that you can run an actually useful coding agent on a machine with a power envelope that is less than an incandescent light bulb. If you go up to micro-itx you can do even large cards like an intel b60 with 24gb or a b70 with 32gb and run even more powerful models. For all of these intel GPU's you'll want to compile the latest llama.cpp version with SYCL support - they are getting speedups every day, so worth staying on the edge.
straightforward solution is probably Qwen3.6-27B-Q4 running on a used RTX 3090. price on those is unfortunately high, in fact so high that getting a new Radeon AI PRO R9700 might be a better deal
a solid step up from there is anything that can run Deepseek V4 Flash but the hardware ask there is a bit higher
Is llama.cpp (and thus llama.app) really that much better than Ollama? I've Only ever played with Ollama, so geniously curious to hear other's real-world experiences.
"Friends don't let friends use ollama" https://sleepingrobots.com/dreams/stop-using-ollama/
> Is llama.cpp (and thus llama.app)
llama.app is just an URL (for the "advertisement" webpages of llama.cpp outside GitHub).
> that much better than Ollama
llama.cpp is the real thing, ollama was a fork that remained inferior.
I was playing with ollama because of name recognition. I was finding pretty quickly that the selection of models on the "ollama store" was pretty stale, and gguf models on huggingface are technically downloadable but don't work as well.
The "friends don't let friends use ollama" article linked in another comment convinced me to try llama-swap. I find it easier to directly deal with gguf files. Hard to quantify, but the outputs of the LLM seem better too. Asking the same gguf the same question with the same chat harness, I subjectively find llama.cpp does better. Might be some different defaults. I haven't dug deeply.
ollama uses llama.cpp
The dev behind ollama is adamant that ollama doesn't use llama.cpp (based on a technicality -- it uses ggml, which is created by the same people behind llama.cpp and is the backend of llama.cpp)
He made such a big fuss about ollama implementing their own kernels and felt slighted about the online comments saying ollama didn't properly credit llama.cpp and it kind of left a bad taste in the mouth among the local inference community.
For me personally, it was this that made me avoid them at all costs: https://github.com/ollama/ollama/issues/11714#issuecomment-3...
Check this DLLM D project, a minimal, clean coding agent built directly on llama.cpp without Python, bindings or overhead [1].
The blog on the DLLM development [2].
[1] DLLM:
https://github.com/DannyArends/DLLM
[2] Teaching an AI to Know Itself: Building a Local LLM Agent in D:
https://blog.dlang.org/2026/06/07/teaching-an-ai-to-know-its...
This MacOS app used to be called LlamaBarn. Really excellent to see the fast progress being made.
Official repo, also has documentation how to configure server parameters:
https://github.com/ggml-org/Llama-macOS
Small tip, install llama.cpp with brew before llama.app, which will pick up the existing llama.cpp. That way it's easier to stay up to date with llama.cpp, since llama.app is on a slower release cadence.
Also, models installed with the hugging face CLI (hf) are picked up by llama.app automatically. The CLI will keep the model cache updated, e.g. when models get updated.
Llama.cpp became part of Huggingface recently.
I tried
and then Then I get: And the web interface says Maybe it gets killed by the OS because it uses too much RAM?When I try
It seems to work. Nice.I think I can probably run Gemma 3 12B on my macbook M3 pro with 18GB. The question is, should I do it? This small model is probably not capable of doing a lot or advanced coding or reasoning. What else could it be used for, since it can run locally and privately?
A quantized Qwen3.6-35B-a3b can run in a similar footprint to gemma 12b, but is smarter. It can do coding tasks, if you specify them at a finer-grain than with bigger models.
Context limit is far too small to do anything serious, tbh.
pro tip: if u are building ur own harness with python (recommended) , use "llama-cpp-python"...
I started with "llama-server" and custom stuff around it, which is great for single model setups.. but for multi-model harness with quick switching, llama-cpp-python is peak
I tried to run in on my Arc A770, but all of the binary releases I could find were compiled without OpenVINO support enabled. I tried compiling it myself, but after two days of the compiler running it failed.
Two days sounds like a lot, both llama.cpp and openvino only takes a few minutes to compile on any decent modern cpu.
the full set of llama.cpp binaries builds in under 5 minutes with an unmodified build workflow straight from their github page on a literally ten year old dual xeon.
It also even easier to get working and integrate into your system in a sustainable manner with NixOS. Do it by hand or throw an LLM at it, it will get you a declarative patch for your NixOS config that brings llama-cpp into your config that you can review and add under version control (no random `make install` build artifacts contaminating your system, no wondering "what was it that I ran? what are all these files? how do I do the same with a newer version?" a couple months later). There's also likely some build cache where Nixoids have already build what you want.
I had a great experience with llama-cpp with Nvidia backend on NixOS.
(Sorry for being that guy.)
I don't know how it works, but does Vulkan not work on Arc?
Does the A770 use the Xe driver? If so then it might work with the scripts that I've been using to build llama.cpp with SYCL support for the Arc Pro B70.
https://github.com/cptskippy/battlemage-llm-gateway
It's designed so that you can re-run the scripts to pull the latest updates. When Muse Glimmer was released the other day I just ran the 02 script to build the latest version of llama.cpp with support for it.
llama.cpp is like the ffmepg of AI, and one of the reasons I so greatly dislike ollama is that the latter completely obfuscates that they're a rebrand of the former. Georgi Gerganov and team did all the hard work; ollama is langchain-like VC-bait with a HF download wrapper.
ICYMI, llama.cpp was also VC funded. Search for "ggml" on this page:
https://aigrant.com
llama.cpp will happily download models from hugging face, btw.
Huggingface now owns llama.cpp, btw.
https://huggingface.co/blog/ggml-joins-hf
Ha the logo is an L made of negative spaces haaa.
Hey folks, a bit of a hijack; but I've taken to using the Kobold gui, which I'm liking much more than Ollama -- but are there any major benefits to going straight up llama.cpp? Yes, this is somewhat of a question about laziness.
Any success at transpiling it to C? Using the cfront transpiler improved with coding AI? :)
Ivorymist_04
There’s now a `llama serve` command? I had to do a double take in case I was reading the `ollama` website.
> No telemetry
Must be tough not to be able to monitor your own models!
(The odds that that tagline was AI-generated seem high.)
and? whats the point of this? Doesn't everyone already know about llama.cpp?
The last time this was posted was in May 2026 (https://news.ycombinator.com/item?id=48325941), and before that March 2023 (https://news.ycombinator.com/item?id=35100086). Maybe those were the times you learned about it? Nothing wrong with letting new people know about it too now and then.
no, if everyone knew about llama.cpp, ollama would be done already
Llama.cpp team has failed to make their tech easy to install and use for years.
Why can’t they figure it out???
This is like complaining that the mariadb team has failed to make their tech easy to use. It's a server side piece of software. There are plenty of user friendly GUIs or wrappers for it. lm-studio or unsloth are to llama.cpp as something like phpmyadmin is to mysql/mariadb.
But it is easy ... clone the git repo, make a build dir, cd into it, run cmake .., run build/bin/llama-server -m /path/to/model.gguf
Browse to served web page with chat UI....
By "easy" do you mean "very lazy"?
I'm confused, is this from Meta? There's no attribution anywhere. Surely releasing an AI tool called llama breaks their trademark if not
It's from https://github.com/ggml-org/llama.cpp -- not associated with Meta, it's been around for years, and surely they know about it -- so I would guess either it's not a trademark violation or they don't care.
> not associated with Meta, it's been around for year
This post (https://news.ycombinator.com/item?id=35100086) from march 2023 says in the title "Llama.cpp: Port of Facebook's LLaMA model in C/C++"
By "not associated with Meta" meant, as far as I know, the authors don't work at Meta -- not that llama.cpp is unrelated to Meta's llama model.
This site seems scam for not noting origins of llama.cpp and fails to quickly and clearly communicate it NOT being affiliated with GGML org.
From https://github.com/ggml-org/llama.cpp:
> Visit https://llama.app and follow the instructions
It's linked at the start of the README.
It seems that llama.app is a direct competitor to ollama.com
I can understand the desire for the llama.cpp project to want to own the end user relationship, it is true that previous to this they were a tool provider and not really owning the end user experience.
And ollama were sketchy about not providing proper credit to llama.cpp, even though that’s all they are, a wrapper for it.
Ollama uses the llama.cpp backend for inference. I find Ollama noticably slower. Llama.cpp has had a built-in webui (used as llama-server) for a long time now so have owned the user experience too.