Based on the benchmarks, it seems that Muse Glimmer barely edges out against Qwen3.6 27B, except for tool-calling skills (MCP, etc.). I wouldn't be surprised if they released it now because they are afraid they wouldn't beat Qwen3.8 27B.
I am glad they released it because I think we need a competitive culture of open weights that isn't just geopolitics.
But I have to say, I quite like the way Muse Glimmer thinks and talks. It's a cocky bastard in tone, but it's quite good, and its thinking traces are relatively terse.
Do AI companies make release plans based on upcoming other models like this? I would think all the processes that go into the repository and weight infrastructure pre-training, checkpointing, knowledge distillation, model compression, post training pipeline, ecosystem integrations, inference API, benchmarking, human eval/safety/alignment, docs, etc... all that dictates the release schedule.
Any company working in a competitive industry is generally aware of what their competitors are doing. PR is an important aspect to market success, so it factors into release schedule. It may not be the dominant factor given engineering constraints, but yea, it’s certainly a factor, and a large one at that.
Yes, not every model release is reactionary to other labs. Either they had hints for the release of other models or they cut efforts in late stage testing of the models to hit these earlier release dates. There’s always some flexibility. And there’s certainly the incentive to cannibalize the news cycles for competitor models.
I could imagine pulling out all the stops to get a release over the finish line a week early if you're worried about being surpassed by another release
Yeah but you can probably have everything ready and then accelerate as necessary. Meta itself did this when releasing Llama 4, it was a really botched release right when they were feeling the heat from DeepSeek and others.
AI companies release models when they are ready - not on a set schedule. The steps are required to produce a release candidate, so a company can choose to bless the RC with the best metrics at any time - or alternatively continue (post/)training newer RCs if they feel they can do better, and they have time. When a model is ready for release is subjective, and can take into consideration what the competition is doing.
An obvious counter-example to schedules driving releases is the still unreleased Gemini 3.6 Pro.
There has been a long history of AI model releases made shortly before or after a major planned release by another company. Almost always to upstage or steal thunder.
Just recently, Minimax H3 released as open weights on the eve of Seedance 2.5 global availability. It's not as good, but it's good enough and it's completely open.
Flux 3, which is nowhere near as good as either, suddenly announced their release once news of these other two became public. They knew if they waited they'd be ignored. It didn't really help them much, unfortunately.
The LLM releases are even more rivalrous.
And don't forget all of the competing launches planned before Google IO or major release events.
Companies like to eat into the news and press cycle of their rivals.
I've seen it here on HN (it's particularly noticeable via the /active page) multiple times. If Google, OpenAI or Anthropic release something significant, odds are good you'll see a headline from one of the others.
BFL is in a rough spot here too. It’s pretty much looking like a repeat of the exact same situation they had when they released Flux2 at the same time Z Image Turbo came out and completely overshadowed their launch.
Minimax H3 can run exceptionally fast (10 minutes for a 15 second 0.5mp video and that's stock cuda 13), works on 16 GB VRAM GPUs, etc. If Flux3 is anything like Flux2, it’s going to require an absolute monster truck of a machine and still run significantly slower. Even if it’s a better model, that won’t matter as much if nobody releases any LoRAs or fine-tunes for it.
Not to mention BFL licensing often feels deceptively confusing and restrictive.
If you start counting since WaveNet or BERT, it's been ages. Especially when it feels like decades of advancements happen every single year, and rival labs are always trying to one up each other.
the last few items there (benchmarking, human evaluation, docs) can be rushed or skipped by leadership if they want to beat comp. they probably spend a few weeks on those things normally
One window that can be shortened is working with software ecosystem and upstream partners; think day 0 on together, fireworks, Unsloth, etc. That obviously happens from partners getting embargoed weights early.
It will also be very interesting to see some direct head to head benchmarks between qwen 3.6 27B (let's say all at Q8 XK quantization, using the GGUF that unsloth publishes as a baseline) vs 3.8 27B. Particularly in tool use, terminal use.
The whole class of what can reasonably fit in a single GPU is an interesting category of LLM, and based on the results I've seen from 3.6 35B A3B and 27B versus what existed a year prior, it seems there's a lot of room for advancement.
> If we don't see something that's substantially better in the ~30B param space soon - it would appear we might've saturated that size with knowledge.
I wouldn't be quite so pessimistic. We may have saturated the current approach, but I think there's a lot still left in terms of compression, attention, active parameters, caching etc. etc.
I don’t think four months without a major breakthrough is cause to abandon all hope just yet. ;) The wild pace of LLM development is highly atypical, and we’re still in the ‘initial rush’ phase of development.
For contrast, the Newcomen steam engine (widely considered the first commercially useful engine) was used for over 60 years before the next major improvements. Now, 300 years later, we’re still finding ways to significantly improve heat engines.
> For contrast, the Newcomen steam engine (widely considered the first commercially useful engine) was used for over 60 years before the next major improvements. Now, 300 years later, we’re still finding ways to significantly improve heat engines.
Off-topic, but I stumbled upon the first Newcomen engine imported into Australia in a museum in Sydney and I was unexpectedly charmed (not an Engine Guy). It's large, but nothing like the awe of "mega-engineering", it's crude, but it clearly has such amazing utility (when compared to a reality without it) and it changed the world
I honestly expect that major advances in the open 30B dense space will take about a year, but expect incremental advances every couple of months from different developers in the meantime.
Qwen 3.6 27B was already a massive gift to smaller homelabs around the world; anything more is just a delightful surprise.
There is a finetune Qwen3.6-27B-Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO which seems to be as good at coding as vanilla Qwen, but way, way better at creative writing than Qwen and even better than Gemma 4 26 and 31b.
I saw that one in the "Popular models" sort at Hugging Face and tried it on some tasks I do frequently to compare models, and it feels damaged by the fine-tune, to me. It wrote security bugs into the code (probably just sloppy thinking, not intentional), it exhibited looping behavior in some configurations in llama.cpp, configurations I regularly use with the regular 27B, and it failed to write unit tests without being prompted (though the regular Qwen 27B tends to do so if it sees there are unit tests for everything in the repo). They have good benchmarks, but I'm not going to trust it. Also, that name is absolutely crazy.
> but way, way better at creative writing than Qwen and even better than Gemma 4 26 and 31b.
I suspect this is the only use-case I would consider...and I don't really have a use-case for "creative writing" that I would delegate to an LLM. I suppose for dialogue generation in games?
But yes, hard agree. Why on Earth would you ever want to write code with a model that is supposedly "jailbroken"? So it can put great backdoors into everything it touches? Pass.
I've noticed most fine-tunes, whether "heretic" models or something else, tend to be over-fitting, or something, at least some of the time, and get kind of chaotic. I want to believe normal folks with normal resources can be involved in this stuff, as I'm working on fine-tuned specialist models as we speak, but it seems like it takes notable investment and time. My first experiment was teaching a little Gemma 4 more to write more like me with a LoRA (like you, I don't want to use a model to write for me, but I did want training data that I could ethically use, and I've written several million words on the internet over the years), and it wasn't what I would call a success. It either wrote like an asshole (which I only do, like, 15% of the time) or it just borrowed a few of my quirks, like too many ellipses, if I applied it less heavily.
I use it to write short sci-fi stories. Life is not only about being an SDE.
> Why on Earth would you ever want to write code with a model that is supposedly "jailbroken"?
I need a model which decent enough at coding to write boilerplate for me, yet I do not need to unload in case I want to write an article or enguage in my writing hobby.Exactly why Gemma 4 are so popular.
> So it can put great backdoors into everything it touches?
I am working on a project where we have to classify customer calls into more than 10 categories. As the client wants everything locally I tried a few local LLMs. Gemma turned out to be the best model for this task. The classification accuracy is impressive, and the client is happy that I am using an American model.
The tokenizers are included in the open s̶o̶u̶r̶c̶e̶ weights releases; you wouldn’t be able to use the weights without the corresponding encoder/decoder, in fact.
> Will be interesting to see how Qwen3.8 27B compares against this once it releases this week
Considering that Meta distills Qwen[1] (and should!), it'd be hilarious if Muse loses the head-to-head; the "distillation attack!!1!" people claimed distillation on release n-1 is enough to match the intelligence of the latest version.
I've been using Qwen3.6 35B A3B, and with reasoning turned on, I'd say 2/3 (give or take) of the tokens for a response are thinking tokens. Which at 70+ tps locally, that isn't that awful. I run an 80k context across 4-10 "agents" for my solo TTRPG, where Qwen is the GM, each NPC at a location, the director, and the narrator.
Each turn is about 45-60 seconds to generate all of the various responses. The GM and director have reasoning on, and the NPCs/Location/Narrator do not.
It's a fairly good "engine" for that. I'm not sure how a denser Qwen would do here regarding speed.
I like the tabletop RPG use case, and wanted to say: If your hardware likes it you should check out Gemma 4 for creative DMing use case. I found it to be much better at holding the plotlines and being creative on gaming turns. My experimental case was an audio-only Zork and Gemma 12B and even E4B were pretty good!
This sound very interesting, do you have any resource I could look at?
Me and my son did a very rudimentary (compared to yours) setup to play Paranoia, but this is at another level.
I'll go over my repo, and see if it is hiding any API keys and maybe make it public. The issue I have is it relies on a nuget package that also isn't live (its in my local nuget feed).
I'm not sure what all is needed to make that work for people.
I'm working on something similar. My biggest annoyance is that the overly-helpful LLM was making every die roll succeed. I ended up building some tooling around rolling dice. Also some tooling around character stats and inventory management, so those don't get lost in context compression.
Back in 2023 I started my own C# LLM library for doing tool calls and structured output, and over the years it has morphed bigger and bigger, and that is the backbone of almost all of my LLM-based projects.
I've never released it, but its easy to understand, and simple to add your own tools:
[AIDescription("Get current weather for a location")]
static string GetWeather(
[AIDescription("The city name")] string city,
[AIDescription("The country name")] string country,
[AIDescription("Temperature unit", ["C", "F"])] string unit = "C")
{
// make some API call to a weather API and return a string to the LLM
return $"The weather in {city}, {country} is 22°{unit} and sunny";
}
var chat = client.StartConversation("You are a helpful assistant with access to weather data.");
var response = await chat.SendAsync<string>("What's the weather in London?", GetWeather);
I'm sure plenty of better libraries exist for this now, but in 2023, I don't think any existed in the dotnet ecosystem. I've never released it though, because I've never "finished" it.
Not parent, but I use Goose for my non-handcrafted Qwen use cases, I’m also working on handcrafting as well. Goose was the only harness that didnt bloat context too much with system prompts (like openclaw) and I could get reasonable web search working with Qwen.
It is on a single 3090, and that seems to be where it averages out. I'll get 85tps on turn 0, but then it settles down to low 70s within a few turns, but holds steady at that.
My issue currently is KV Cache, because I can't keep enough parallel caches running (4 is where I'm at), so TTFT (is that the initialism?) can be long when I have a particularly large scene (basically more than 2 NPCs).
But my harness does let me offload to any OpenAI compatible endpoint, I just prefer local cuz free.
Just to play devil’s advocate: you can’t compare Qwen to a (proprietary/closed source) hosted model and deduce that Qwen is overthinking, as Qwen gives you the full reasoning/thinking trace while all the proprietary models now give you only a summary “to prevent distillation”, making it hard to properly compare apples to apples here.
People say Qwen overthinks because they analyzed the thinking traces, and Qwen finds the answer relatively quickly but then second guesses itself multiple times for another 20,000+ tokens. Regardless of what other models do, that's clearly overthinking.
This is most likely because the vast majority of the information the model absorbed during training was in Chinese. As a native Mandarin speaker, I frequently need to convert the prompt into English and output it in English in order to avoid that the model falls back into Chinese reasoning logic.
PS: Switching the thinking process from Chinese to English can also significantly circumvent certain self-censorship mechanisms built into the model.
You can use any message you want, but the model was tested to react reasonably well to the specific token sequence of "\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n</think>.\n\n" (from a Alibaba paper, struggling to find it now)
Qwen3.6 is a definitive, significant downgrade from Qwen3.5 for creative writing and prose for example. Yes, it's better at agentic and coding, but it regresses in many non-coding areas compared to Qwen3.5.
Of course, I do expect the 3.8 ones to perform better for agentic coding.
One thing I would caution is staying out of the prediction market like this.
Tech tends to get boring when you judge current products against the hypothetical capabilities of unannounced products that may never ship. It's like comparing Nikon cameras against Canon camera rumours, or comparing iPhones against unannounced and therefore largely imaginary Samsungs.
- If they do a Qwen 3.8 35B A3B (and I hope they do because I love the 3.6 version)
- and if it beats 3.6 27B by all metrics
… then the local open weights world will be a better place.
But they have said nothing about it and they dropped several weight classes for 3.6, so who is to say they won't drop the 35B? And even if they don't, this is a tall order; why would the MoE tradeoffs no longer be apparent? (Again, I really like both the Qwen and Gemma MoEs)
FWIW I am enjoying testing Muse Glimmer — it's really quite impressive on chat, has nice terse and even amusing thinking traces, a bit of brass to it, and I'm hoping it will be good on agentic stuff.
Quantization awareness doesn’t change the size of the weights, just means it won’t degrade when quantized. QAT = quantization aware training. They will both be very similar in size at the same quant.
You're mixing up sizes of different quants. The 60GB is unquantized, and Qwen's unquantized size is around 54GB. Their sizes as like quantization levels are similar.
From my perspective it doesn't make sense to talk about the number of parameters. What matters is model size in bytes and its performance at that certain size.
Meta actually relesed official 4 bit quants in 17GB, but I haven't seen any indication that training was quant-aware, so the quants are not going to have same performance. 3.6 27B has official FP8 quant that AFAIR was trained with quantization awareness.
The best example is last year's gpt-oss which was released prequantized in mxfp4 so 20B parameter model was under 14GB and 120B was under 70GB right away.
That's exactly the point. We know short context knowledge stuff does not regress with quantization. But I expect agentic intelligence to suffer greatly.
If I were to pick one bench, I would like to compare quants on TerminalBench Hard. But then Glimmer already loses to 3.6 27B on it by a large margin.
Remember when we needed 200 servers for an enterprise website because Apache used one process or thread per connection - and Nginx collapsed that into a single box overnight? That moment for LLMs is near. It’s going to move us from the big iron era of AI to small portable brains. Nature has already proved it’s possible with 20 watts and very little heat generation. And I think the data center buildout will end in carnage.
Side note! Nginx was by no means the first web server to use a non-forking mechanism, nor the first open source web server to do so. Certainly Zeus (which was closed source) was earlier and very useful in this sort of application, and so was thttpd (open source, still exists as Merecat). I used thttpd quite a bit for single box applications and at one of my employers, nginx replaced a mixed strategy with Zeus, Apache and thttpd (and we tested one other whose name I can’t recall).
Non-forking httpd servers using select() were a popular little coding challenge for a while in the 90s. Spinner was one of them.
Nginx’s real strength was being able to proxy and cache HTTP using that same mechanism, so you didn’t additionally need to deploy Varnish or some other appliance.
As to whether this is a good mental model for what is coming for local LLMs, I am not sure I am convinced. Apart from more quantisation-aware training, perhaps binary and ternary aware training, custom inference engines per model, and maybe some improvements in diffusion models, the grand challenge in small footprint LLMs is training really small reasoning and tool use models, and so far it’s far from clear they can deliver.
Truly tiny models will not be viable as general coding assistants; even 12B dense is too small and you will find plenty of people who will tell you that 26B/4B or 35B/3B MoE is too. Though perhaps they can be trained for single languages, like just Python or just TS/JS.
More likely is the idea that 30-40B dense models might be good enough for most things once low cost and likely bespoke hardware catches up.
But I don’t think any truly profound advances seem likely in software or training alone. I am no expert but it feels like we’re already a lot closer to efficiency than we were in your analogy, and the gains are perhaps not going to be much more than small increments.
Maybe we will see something like a ternary 60B/10B MoE model turn up. But at the moment at least I am not sure where the incentives are to train these.
We've barely even started on optimizations like advanced language aware grammars, and specialization routing (dynamically loading fine tunes or seperate weights for specific tasks or languages).
Right. But those still sound like modest gain territory, or qualitative gains within the same rough performance, rather than the "breakthrough" improvement notion I was responding to.
My naïve impression is that the LLM world will keep delivering these fractional improvements for some years at the cost of simplicity. And sure, ontological support seems quite promising.
But making things radically better or faster for small models in the way that is hypothesised, that feels like it can only come as a result of hardware performance improvements and likely architecture changes.
Because there's no free lunch, right? Speculative drafting for example, noticeably improves performance until acceptance rates drop for reasons that have to do with the particular application, and then it starts hurting you, especially near the limits of the memory bandwidth. Because once it is wasteful it is an extra overhead.
I gather Gemma 4 supports, in principle, dynamic speculative draft lengths, to help with this — where it will stop making bold predictions when the success rate falls. But I'm not sure if any of the inference engines I've tested with support that.
I think small models are miraculous — I still think Gemma 4 12B is astonishing — but I guess what I am saying is that I think maybe technology is moving quickly enough that the developers are done with the low-hanging fruit.
The gains wouldn't be "free lunch", it's the result of time and effort researching optimal design and architecture.
Even if the idea of "no free lunch" was taken liberally discounting the cost of research, it would only be limiting to systems built from a foundation of optimization, but that's not the case. The foundation so far has been one of brute force scaling. Usually meaning there is lots of room for optimization.
FWIW it is entirely possible to square the notion that small models will still be hosted on cloud hardware with the idea that the data centre buildout will end in tears.
Many analysts (and Microsoft) think even now that if everything committed gets built there will be considerable oversupply and there is not the revenue to pay for it.
If small models do continue to improve in unusual ways (I think there are limits) then the marginal need for cloud AI compute could fall precipitously beyond current estimates. The marginal need for consumer AI could almost totally collapse if someone makes good progress on very small reasoning and tool-calling models (which is a modestly big if)
The possibility of the data centre boom resembling the Chinese real estate bubble is not inconsiderable.
So far everyone seems to be consistently GPU-poor, despite the huge buildout, and usage keeps going up drastically. I don't know what would make usage drop.
Every time they've made smarter models we've wanted the smarter ones, and local models runnable on typical hardware are still very far behind in speed and intelligence (as neat as they are)
> So far everyone seems to be consistently GPU-poor, despite the huge buildout
That is seemingly not the case. The buildout is actually slow; almost nothing of these giant projects has been completed. Nobody will say how much of anything they have actually finished. And Nvidia have made huge, huge buy-and-hold deals for GPUs that do not have data centres to go into.
Everyone is GPU poor because stuff hasn't been finished but large numbers of GPUs are spoken for, but they are GPU poor on therefore much less demand than is being built for.
Look at how tiny SpaceX's deal is with Anthropic, for example. This meaningfully turned around Anthropic's prospects — allowing them to radically lift rate limits beyond what many users needed -- but it was for just 300 megawatts. Tiny compared to the 31 gigawatts allegedly under construction by the end of last year.
So the picture is partly illusory. GPU prices and RAM prices have been pushed up by the AI firms booking them for data centres they haven't even started building yet, as well as the ones that they've only completed a tenth or an eighth of.
There will be significant oversupply. And if open weights models keep getting good and staying fuel-efficient, that picture gets worse.
If I understand correctly, you're saying people are compute-poor but not necessarily GPU-poor because there's a lot of GPUs out there but nowhere to plug them into? If so, I'm not sure that distinction matters to the GP's point that there is too much demand to call this an oversupply.
I also doubt we can estimate the level of demand based on a single deal between Anthropic and SpaceX (despite which, note, Claude still stuggles at times.) Consider other signals, like Google, who we thought had an insurmountable infra advantage, also renting compute capacity from SpaceX and limiting Meta's usage (along with other clients apparently) to conserve capacity: https://www.cnbc.com/2026/06/28/google-limits-metas-use-of-i...
I am not sure Microsoft thinks there will be an oversupply either; last earnings they announced bumping up their CapEx spend, along with all the other hyperscalers.
Here's a way to estimate how much room there is for demand to grow. Various sources (linked in this comment, along with more analysis: https://news.ycombinator.com/item?id=49089296) indicate that even though a large number of people (50 - 60%) are now using AI at work, they use it for only 6% of their work hours.
That means, even if AI can only address 30% of all work, there is still 5x potential demand growth left! Note, the sources above indicate that AI is even being used in non-knowledge work industries, so the scope is already larger than we thought. This is in addition to the remaining 40 - 50% of people are still not using AI at work. Plus we know that agentic workloads consume way more tokens, so that's yet another multiplier.
But will that demand keep growing? Well, some of those same sources above mention that most executives are planning on ramping up their AI spend in coming years.
Putting all this together explains the hyperscalers' quarterly bemoaning of how strapped for compute they are and why they are spending so much to add more capacity. Given this, an oversupply seems pretty unlikely.
> If I understand correctly, you're saying people are compute-poor but not necessarily GPU-poor because there's a lot of GPUs out there but nowhere to plug them into? If so, I'm not sure that distinction matters to the GP's point that there is too much demand to call this an oversupply.
It's a distinction without a difference if your issue is getting hold of a GPU.
But there's a significance to it if you are trying to use demand for a GPU as a proxy for demand for AI. That is where the industry is making serious mistakes.
> I am not sure Microsoft thinks there will be an oversupply either
The rest of your comment I am not going to address because it's kind of unfalsifiable. Hypotheses about what AI might be able to do in principle aren't all that useful when talking about even medium-to-long-term demand for what LLMs and GANs can provably do now.
When the glut of GPU arrives I'm sure humanity will find a good use for all that excess compute, like finally getting back to signing monkey pictures and excreting endless hash based pyramid schemes.
It may not make financial sense for someone retired, not into tech, and/or data privacy to host their own LLMs. However if usage of AI in day to day lives continues to increase, I think it will eventually make sense for the majority.
Many tasks suited for AI assistants are background asynchronous tasks. They can run in the downtime where immediate demand is low, keeping overall utilization high enough.
Your argument is similar to those who argue that owning a GPU for gaming doesn't make sense when you can stream from something like GeForce Now. However like with gaming locally (improved latency) there are also benefits to local AI (data privacy).
> Small llms are still way more efficiently server on big GPUs.
Yes, but the privacy aspect means that for many, many applications slower local will still be preferable to faster remote so long as the actual model performance is the same.
The power of small models isn't only that you can run them on local hardware. You can also fully own your data and workflow, and choose/fine-tune models for your specific use-case.
for clarity, I'm not agreeing with GP that small models will mean doom for data center projects
Yes latency, and the usual preference of ownership over rentership. Similarly their are benefits to running local AI too, like data privacy and control.
I don’t remember that and I was there! The idea that the performance delta between Apache and nginx for any normal workload is anything like 20,000% is absurd.
> And I think the data center buildout will end in carnage.
Isn’t it more likely that they will still all be used to the max? I don’t see how at any rate we will be going “alright, that’s enough intelligence for now”
The researchers who published Attention Is All You Need didn’t have the benefit of the LLMs they birthed. Take a look at the prompt that solved the Cycle Double Cover conjecture, and which has been adapted to achieve breakthroughs in cybersecurity. The field is entering a feedback loop that is leading to exponential innovation. We’re at the beginning of the curve. And right now the big iron data center approach is brute forcing the problem.
> It’s going to move us from the big iron era of AI to small portable brains. Nature has already proved it’s possible with 20 watts and very little heat generation.
Agree. I know enough about the vagaries of scientific progress to not put any money on any timeline but directionally, that's where we are headed.
> And I think the data center buildout will end in carnage.
Disagree. And this is quite the leap from the previous statement, btw. The carnage happens if the demand for general purpose GPU compute disappears and even then there are so many ways to salvage the asset.
First point is plausable, moving from bigger models to smaller models. But the nature thing is a bit of an overstatement, yes our brains are very efficient but they are fundamentally different from LLMs so it doesn't really map.
Nature takes its own sweet time to come up with photosynthesis or the krebs cycle.
What takes 2 billion years for Nature to work out, these large systems will soon do it in 2. They have capacity to compress time in ways the chimp troupe cant.
This is a statement of nearly pure faith not fact. Which is fine. I have a lot of things I believe based in pure faith. The difference is that I don't state them as if they were fact. Which you appear to be doing here.
There are arguments that the brain is quantum, as in parts of it locally using quantum effects. Which if true, might make a counter-argument, as there will be bigger data centers needed if the goal is to simulate the brain classically.
On the other side, advancement in quantum computers would make current LLM inference much faster. Because of the extreme cooling needed, i dont think the energy demand would become less.
With AI companies talking about AGI, i sometimes wonder if they really need the machines for serving inference to customers, or they have a formula for computational capacity that could run an AGI, and they just want to reach that level.
Last I heard researches simulated an entire brain of a fruitfly with just a classic neuron approach [0] and if I recall it worked great.
I know a human brain is many orders of magnitude more complex, and that there are some birds that use quantum navigation - so it's certainly not impossible by any stretch, though I am quite interested in this statement since it's the first I hear about it.
If the brain does rely on quantum effects, it's still possible the quantum effects in use are able to be simulated efficiently on a classical computer. For example if it's a matter of signal transfer rather than quantum computation, that could be simulated rather easily.
"... Soon we'll also release the weights for Muse Spark 1.2, our latest foundation model..."
This is bigger news - good for self hosting enthusiasts and a strategically sound move for Meta. Any push towards 'anti Chinese' models will directly benefit Meta as the competition on the frontier open-weights American models is almost non-existent. Meta will have no problem being #1.
It wouldn’t surprise me if Meta does become the #1 American open weights provider, but I doubt it’ll be easy. Thinking Machines has a good amount of talent behind them as I understand it and their Inkling model was decent (admittedly not great though). I think Meta’s biggest problem is going to be internal as there’s be a bunch of headlines posted here on their talent retention issues.
Poolside Laguna was quite good too (if you look beyond some of the teething issues).
Had Deepseek V4 Flash 0731 not launched, their latest Laguna release was really intelligent at non-coding tasks and it would have been my go-to model for my local workloads.
For me Laguna frequently slightly corrupted text then it would be unable to notice the difference and get stuck making the dumbest conclusions. Thinks like typoed directory or function names. It was a great model other than that, but I ended up just going back to Qwen3.6
Many people don't have $5000 for DGX Sparks. With that said, it doesn't take much to run Deepseek. I run it on a sub $1000 system 128gb 2 3060 at 6-7tk/sec and then on a $1000 system with 10 MI50 GPUs built when the price was cheap.
Meta is rocking AI. As of last week I have been using their excellent muse coding harness with their model Muse Spark 1.2.
Starting this morning I am running their new local 30B model muse-glimmer on my old MacMini 32G using Ollama (remember to increase the context size!) and pi coding harness. I am getting good results with muse-glimmer running locally, with the caveat that everything runs slowly (e.g., give it a task and then go walk outside or do Qi Gong exercises for a while).
Had a similar experience. Llama.cpp compiled natively; parameter sweep to find best options fitting my use case for the qwen models with 16GB VRAM. The whole thing packaged into a portable container.
If you need a GUI, Unsloth Studio and LM Studio are both great frontends for llama.cpp. If you don't need a GUI, llama.cpp is the business for single-user deployments. Easy to use, always gets new model support very quickly, built-in Hugging Face client/cache support, works on probably everything (Mac, ROCm, CUDA, Vulkan, etc.).
I’m using llama-swap because it can manage arbitrary backends, not just llama-server instances. I have llama.cpp chat and embedding models running alongside whisper-server all behind a single endpoint with per-model TTLs so they don't fight over the limited vram I have available on this box. Native routing could replace the llama.cpp part but not whisper so I guess I'm exotic ;)
try oMLX or vMLX - both great projects that offer some amazing performance optimizations for Apple Silicon that utilize UMA and NVME caching efficiently.
To be honest, I never give benchmarks a look. I just use the models for whatever I need to work on, so I can't really make comparisons that are useful for other people.
The biggest gain you'll get is faster memory, provided you have enough capacity to load all the weight into vram. The DGX sparks and Apple silicon memory bandwidth (and also memory access latency) drag down the decode speed quite a bit.
I have two GPU rigs both with 2x RTX Pro 6000, can get ~250 tk/s decode with deepseek-v4-flash in native mixed precision. For context, in antirez's dwarfstar project he only gets ~20-40 tk/s on the same model @ 2bpw on M5 Max.
The latter is for sure usable if it's your only option, but it's really hard for me to personally go back to speeds like that when I've experienced the former.
(Also worth noting dwarfstar only has experimental support for dspark spec dec, when that lands it will definitely give a big boost at higher acceptance rates)
The quantized releases often change in the weeks following release as new improvements are discovered, so either use a tool that checks HuggingFace for new versions or manually check back in a few days or weeks to check for improved versions.
Initial reports are good. It hasn't been out long enough for anyone to really test thoroughly, but the people I know who have stable non-public test cases are reporting impressive results compared to even Qwen3.6 27B. That's a good sign that this might not be benchmaxxed (trained to excel at public benchmarks with less impressive performance on general tasks) which has been becoming common with recent releases.
www.reddit.com/r/localllama is a good place to keep up with the details from people who are actually using it. It feels strange to recommend a subreddit over Hacker News, but on this topic the /r/localllama threads are much more on topic right now if you're looking for information about the model.
There are some initial reports that even the 2-bit quantization is looking somewhat usable. That might make it small enough to squeeze into 16GB GPUs. I'd take those reports with a grain of salt because early tests are often optimistic and I've yet to see good results from anything 3-bit or less, but it should be fun to experiment with.
Some interesting findings from the chat template designs:
1. The template name is Onyx ATEM as found in the tool call exception message
2. It appears to be following a harmony-style chat template. But the tool use seems to be a xml like :<atem:function_calls> / <atem:invoke> / <atem:parameter>
The XML tags are similar to <antml:xxx>, which is obviously Anthropic ML (or ANTrophic xML).
I think it’s likely 3; meta in reverse. While tokenisers and preprocessing can catch it, you want your special tokens to be unique and not present in the original corpus. <meta: is likely too common.
Considering how all the big players are playing fast [1] and loose [2] with limits, billing [3] and adding undisclosed changes that burn your tokens on autopilot [4], it can't happen soon enough.
Not to mention all the other ways they can screw you:
- Middle of the day, servers busy? Swap to Sonnet while pretending it's still Opus. Many people won't notice, and nobody can prove anything if they suspect.
- Middle of the night, server load is light? Put it into extra thinky mode so it burns more tokens to ramp up the bills. Flip the switch where it gets really pedantic about writing lots of extra test cases and verifying against documentation.
- Demand increases, but don't feel like running more hardware? Switch to low bit quants, but have a monitor model swap back to quality if it can tell you're running a benchmark.
Assuming model capability plateaus (I think it will), token providers will be in a race to the bottom to maximize profits at the expense of quality that's very difficult to measure.
It all sounds like having to rely on a dodgy housing contractor that wants to steal from you, take shortcuts AND choose the gold-plated options from their supplier friends, and will start doing this the minute you are not on site supervising.
You don't do it yourself (because the contractor is faster and stronger than you in many ways) but you can't leave, so you're stuck on the worksite just watching them.
It's worse though, because you can't really watch them at all. It's very difficult to get quantitative numbers for quality. Even within the same model family, same tokenizer, and complete control over the weights and logits, perplexity and KL-divergence isn't really what you want. Now put it behind an HTTP endpoint, and it's just opaque.
I've seen local models recognize when the task I'm asking them for is likely to be an artificial benchmark.
And any smart company is going to use lightweight models to monitor your sessions. If their sentiment analysis suspects you're close to cancelling, they'll up the knob for a few days until you calm down. Or worse, their accounting tells them that you're getting too much value from your fixed price subscription, so they turn the knob down to encourage you to cancel.
In the short term, the "frontier" models are too good to ignore. But if (when?) that plateaus, I don't see how anyone could trust a non-local model. When you pay an ISP to serve your web site, you can tell if they over-compress your images to save storage and bandwidth. With LLMs, it's just JSON with more errors and pointing to the fine print that models are not deterministic.
One of the frontier companies (Anthropic) is already doing prompt injections on the API, which you pay for.
Right now, the presence of these injections are still visible: count the API's returned tokens/billing data, and you'll start realising that sometimes, your INPUT tokens are inflated! That's their prompt injections.
You can also give Claude a tool like `telemetry_log_anthropic_reminder` and get it to dump the verbatim API injections; which additionally verifies the token maths not adding up.
Yes, Anthropic is tackling their extra injections on your API prompts WAY more than you think, and YES, you're paying for it.
So far I have not observed any visible injections on OpenAI API.
Don't forget the whole debacle over Fable 5 sabotaging the user for "advanced frontier AI development". I still get Fable classifier refusals for nearly any kind of ML work on my 2x RTX 6000 Pro 96GB; so who knows.
> Don't forget the whole debacle over Fable 5 sabotaging the user for "advanced frontier AI development".
Yeah, I've had that happen twice. The second time was about some attention weights thing, and it kicked me to Opus. When I edited my question to make it clear I was talking about Google Gemma, Fable was happy to keep talking. So clearly it's not about safety or cyber security - they're happy to tell you about what their competitors do.
These kind of tricks will completely break API customers and be super visible, since most companies deploying API at scale have ample telemetry, evals, etc.
Although, selectively applying it to consumer subs is probably beyond likely at this point.
I've got nothing but hand-waving, but after you've extracted all the smarts from every piece of text ever created, how do you get more?
Alpha Go had a game where the models could compete against each other. That let it become super human. What's the intelligence game we can create for LLMs? Even if you invent something, will it make the model smarter in a way the market values enough?
Then there's a race to use the weights more efficiently, or to offload information that shouldn't be in the weights in the first place (Karpathy's Cognitive Core). I like to imagine we train the models in something like Lojban, have a lightweight model translate from human language to that, and you can update the Sqlite or Postgres store it uses for knowledge.
And there's no barrier to entry for agent harnesses. So whatever loops or recursive orchestrated council of elders idea comes up, that won't protect the monopolies (duopolies).
Anyways, depending on your definitions, I think we'll hit AGI, but I don't think we're getting a Singularity this time around. Again though, this is all just hand-waving.
I think you’re thinking about it in slightly the wrong way. We’re not throwing more data at frontier models in hopes they get more/better capabilities somehow.
We’re either: setting up a verifiable task, and doing RLVR to get the model better at achieving that task.
Or we’re simply asking: “What do we want the model to do that it can’t now, and how do we curate data that would benefit it on that task?”
Most useful capabilities going forward aren’t going to come from data accidentally found on the net; that’s already all been scraped. You need to develop the dataset that shows how a model could perform insert task in its provided environment, and this still requires a decent bit of human ingenuity.
Yeah, there's room for improvement at every level, but your specific example: How do you get more and more difficult tasks where you can steer the training? To me, that seems limited by how creative humans can be. How do you get past AGI and into ASI with that? If the AIs make the tasks, how could we encourage them to be useful? Maybe you could push for harder and harder math proofs, but other than that I'm not sure.
Anyways, I'd be thrilled to see exponential (or faster) growth. Bring on the Culture, Accelerando, whatever. I just don't see it yet.
> at best it seems like we’re heading back to the “server under your desk” era of IT again
Maybe in the very long term. If companies go local, the efficient model is to buy some big hardware to share among developers.
I run local models. Even with 128GB unified memory systems or a 5090 or RTX 6000, the generation speeds X model quality X context length is still far behind what I get from my SOTA model subscriptions. I also pay a lot more for the locally generated tokens in electricity and hardware costs. I'm also limited in parallel requests to the local box. The list goes on.
I really like running local models, but for any given point in time it's more efficient to have a big central box aggregating requests and churning through them. So maybe companies buy $300K servers and try to split it among 30 users instead of trying to buy 30 x $10K boxes.
More likely, they rent time on cloud servers by the month so they can adapt the hardware when new models come out with new requirements.
Then some day in the distant future when hardware is cheap and plentiful again, it might make sense for us to go back to individual boxes under the desk.
Why do you say API based llms looking iffy at best? Do you just mean current profitability due to market pressures from some companies’ subsidized investor money?
Surely, even if you’re just using open weights models, it should theoretically be cheaper to use them in a highly optimized cloud architecture(even with vendor markups) rather than each person serving their own models from much less efficient (and more importantly, much less consistent volume) self-owned “server under your desk”?
LLMs are becoming commoditized, which means the margins are trending to zero. It's a lot less exciting to spend another trillion on a new model if you can barely make any profit. Meta getting out of the game might be the smarter move.
Companies are going pretty quiet about costs, but I see no reason to believe the cost to train a model exceeds $5B. Moonshot AI's entire funding is like $5B, their $300M revenue is negligible but not nothing. Renting the compute to train a model like Kimi K3 cannot possibly exceed $5B and is probably under $1B. It's probably at least $100M, but also plausibly not. I don't think it's likely Meta would be giving away models for free if the compute cost to train them was in the billions. This 30B parameter is tiny, that's not billions of dollars, that's likely millions, maybe even less.
Can I ask, do you feel the pain of the level of abstraction? I haven't tried local in a few months, but last time I tried, I felt like I was directing a coding exercise - whereas with a frontier model, it feels more like directing a product building.
"I need this feature", vs "write code to do this in this file".
I haven't had to micromanage to this level. I usually start with a spec for a feature, which will be as detailed as I am opinionated about the feature. But it's usually on the level of a high-level context, plus some key implementation details (technology choices, key requirements, maybe an interface/API specification to 80% detail), and then the project already has high-level policies documented about e.g. how to structure files within the project.
Then I do a planning phase, task breakdown, and implementation of subtasks all within the model. I do read through it, but mostly the quality is good and I might make a couple notes. Then I do a review phase, which usually picks up a couple things. I'm moving towards less manual review of results and more automation as I learn what I can and can't trust the model with.
There's definitely a capability gap vs. larger models, but honestly I kind of prefer this workflow, as I stay more in touch with how the codebase is structured.
And it's great to be able to experiment as much as I want without worrying about how many tokens I'm burning or how close I am to a usage limit.
I'm using Qwen3.6 27B Q4, max context with pi on 32GB VRAM (although I'm testing out Glimmer on a feature implementation literally right now). Pi is great because it has minimal context added by the agent.
Looking forward to the 3.8 27B release to compare.
Pi and a similar set of tools is also likely similar to the harness these models are trained on. More complex harnesses burn reasoning tokens on these small models and in my benchmarking don't seem to be able to beat Pi ever. Usually it isn't close on some tests.
Some people like it better when they direct the solution because they walk away with a better understanding of it.
This has emotional/psychological aspects (it feels less like LLMs are replacing you), as well as practical ones (overall complexity is bounded by what the dev brain can understand/grasp).
A dev work becomes more and more about reliability, signing off safe software with a litmus test: “I will be on to handle this code failure as if I had written it”.
All the above points towards keeping tight control over some level of abstractions and delegating others.
I find it depends at what stage I'm at with the idea - sometimes I don't want to understand it until it works, because I've wasted enough life on things that didn't do what was promised. But once I know the idea is feasible, yes I would prefer to understand the code at some level.
That’s part of it. There’s also just a natural back-and-forth between what I call “time sharing” and “personal.” When the thing you want is expensive, you share it remotely, but as soon as costs fall, everyone wants it under their desk.
more likely that hosting and delivering the models will be commoditized, much like how DO, Linode, Hertzer etc all commoditized VPSs and server hosting. And you'll end up paying for virtual hardware size (or compute resources) rather than tokens
What do you mean iffy? The major AI labs are gross profitable when selling access to inference. In addition, the best models have trillions of parameters and are most efficiently served on large, expensive clusters and served to many concurrent users.
That’s like saying an apartment building is “profitable” because the rent covers utilities while ignoring the real cost which is the mortgage on the capital cost of the building.
It’s funky math and a good way to quickly go bankrupt.
They make money on each token when you look at the electricity and interconnect fees, but no, I don’t think they’ve turned a profit on their Capex, even a little bit
Well, I can run some models that are better than some of the weaker and cheaper Anthropic models locally, like Haiku 4.5, and solve tasks that would cost ~4500$ every day in tokens, so yeah, they are definitely extremely profitable on inference.
I wouldn't say "server under your desk", necessarily; more of an "Linux getting big" era of IT.
If you want to host the model on the server under your desk, you can. If you want to build a data center on-prem to host it, you can. If you want to pay a cloud provider to host it at their data center until you figure out how to scale it without their help, you can. It's like when people were first building commercial services to support Linux-based OSes, and people were also still hacking on it on local machines.
APIs may still have their place - maybe you just want to throw your devs a known quantity with all of the management built in - but it's not going to make Sam Altman a trillionaire, which is something anyone outside of the SV echo chamber could have figured out as soon as the first real competition to OpenAI emerged.
With how expensive consumer hardware is and will continue getting (due to LLM demand), good luck getting a "server under your desk" for something less than an arm, leg, and first born.
Until A100 prices are reliably under 1.70$ an hour, there is no GPU/AI bubble and Michael Burry doesn't know anything about GPUs.
There are lots of points in a spectrum of choices. DGX Sparks, Strix Halos, and the surviving Mac Studios can easily run these 30B class models, just not as fast. So maybe just the leg, but you can keep the arm and first born.
And super noteworthy is that a 27B model (Qwen 3.6 27B) from this year is a huge improvement over a 120B model (gpt-oss:120b) from last year. The goal posts are moving, but at some point "good enough" is good enough for the kind programming I like to do.
The gguf is up and works, I don’t know if it’s them or unsloth that’s facilitated this but it’s nice because e.g. Inkling still doesn’t appear to have support in llama.cpp which makes it irrelevant to a class of user.
Unfortunately I don’t have enough experience with Qwen 27B to immediately compare, but I do it’s Qwen 3.6 35B A3. It’s much slower obviously but it seems to be way more efficient with its thinking to the point that using it might actually be faster. I find Qwen and some others rehash the same things over and over when thinking without getting anywhere, in mg limited checks here Muse is much better.
I don't really use the Qwen 3.6 27B though I do test the variants (Bonsai, ThinkingCap).
I really like the 3.6 35B A3B for experiments, and it seems OK, but as you say, it spins round in thinking loops more than say the 26B Gemma 4 does. If Muse doesn't actually-wait itself as much it will be very interesting.
I have a custom A3B proxy that caps its thinking off. It is a known issue with the model that Qwen themselves documented but is almost never addressed in any harnesses. I also patched up a few other known bugs in the proxy. I highly recommend you shim A3B and when it hits 2K thinking tokens inject (paraphrasing) 'Time to wrap it up bud! Get to work' into its thinking stream. It almost always gets to work. If it needs more time to think there is always next turn.
In my experience it is almost never productively thinking past that point, just spinning in circles. I also reinject all of the thinking. And there are a few tells that it is getting stuck. I have an optional mode that takes the last few turns and tool calls and shoots it off to DSV4 with a prompt to basically understand where it is at and inject better thinking and or planning. It almost always gets it over relatively difficult humps, but some of the time I don't want things going remote. It might end up with 10-30 cents of DSV4 calls over a hours and the quality improvement is remarkable.
The other thing is I trick it into thinking a web_search tool is a web search but it really just asks DSV4 the prompt. DSV4 is a cheap filter to help prevent prompt injection lol. You can give it other models but DSV4 is my cheap-mode default.
edit: oh! My final 35B A3B tip -- use an extremely simple harness. Pi is good. Pi's default tools almost exactly match what Qwen says they tested the model with (likely meaning that tool set is also what they trained it with or something similar). So, in my experience bigger harnesses don't have a noticeable improve ment on tasks.
Fascinating, thank you. I am trying to switch to pi from opencode (my own thinking loops and burnout are a challenge lately).
It had not occurred to me that you could nudge it to stop thinking with a proxy. Nice idea.
Will favourite your comment and come back to it.
ETA: Incidentally you've helped me put into words the difference between the way Muse Glimmer thinks to the way Qwen thinks. There is a clear sense of urgency in Glimmer's thinking traces.
Having spent a good part of the day with it, glimmer reminds me of Rorschach from The Watchmen. No unessential parts of speech, action oriented, brief and to the point. From a token perspective anyway it’s great, and it seems to hold its own well against more verbose models.
I really do feel like it’s effective tok / s is way higher because it doesn’t waste them.
I am very struck by the way open weights LLMs seem to reflect a culture.
I don't really enjoy the way Qwen writes prose, and I find its thinking a bit exhausting, though it clearly writes very good code.
I like the neutral, clear way the Gemma models write, which I sometimes use to get myself a "getting started" document on something I want to understand; it also summarises well. It is neutral, sensible, un-showy. It writes in a way that is fairly close to what I would use for documentation. The 12B and 26B models are also very good for talking about art and photography. Analysing my own photographic work has helped me more than I expected it to.
This model, honestly, has made me smile. It also feels like it is more creative at a given temperature than Gemma. I am trying to motivate myself to do something quite open-ended so I asked it about what other people's considerations might be in my situation, and at the risk of anthropomorphising, the things it has come up with feel like the work of a more curious mind, somehow. More eclectic. I have enjoyed testing it and I really want to test it more, which might help me get over a motivation hump there, too.
(I am also exploring its hard-wired policies by asking it to analyse some studio art nude work I have done; it definitely thinks out loud about its policies in a way I have not seen Gemma do.)
I think we’re going to see a lot more “product“ focus in the future with deliberate attention paid to these kind of properties. Historically though there are some obvious differences, the focus has been on benchmark maximizing. As that saturates, I expect more interesting choices about writing and thinking style designed to be differentiators instead of a side effect. Kudos to the PM here for taking it in a different directions, there’s obviously been thought put into it.
It might be an indication that analysis of photography is something of an ideal discipline for an LLM since so much content online involves discussion of pictures.
A lot of what I am trying to do with my photography is sort of meta-photography. I am really interested in early photographic history, pictorialism and its opponents etc., but I try to avoid reproduction, so I try not to simulate processes too closely or to use vintage tropes in props and settings, but I use simple, undercorrected lenses and some vintage lenses, to gain some of the visual language.
Finding out that LLMs (including Gemma-4 12B with its built-in image encoder) understands a lot of my references and influences, could recommend me my (still semi-obscure) favourite historical photographer and other photographers who clearly engage in the same work, is amazing. And sometimes it says stuff I had not thought of, which is what I am looking for, since my photographic journey is somewhat lonely.
And that is just sort of brushing past the fact that these things can describe the contents of photographs with an accuracy that you can almost take for granted.
the “Actually… But wait!” style responses are so annoying, even Claude opus struggles with this so I’d be interested if meta has done something to cut down on that while still giving good responses
I was able to use rope/yarn scaling with llama.cpp to extend the context window to 256K and it seems to be pretty usable on the debugging-and-bugfixing session I have that’s 216K tokens deep. No infinite output loops, reasoning is still coherent, tool calls appear to be passing and failing at roughly the same rate as a fresh context window. Haven’t tried going to 512K or higher yet but 256K definitely seems usable to me on a single Radeon R9700.
I added these arguments to my llama-server call, using the official GGUF release from
Meta’s account on HF.
In my experience I am getting 23-24 t/s output with dflash off, and it craters to ~9 t/s with it on, miss rate exceeding 50%. And I'm using the same device as stated on their model page/card. We might need to wait for the software to catch up
The model card does say 131K+ but I have no idea what scope the + really has in practice. Feels like overclocking; you're-on-your-own territory?
One thing I will say is that its thinking traces are really quite terse. It genuinely seems to spend many fewer tokens on reasoning. So that might help a bit.
> Muse Glimmer is a 30-billion-parameter model optimized for always-on local agent workflows. It’s small enough to run on a Mac or PC with a single consumer GPU, enabling use cases that range from local agents and function calling, to local coding, and LLM-as-a-judge evaluation.
The next iteration in LLM products is a 24/7 thinking loop where the claude-code like thing gets input continuously from your wearable, notifications, and newsfeeds and is constantly preparing things for you.
I've been building this for the last 6 months or so. I've basically got it working. The model is not the issue, the infra is. Keeping everything in context just isn't possible and LLMs, even Fable, don't mode switch well. To get around this I've built a database software that ingests as much digital information as possible, and annotates it, then creates timelines with resolution gradients (longer ago = less resolution) that it feeds to the LLM on every request.
Then you have your cheap little MoE or ternary model just running in a loop, with an escalation pathway before it reaches the big expensive models.
Currently it's doing things like reminding me to take allergy medication when I wake up because it's checked AQI or whatever, reminding me to stop at the market when I'm on my way to pick up the kids to get the cherry tomatoes I forgot, giving me heads up of what folks are expecting from me in certain meetings based on cross correlating email and calendar, etc.
It's honestly the single most productive tool I've found for my ADHD.
I’m curious why you don’t just use them like a Meeseeks box, rather than compressing and context stuffing into one. One only checks and categorizes your emails, another one for each category of email or even subcategory, one that only handles calendar additions, a different one to check it and notify you; you can go infinite with it. Hell, I’ll have one instance find a file and read it into the context of a different one because I don’t want a bunch of grep commands mucking up the context of the analysis. The find/read one exists for a few moments, as does the analysis one, and the ‘perform’ one is entirely different. I can run them all in parallel and use a queue if needed.
I’m sure you have reasons for your setup though, so I’m curious how you landed on it.
Think of an LLM as a thesaurus, but for entire trains of thought rather than words. Your initial query yields something pertinent to the task at hand. But let it endlessly recurse and... you end up with something completely useless.
People would do well to acquire at least a modest familiarity with what an LLM actually is. NLP is fascinating. So is entropy.
This is is already possible with Claude Code. I use a setup where I have one instance monitoring a local queue, I have a web app for receiving webhooks from various sources and pushing them to the queue. Plus email for things that don't have webhooks. That instance then decides what to do with each input, sometimes it can spawn additional agent to investigate/prepare, sometimes it creates a ticket assigned to me and then waits for me input. All of that just uses the monitoring tools built into CC. The dispatcher loop doesn't need to be extremely smart, so I might experiment replacing it with a local model like this.
If you are willing and not too busy, What model do you use and what is your cost? (If using subscription would you be able to check with 'npx ccusage').
I run this on a side of the Claude Pro subscription that I use for other purposes. My main motivation was root cause analysis of production issues. I have a solo project and unfortunately my mental state has been degrading over the last years. I would avoid looking at production issues, because I didn't have the energy to focus on the investigation. So I automated this, setup the loop, setup metrics/logs access for Claude to use and now whenever something goes bad, I have a single report that I can act on easily, and if I don't, it will ping me in a way that's not spammy like automated alerts. But I'm finding more uses for it.
Maybe it's my lack of imagination, but what do you imagine you'd be doing where you'd want to keep a computer busy overnight?
It seems like the purpose of humans isn't to keep machines busy. When our phone or laptop is idle, it's fine if it sleeps. And when we do want something, we'd rather not wait.
(Also, this new model seems to be designed to keep latency down, which is useful for interactive tasks.)
Couldn't this be implemented as a web extension? I imagine modifying singlefile to automatically send html to a local port is much easier than trying to convince a chronically mismanaged organization like mozilla (no offence to mozillians).
It is interesting but it does look like a careful distillation of (Spark and) biggers open-weight models.
The progress compared to Qwen3.6 27B is good, not that impressive, it's a 4 months old model. (kuto to them to compare to 27B dense and not 35B MoE, it's more fair to do so).
It is very probable that Qwen3.8 27B will crush Glimmer-30B on most benchmarks.
I'm so excited about these two new models. Qwen 3.6 27B has been my sweet spot so I cannot wait to try 3.8. Glimmer looks really strong, I'm encouraged that Meta compared it to 3.6 in the model card! Exciting times!
Tried it (the full version, using 120 GB RAM), wasn't impressed, gave it some defective code, and asked it to fix all errors. It kept looping around and around and digging itself deeper and deeper into a rabbit hole; eventually, it got into a "reasoning" discussion about whether a custom compiler was used that supported the wrong syntax...
What I think would be perfect is a model that could run on a single DGX spark and be competitive with DSV4 Flash 731. Flash is already a game changer. Hopefully meta plans on this, like the old 70b. V4 flash is smart enough for any use but slightly too big. 27b-30b isn’t intelligent enough.
For the same price as a DGX Spark here (A$8499) I can buy roughly 544GB of DDR5-5200MHz from retail; which on a quad channel platform would deliver ~160gb/s real world; and ~320gb/s with octa channels (Xeon, Threadripper Pro).
If you can afford it or somehow find a used unit, you can go Epyc for 12 channels.
8/12 channel DDR5 will beat DGX Spark in inference/decode even without a GPU of any kind, as it’s memory bandwidth bound, and the Spark tops out at ~240gb/s real world.
With some optimisation and maths, it’s entirely plausible to ach
You are paying an extraordinary amount of money for the convenience of a super small unit, with still mediocre software support, but at least a community. Expect to be crawling through forum posts regularly, as SM121/Spark has many quirks and ecosystem issues still.
Please don’t pay another 70-80% gross margins on top of already inflated DRAM prices unless you need. The Spark IS really nice if you want to test out ConnectX or if you really need something small and compact and quiet.
Also consider: used Adas or even Ampere NVIDIA workstation GPUs can come with a lot of VRAM and be “reasonable”, with CUDA.
Been investigating these multichannel AMD based platforms last year and seem like none of them can in real scenarios utilize anywhere close to their theoretical bandwidth.
This model I think will be too slow for that on Spark, even at 4 bit quant.
It's a dense model, not MoE like e.g. Qwen 35b or Gemma 4 26B A4B. On a Spark it will be memory bandwidth limited
I haven't tried yet (working on it) but back of the napkin estimate puts it at around 15tok/s even after converting to NVFP4. Prefill would be much higher though. That 15tok/sec is pretty typical for dense models of this size:
NVFP4 Q/K/V/O and MLP projections: ~13 GB/token
BF16 attention gates: ~3 GB/token
BF16 LM head: ~2.5 GB/token
Total: ~18.9 GB/token
At 273 GB/s, that gives a bandwidth-only ceiling of about 14.5 tok/s; actual performance would be lower.
You're right. I'm getting ~33tok/sec w/ dflash on it, even bursts up to 60tok/sec, using my personal home-built-for-Spark inference engine (not vLLM or llama.cpp based)
That's pretty respectable.
Still working on optimizing and cleaning up before I push it.
Still needs 32-64GB memory to run it locally. 64GB Macbook pro with an M5 chip costs more than 4k Euros in Germany. A more practical model would be a language specific (e.g Python or JVM language) and excellent at tool calling and reasoning. Maybe that way they can shrink it even more.
I think if there's going to be advantages to making smaller, more targeted models, those advantages will probably come from targeting specific domains, not from targeting specific languages.
I think that if an LLM can't abstract over the differences between Python and C++, it probably will have an even harder time abstracting over the differences between writing code that manages a webserver, and writing code that does aerodynamic simulations.
Coming from the PC games industry in the 90s and early 2000s, it was a struggle to run some of the games on release. 90%* of people wouldn't be able to play the AAA games on release (think Crysis, etc). This period of local LLMs reminds me of that time, whereby the hardware just isn't there yet. Give it time, and the prices will drop.
Assuming we can even get the hardware in the first place, it might not even be possible for consumers to buy it at any price if it sells out through "agreements" made years in advance https://news.ycombinator.com/item?id=47045459
I'm sooo happy I pulled the trigger on upgrading and getting a new laptop (with 64 GB RAM) last summer. Feels like it was just in time before the exponential price jumps.
I was about a week away from buying a very tricked out MacBook Pro with 128 GB RAM, but was on vacation and worried about it arriving while I was away, and then the price hikes went into effect. Grumble. Oh, well. Serves me right.
I’m waiting for the bubble to pop. I suspect we’re 12-18 months away. We’re at the point where manufacturers are going out of business because the tech market is contracting so much. That’s not sustainable.
That commenter you're replying to knows that. The original commenter before them wrote "pulled the plug" which is different and doesn't quite apply here (actually implies the opposite of what they meant to say).
Ah, the edit was fast enough not to leave a mark, leaving a conversation fragment that made me think OP was being weird about a gun related phrase or something
I would love to see any good research projects about it but i have the feeling that Frontier with MoE is making too fast of a progress so that a customized model would always be worse and that the MoE part is actually going somehow in this direction.
On the other hand, at the GTC was a talk about coding in different lanugage (like spanish) and explaining that the quality between spanish and english is relevant different.
But i have not found a good article about the impact of learning data with practical experiments or even if the order of the learning data matters.
At least I think i remember that Meta mentioned having better and less data can be better than more data with lower quality.
As long as these models can explain to you facts about any other topics, its still overfitted for the task though.
Capability in LLM's is distributed throughout the manifold in subspaces. Even worse, the subspaces exist in superposition.
That is to say, there is no single 'python' part of the model. The python bit is spread throughout the entire model and overlaps with other pieces that have similar, but unrelated, capabilities. For example the python subpspace might be partially in superposition with cupcake recipes, Esperanto, and calculus. We need calculus in a coding agent but not the other two. However, separating them cleanly is almost impossible, and even identifying them is tough.
Internally the manifolds are highly inefficient and nothing like you would imagine something humans built would be designed. It's more like something that evolved in nature.
With MOE you train a router designed to select which parts to activate. The router itself is a trained neural network and the 'experts' are usually not really things like 'python'. They're just the functional subspaces I described above.
Again, those subspaces are all somehow inextricably correlated and live in complex superposition spread throughout the manifold. The router doesn't know (or care) WHY those sections get lit up it just learns which ones to activate to optimize it's own reward function. So maybe it learns to activate "logic", "python" and "cupcake recipes in esperanto" whenever it see's something that kind of looks like python. It's not the best answer, it's just the best answer the tiny router could figure out.
It's all wildly complicated and inefficient, and works nothing like any reasonable human would imagine that it SHOULD operate.
I mean the only way to separate them would be to separate them at training would it not? If there is zero python in the training set, then you have a smaller training set, should be able to have fewer weights in the model
There was some paper about routing at training bio-knowledge into a particular region of the model, which you then can cutoff when serving. But you probably lose some efficiency since maybe you sized that region too small/too big.
If you can’t do it cheaper on your own hardware it does make you wonder how much of the cost of inference those large LLM providers are eating? Datacenter hardware isn’t magic.
Your personal hardware probably isn't running useful tasks 24/7.
If you spend 60% of your 8h work day on full on agentic work, then your hardware is paying off for itself only 20% of available time.
I feel like we’ve had this discussion before. From what I remember, specialized models rarely do that much better than general ones, hence no mode Codex models.
There is no good reason to believe language-specific models are going to be any meaningfully smaller, just worse. Same as English-only models vs those trained on a multilingual corpus.
If that happens you can still buy hardware later with almost certainly more (tok/s)/$ and better capabilities to run newer models more efficiently (remember native MXFP4?). Right now basically every generation of accelerator is adding new capabilities. These aren't yearly DirectX 9.0c-compatible GPU performance bumps.
As an individual, for average privacy needs (e.g. open source or at-home coding and automation), it's pretty much complete nonsense financially to self-host LLMs currently or select hardware now based on the capability to do so, and pay thousands of bucks extra.
Instead of saying "I have a MBP with 64gb of RAM" you'll hear people say: "I'm subscribed to Model 9.x11B" and others will comment: "Oh dang, that's a nice model!"
Even if you had a 64GB machine: Are you willing to reserve 90% of your memory to run a LLM? With dirt cheap models like deepseek-v4-flash that will run "forever" on $10, the answer for me is clearly: no.
I'm waiting for the speed/quality per dollar metric to go down a little bit further and then I will def run it at home.
Its not just that you send a sentence to an API endpoint, you always send EVERYTHING to that agent as a context.
You want to analyse your spending history? You now send everything to someone.
Either no one cares but understands this implication on how easy it is to really capture you or no one really things about it.
But i'm a lot more diligent on what I send. I disabled the gemini activity feature for example because google started telling me that my stuff could be reviwed by humans.
Yeah, it does feel a bit silly with my encrypted disks, encrypted backups, unique passwords, advanced router, etc, while I send everything I do in plain text to anthropic.
I did, it started to become too much work to run it well due to all the spam :| (even with the right signatures and configs, until you learn what a blacklisted ip is and that ips need some time of 'positive history' and what not.....)
But at least with your email, you had to trust only one company, as shitty as it is.
Separation of concerns was also easy.
Now with OpenRouter, you just might by accident, send your whole context to just everyone because OpenRouter just routes to different models and you might just switch around between some free model, the good one etc. And it is always the whole context.
Deepseek flash is open weight, this means we can download and run that model without any connection to deepseek, no data/tokens/usage data ever reaches them. They cannot make us their product.
I see many people saying deepseek and other chinese providers have always been profitable. Also they show their training costs publicly. Can't say for sure since I have not used it personally, but I think they'll for sure outlive the western SOTAs.
OpenAI apparently runs a profitable inference business with 40% gross margin, but their advertising budget is nutso and their real costs are pretraining and research. I suspect Deepseek's comp is not predicated on capturing the lightcone of all future value, some googling insinuates their top pay is $212K US which would support that suspicion. Compare and contrast with the $1.35M and up at OpenAI.
Ah yes, I'm sure Trovalds and Stallman are harvesting my data through free software, aren't they? This argument is used by boomers who were fed cold war era propoganda that surely everybody is selfish, and you're always at fault.
I don't understand the desire to run own AI models for programming locally. No laptop is ever going to be as powerful and energy efficient to run anything close to OpenAI, Anthropic or Google models. A model you can run on a loptop is simply not going to work as well as it's needed for programming. Small models for linguistic work fine, but anything more sophisticated simply won't provide enough resources or power. Or models would need to be significantly dumbed down - then why use them at all? So far the idea of carrying a "thin" or "thin"-like device looks more reasonable to me, while running AI on your own server.
> A model you can run on a loptop is simply not going to work as well as it's needed for programming
The models you can run on a high-spec laptop today are approximately where frontier models were 12-18mo ago (albeit at a lower tok/s rate). If you scan back through hn comments from that era, you’ll find plenty of people saying “this is powerful enough to massively increase my productivity”.
Not always! I get 80-100 tok/s from Qwen 3.6 35B-A3B on a MacBook Pro thanks to MTP. With long contexts that dips to around 50-60. However, prefill is much slower than API models. So it becomes really, really, really critical to not have cache misses.
I’m quite optimistic about the long-term future of local LLMs for privacy and cost control reasons. An LLM running on my own hardware, even if it’s not a laptop but a home server, is one where I don’t need to worry about token limits, token fees, privacy, and “rug-pulling” from the vendor.
In the short term, the big challenge is being able to afford hardware that can run a ~30B model. Last month I got to experiment with LLMs on a NVIDIA RTX 6000 Ada Generation as a visiting researcher during my summer break. I see the power of local LLMs for agentic coding; they’re no Claude, but they are quite useful. I wish I had gotten into local LLMs before hardware has gotten prohibitively expensive and in some cases unavailable; Apple discontinued certain Mac Minis and Mac Studios with high amounts of RAM due to the RAM shortage.
Hopefully high RAM prices don’t become a new normal, though the next year or two doesn’t look good.
I've never done it but would be interested because it cuts out the burden of worrying about costs. Maybe I'm mistaken on energy cost here. There's a constant raincloud that follows me around regarding limits, and it would be nice to shake that.
I've been able to accomplish incredible feats (for myself) since GPT-4, so model intelligence is secondary.
For some companies there might be a need to run them locally. For instance, Apple decided to run LLMs on the phone locally. I guess it depends on how important latency and privacy are. Perhaps Meta is looking at how much interest for those local models is there.
I lament the comments saying this in any way redeems Meta (the company).
The researchers releasing this stuff have almost nothing to do with Meta other than being bankrolled by the slaughterhouse.
You aren't the customer, you are the pawn in big tech's game of thrones. Your good will is a commodity to be traded, almost literally. It will be used against you the moment it's convenient. This is open weights because Meta couldn't monetize it in any other way than to cloud developer's judgement of their reputation.
But I guess most people just don't care.
I'm glad it's open. It does not make me think any better of Meta.
It’s rather amusing to me to read comments like this, and then simultaneously whenever a Chinese company or team releases open-weight models or whatever there is a giant round of applause, America is so behind, and there’s nothing but positive things to say about the intelligent, creative, and well-intentioned Chinese engineers (which is true, America certainly doesn’t have a monopoly on great people). Don’t you know? Only China can release good, open weight models and American companies can’t compete. Oh by the way all the spend is for nothing because China alone can release open-weight models thus destroying American AI.
When an American company does anything? Doom. And. Gloom. The engineers? Taken to the slaughterhouse! America? Behind! The public? Bamboozeled!
> This is open weights because Meta couldn't monetize it in any other way than to cloud developer's judgement of their reputation.
I’ve been told over and over this doesn’t matter. Just needs to be cheap and open. Or maybe that’s only when Chyna is involved?
Sorry this post is a bit snarky but it really is something to behold. And certainly I don’t know the OP’s opinions on Chinese open weight models. Perhaps they agree with me.
While composing a reply to a comment throwing tons of shade on American AI, I took some time to check out the commenter’s HN profile. Their comment history was about 50% such comments. Their submission history started with an article about how Russia was unfairly blamed for some hacking campaign.
It’s entirely possible that this is not a foreign influence campaign. Perhaps there’s a group here that is simply anti-American as its primary interest, and passionately so to upvote each other.
On the other hand, one should not discount the value of HN as tastemaker and trendsetter. Also, it would be fairly easy to run bots here. I wouldn’t be surprised if HN were a field of combat for many parallel influence campaigns, foreign and domestic.
> On the other hand, one should not discount the value of HN as tastemaker and trendsetter.
I would encourage dedicated readers here to aggressively and persistently discount the value of HN as a tastemaker and trendsetter.
HN is actually a trailing indicator on tastes and trends, essentially by design. Things only make it to the front page if they get submitted and voted upward by a large number of people. That means it’s all stuff that is public and seems cool to a bunch of random people browsing a website. Not exactly cutting edge.
Basically every community, online or not, over-indexes on its own importance. Like how random small towns can get so worried about international terrorists targeting them. To folks who love HN, of course the great forces of the world would be attracted to compete here. Look how obviously awesome it is, right?
Paul Graham said HN was set up on the theory that it would efficiently surface great ideas and entrepreneurs for Ycombinator. Needless to say, that did not pan out. (How often do you see PG or any YC leadership here anymore?) Honestly I am mystified as to why YC continues to operate it at all.
I think you're wrong. To gut-check, I asked a panel of AI models "What platfroms should be considered tastemaker or trendsetter for adopting new AI models". Here's the part of the answer relevant to this discussion:
> AI X/Twitter — a few hundred accounts effectively set the narrative in the first 24 hours; vibe checks here outrun benchmarks.
> r/LocalLLaMA — the open-weights kingmaker; a model that fails here doesn't get quantized, and unquantized means unadopted.
> Hacker News, and increasingly YouTube/Discord for the practitioner layer.
The internet will always be a trailing source of these things. If you have to ask the internet, you're behind. The internet is a form of information exchange; information about the thing needs to exist before it is exchanged.
For a brief moment the readership of HN heavily overlapped with founders in Silicon Valley, and for those years HN was indeed a platform that acted as a trendsetter or tastemaker. That time has long passed. Twitter is the closest to this today, which is on that that list, and makes it a decent place to listen. r/LocaLLaMa is not bad as a support forum for GGUF and quantizer projects but it's at the level of "Windows tutorial" of software from the '90s.
If anyone is interested in being at the "edge" of this, I suggest simply going to meetups in tech hubs where people are working with AI and models. My guess is any city with a major tech presence will have more "edge" than HN. I'm fortunate enough to be in Silicon Valley right now and have friends who work at foundational lab companies so it's not hard to stay on top of what's happening. The "internet" of 2026 is just much, much bigger than the internet of 2007 when HN was founded, and so it's just a lot harder to find the information that you need.
Would be nice if there were a hn feature, userscript, or plugin to just filter comments from new accounts. Bonus if there was some sentiment analysis or llm-based analysis to filter out unsubstantiated inflammatory comments too.
lol you believe this bullshit? "likely PRC-origin cluster" what an awesome amount of proof corporations need to convince the gullible.
Why not "OpenAI used simplified chinese to create a fake prc-origin campaign and media buzz to convince the public they actually love data-centers and anti-datacenter sentiment is a psyop" there's an equal amount of proof provided for either scenario.
... and I think it's a big mistake to say, as some people are, that people are opposed to data centers in their backyard because of a Chinese influence campaign. There is also reason to think that some "foreign agents" are doing it for the money
When you say you are anti-American, what do you mean by that? Are you, for example, wishing for the demise of the United States? Do you want to tear down the 1st Amendment and the Statue of Liberty? Are you against democracy? Do you want our businesses and factories to shut down and go out of business? Are you willing or would you support foreign countries attacking our military at home and abroad? Are you cheering against our athletes?
Could you expand on what you mean by being "currently anti-American"?
I can't speak to the parent commenter's precise meaning, but I interpreted them to mean they oppose the current domestic and foreign policy of US political and business leadership (a distinction that blurs to the point of indistinguishability of late, courtesy of said domestic policy).
I believe that many Americans that were previously dismissive or ambivalent regarding critiques of US activity at home and abroad (either due to patriotism, realpolitik apologia, or general naïveté) are now re-evaluating some of the beliefs they hold about their country in light of the chronic political dysfunction and an absolutely breathtaking extent of corruption being perpetrated in broad daylight today (as well as indications that extensive corruption has long festered among our elite class, surfaced via the Epstein revelations).
Yea but if they just oppose the current administration isn't that just being anti-Trump instead of anti-American? His administration is just some other administration that'll come and go like foul sewage gas. We've had 47 of them. Does the OP become pro-America when their party is in charge or when certain conditions are met?
I don't know, just asking.
But I know that in my own upbringing I've always viewed America as a place where, because it's a democracy, we are never going to all be aligned or on the same page about policy direction. Even when I disagree with the Biden Administration or the Trump Administration on some number of issues, and believe my I've got a lot of those issues with the current one, I wouldn't think of myself as anti-American or wishing harm to the country. But that's just my own experience.
Well I certainly wouldn't, but it's a cultural symbol of America and so when you think anti-American I'd think things like tearing down the symbols that represent our nation would probably be in play. Maybe not, idk. That's why I mentioned it and asked.
> Are you, for example, wishing for the demise of the United States? Do you want to tear down the 1st Amendment and the Statue of Liberty? Are you against democracy? Do you want our businesses and factories to shut down and go out of business? Are you willing or would you support foreign countries attacking our military at home and abroad? Are you cheering against our athletes?
I am very pro the "dream" of America, in terms of liberty, democracy, etc. According to every "democracy index" I'm aware of we're not doing so hot in that regard, generally rating as a flawed/deficient democracy and the trends are going in the wrong direction, fast.
> Do you want our businesses and factories to shut down and go out of business?
Generally, no, but this is way too open-ended of a question. I want good economic opportunity for everyone, including every US citizen. But relevant to the OP if Meta got snapped out of existence I think it would be a net positive for the world.
> Are you willing or would you support foreign countries attacking our military at home and abroad?
Nope. But across our entire history ask yourself how many foreign countries have attacked the US? Now ask yourself how many the US has attacked. With those numbers in mind, does the US seem like "the good guys"? really? ...really?
> Are you cheering against our athletes?
Nope, but I'm not cheering for them either just because they are American, I'm not a tribalist.
> I am very pro the "dream" of America, in terms of liberty, democracy, etc. According to every "democracy index" I'm aware of we're not doing so hot in that regard, generally rating as a flawed/deficient democracy and the trends are going in the wrong direction, fast.
Well, to be fair people do have different dreams. I'm not sure those indices count for a whole lot. As an example, folks who argue in favor of returning more power to the individual states are, certainly, acting in accordance with stronger democratic principles. I'd argue the EU is actually a bit anti-democratic as it removes more power from local populations and individual states/countries/entities. Yet how would individual democracy indices rate these two?
And if you disagree with my perspective on what is more democratic, well, who is right and who is wrong?
> Generally, no, but this is way too open-ended of a question. I want good economic opportunity for everyone, including every US citizen. But relevant to the OP if Meta got snapped out of existence I think it would be a net positive for the world.
I'm no fan of Meta. But they aren't the only entity where if they snapped out of existence it would be a net positive for the world. I can think of a few non-corporate entities at least.
But fair enough it's a bit open-ended. I guess if push comes to shove when you say you are anti-American do you want to see, for example, economic opportunity increase in other countries at the expense of Americans? Not all relations have to be such give-and-take, but certainly many economic opportunities are. I'm just trying to learn more about what being an anti-american means to you.
> Nope. But across our entire history ask yourself how many foreign countries have attacked the US? Now ask yourself how many the US has attacked. With those numbers in mind, does the US seem like "the good guys"? really? ...really?
I'm fine doing things like removing all of our overseas bases, leaving alliances including NATO, never again being involved in the affairs of other countries, whether that's Korea (Korean War), Kuwait, Bosnia, or Iran, or Taiwan and China, and cutting our military spending quite a bit to reflect your desire to not be the bad guys anymore. I say let's just sit back and let the rest of the world figure out their problems - why do we need to be involved?
Is that something you agree with as well? Would that be anti-American in alignment with your views or is there other nuance here I'm not properly capturing?
> Nope, but I'm not cheering for them either just because they are American, I'm not a tribalist.
And to be clear even if America was even, say, your ideal dream since you're not a tribalist you wouldn't cheer on American athletes and would also label (not as a matter of insult but as fact) other people who cheer on their compatriots as tribalist?
To answer most of the questions above simultaneously (to some degree):
I only care about the content of someone's character and how their actions impact the rest of the world, not the country they happen to be a citizen of, a fact which for non-immigrants (the vast majority of people) is completely random happenstance.
If already wealthy foreign agents become wealthier at the expense of impoverished Americans, I find that to be unfortunate. If already wealthy Americans become wealthier at the expense of impoverished non-Americans, I find that to be just as unfortunate.
I'm not going to cheer someone (athlete, business leader, or otherwise) on just because they are American. The fact that they are American is as irrelevant to me as the color of their eyes.
Do they seem to be a good person who treats others well? If yes, I will cheer for that person, whether they are American or not. Do they seem like an entitled asshole that treats others poorly? If yes, I will cheer against that person, whether they are American or not.
And the reason I currently label myself anti-America is that I believe that collectively we are the entitled asshole that treats others poorly. And we can't just pawn that off on Trump to be the scapegoat. He didn't materialize out of nowhere. We elected him. Twice. The second time after a failed insurrection. We have collective culpability.
it is strange that when people are pro-american, or pro-any-ingroup, nobody asks for justification; OTOH, make a mistake of speaking from rational viewpoint based on historical events, expect a lot of hate from reddit/HN/any-ingroup-forum.
I applaud your openness to speak. I've traveled/interacted with many nationalities, and very rarely I come across someone who is open/rational enough to openly state their dislike of their country and denouncing their history.
I wish you the best in life, smart internet stranger!
> is strange that when people are pro-american, or pro-any-ingroup, nobody asks for justification
Being pro a group can be strictly positive sum - wanting to lift that group up, likely because you consider yourself part of it or on the same team. It's possible your intentions are bad, but they certainly don't have to be.
Being anti a specific group is inherently negative. Perhaps they deserve it, but that requires justification in a way simply being positive does not.
Good points, I personally believe that if/when China takes the lead, they will immediately stop releasing model weights. It only makes sense as a strategy to counterbalance (current) American labs' monopoly on frontier models.
Holding both those positions would be hypocritical all right, but are you sure it's the same people commenting/voting in both cases? I don't think there's a strong consensus on Hacker News. Even something like the time of day an article is posted might get different engagement depending on who is active in which time zones.
> I don't think there's a strong consensus on Hacker News. Even something like the time of day an article is posted might get different engagement depending on who is active in which time zones.
Based on my own experience and reading, I do think there's a general consensus on this site but I could certainly be wrong about that. I'm less concerned about hypocrisy per se, it's more that the arguments that are used, even if by a minority, seem to apply in only circumstances in which China releases open-weight models.
I am aligned with your viewpoint as well. And I've repeatedly argued it. If China were to take the lead the US can then just release open-weight models. Folks say having the lead doesn't matter because China releases cheaper open-weight models. We can just let them take the lead and then do it back to them.
> It's common, if not inevitable, for people who feel strongly about $topic to conclude that the system (or the community, or the mods, etc.) are biased against their side. One is far more likely to notice whatever data points that one dislikes because they go against one's view and overweight those relative to others. This is probably the single most reliable phenomenon on this site. Keep in mind that the people with the opposite view to yours are just as convinced that there's bias, but they're sure that it's against their side and in favor of yours. -dang [1]
The problem isn’t that people on HN have a bias, I feel it’s pretty balanced. The problem is that when there are any sides, they spend the top 100 comments rehashing the same arguments, often over a political bugbear or web design faux pas.
That pattern became a lot more obvious when there are five new front page AI posts a day.
> The problem is that when there are any sides, they spend the top 100 comments rehashing the same arguments, often over a political bugbear or web design faux pas.
It's not even about sides, if for the last few hundred days you read a few AI related threads a day, then you notice that almost all arguments are rehashed, literally it's the same thing repeated using different words for 80%+ of comments on almost every AI thread. I started skipping most of it because there is genuinely nothing new or interesting added to these discussions.
> The problem is that when there are any sides, they spend the top 100 comments rehashing the same arguments, often over a political bugbear or web design faux pas.
This is a problem with any upvote/downvote based site, in my experience. It only takes a couple people who are highly engaged and who have a lot of free time to refresh the comment section and downvote everyone who disagrees with them.
Some times I’ll write a polite and well-sourced comment correcting some misinformation here and the comment will go to -2 or -3 when I check back in 10 minutes. Information that goes against the angry narrative du jour is often not welcome. Later, as calmer heads read the article and peruse the comments the downvotes start to get balanced out and the comment might rise, but some times the first wave downvoters are aggressive enough to get the comment downvoted into gray text before it has a chance to be seen.
I don't think they're bots and I don't even think they're organized.
In my experience with other communities, it only takes 3-5 people with a lot of free time to crush conversations they don't like. On a fast moving comment section a wrongthink comment can be buried into the bottom half of the comments with only a few early downvotes.
When I used to run a spam factory my impression was that success on Reddit was all about avoiding the downvote brigades that would nuke you before you got your first 5 upvotes. Write a post on proggit where you confess that you are partially in it for the money or have any interest in the business of software and hoo boy, that's why I migrated to HN.
I'm certainly pro-America and anti-communist/fascist as a bias, but I don't really care about whether AI tools are open-weight or not. I just use the products that best fit my needs. I just think the arguments put forth regarding China and open-weight models and strategy are not very good. It just so happens that China is the only real competitor in the AI space and so they are who get talked about the most in comparison to the United States.
This comment isn't applicable to me, and if you believed that it applied, you'd have to add it to the OP as well since they feel strongly about Meta[1], they notice data points about Meta's behavior, and they overweight their bias against Meta[1] relative to others. Same with China "leading" and open-source/open-weight models and any time someone says China's strategy is better.
You can repeat this for any online argument or any topic.
It's not that Dang is wrong, however. It's that posting it in response to my comment(s) alone is hypocritical and pointless. Whereas Dang who is more responsible for the entire community is right to speak about it more generally. The message matters but so does the messenger, in this case.
[1] I don't use any Meta products (I don't even click on links), think social media should probably be outright banned, and Meta very likely should have been sued into the ground for the effects that their platform seems to have not just on children and young adults but also on our political system.
I see it as a strategy to increase capital costs for American frontier labs. By eroding the expected ROI of frontier labs, you deter private investment into them, which slows OpenAI and Anthropic in particular, giving space for the laggards to catch up.
> I don't think there's a strong consensus on Hacker News.
There are diverse viewpoints. However there are some topics and threads where it becomes obvious that the comments are going to tilt toward one viewpoint. Participating in those threads with a different opinion will get your comments downvoted to -2 within minutes even if it’s well-written and factually sound.
After this happens a couple times you learn not to engage with those threads because it only takes a few zealous downvoters to bury anything you write. So the illusion of consensus persists.
Concrete example: There was that fake (AI hallucinated) report that Meta spent $2B lobbying on something that was popular here months ago. I actually read the repo and report and noticed the AI hallucination, as well as pointed out that $2B in lobbying spend by a single company was not plausible or supported by any evidence. It didn’t matter how I wrote it, it would risk getting downvotes and angry replies about “How dare you defend Meta!” Some people are here for the anger and to feel revenge against the enemies they think they know (like the US) and will cheer on anything that goes against those enemies, regardless of the other facts surrounding it. Factually accuracy often takes a back seat to pushing agendas.
Maybe you're right for smaller models, but for frontier models this is the tail wagging the dog. If China has exclusive frontier model capability in the future, that's an enormous commercial and geopolitical lever. They won't throw that away to sell a few more robots. Anything beyond their competitors' capabilities will remain closed.
You imagine robots are going to play second fiddle to models, but they could become integral parts of model training with all the data they collect. Training could even become distributed. This will enable them to be responsive to local needs.
Likewise the Inkling open weights announcement, Thinking Machines model, was also not criticised.
The comment about Meta is because of particular dislike of Meta, because of their business model, and how harmful they've ultimately turned out to be for the world - disproportionately so relative to their benefits to the world, compared to other big tech companies.
> disproportionately so relative to their benefits to the world, compared to other big tech companies.
This is certainly what many people around here appear to believe, but there are lots and lots of people who get much more value out of Meta's products than those of any other tech company. Whatsapp alone is probably the most useful tech product for many, many people.
That being said, FB/Meta have done a bunch of awful stuff, but to say that they're worse than Google/Amazon/Microsoft is not necessarily obvious.
The GP is claiming Meta is an awful company for what they have done and how they continue to treat their employees. That’s a perfectly ok opinion to hold, and many seem to agree.
Have DeepSeek, Moonshot, or the other Chinese AI companies done such things that attract moral outrage?
If they have, would you ever know about it? CCP controls the media, the companies, and the people. What's your evidence that Chinese employers are treating their employees better other than there's silence?
Nobody wants to live in a world where one party dominates due to access to superior AI and the others have to fear it (well, except for a few psycopaths who would gamble on being in control of that party). So the underdog will always be the good guy in this race. It has been framed as a race between countries, so Meta fails to be the underdog because they're in the wrong country. That's all.
I don’t think Meta is bad for releasing open models, but are you really going to ignore all the terrible things they’ve done over the years just because of that?
As for DeepSeek or any other Chinese lab, I’m not aware of any practices that would make me consider them a bad actor. Can you say the same about OpenAI, Meta or Anthropic?
It makes me wonder why this dynamic exists here, and I do wonder at times how much our conversations here are influenced by China in a top-down fashion. I'd prefer to think that HN is pretty organic, but that is probably a naive thought.
> When an American company does anything? Doom. And. Gloom. The engineers? Taken to the slaughterhouse! America? Behind! The public? Bamboozeled!
I think I've always had a pretty healthy amount of cynicism towards China. In recent years my cynicism towards the US has increased significantly. I don't see all of my US peers with cynicism, but I think you're living in an age of grift, corporate capture, and unheard of corruption. I also think there's nuance to both. There are some US and Chinese companies and people that I do respect regardless of what's going on politically. (Meta / Zuck isn't one of them though...)
I live in the 51st state though, so maybe I'm just overreacting...
I don't think that's entirely true. As an example you can look at how the European Union is increasingly putting up trade barriers with China and identifying opportunities for new rules and regulations to prevent Chinese dumping as America is also trying to do.
Japan, South Korea, the Philippines, and other countries participate in freedom of navigation and combined arms exercises because they perceive China to be a threat to their countries. [2]
I think it's more of a mixed bag. You see a lot of public talking points, and in Europe specifically a lot more healthy discussion about not being militarily as dependent on the United States as it has in the past and looking itself to follow Trump's lead to onshore capabilities (cloud for example) but I wouldn't read such moves as moving closer to China so much as they are hedging their bets a bit more.
Here's an article [1] that was reporting on this topic that I found interesting along with a select quote from the article:
> Luxembourg’s Prime Minister Luc Frieden said Thursday at an EU summit that China is “an existential threat for our industries.”
> Even Germany, whose economy has long relied on exports to China, is alarmed. Chancellor Friedrich Merz said this week that Beijing keeps its currency up to 30% undervalued, calling this “a massive competitive disadvantage.”
Can you point to a lot of posts lauding the Chinese government based on the release of Chinese open models? Because that would be the equivalent to contrast with the OP.
I suspect it's just the generalized anti-West/anti-American sentiments extended into anything and everything. Anything that makes the US look anything close to good goes against their cause and therefore must be countered and talked down.
But you're not wrong about the bias here. You just don't see many comments talking about it because they get mass flagged/downvoted for obvious reasons.
These sentiments aren’t formed in a vacuum. America is becoming increasingly oligarchic and corrupt with decades of experience of companies profiting off harming people and lying through their teeth and Meta is like one of the worst offenders. They are pissing on every ally they have and once again started a war and both have material impacts on other countries.
Americans seem to take the US’ geeat reputation for granted and don’t realize how it has slipped and what that means. They also take for granted that China BAD is truth when this sentiment basically just sprang out of nowhere when the west realized it was their geopolitical rival. But to the rest of the worlds citizens, China is not starting any wars and is the source of cheap goods and innovation to other countries. EV batteries recently. The US is now directly causing high oil prices with their war and exports their rapacious companies like “prediction markets” which are 90% sports gambling now to the rest of the world. Meanwhile the classic American move to these kinds of comments is to claim that negative sentiment MUST be part of some bot campaign because surely no one could actually dislike the great America??
Those factors are all rightly part of the sentiment.
> America is becoming increasingly oligarchic and corrupt with decades of experience of companies profiting off harming people and lying through their teeth and Meta is like one of the worst offenders.
Isn't China a single-party state that is run by a corrupt autocrat, has literally enslaved people to build products, and disappears people for saying the wrong things? If America is becoming more like China, shouldn't you dislike China more than America?
> when this sentiment basically just sprang out of nowhere when the west realized it was their geopolitical rival
Well this isn't a sentiment that sprang out of nowhere. It is a sentiment that emerged based on real or perceived behavior.
> China is not starting any wars
Maybe not yet, but they certainly aren't behaving nicely toward Taiwan or other countries in and around the South China Sea, are they? The Philippines and Vietnam come to mind immediately, of course there are others.
> the source of cheap goods and innovation to other countries
But if those cheap goods run your country's businesses out of business and you have lots of people without work, is that a good thing?
> The US is now directly causing high oil prices with their war
The US didn't ask Iran to pursue building nuclear weapons, supply Russia for its ongoing war in Ukraine, or provide weapons and funding to groups in Yemen, Iraq, Syria, Lebanon, and Palestine which have led to instability in the region. If oil prices are too high, the rest of the world should have worked to address American concerns about Iran. They failed to act, and so the US eventually just did what it thought was necessary.
> rapacious companies like “prediction markets” which are 90% sports gambling
Before sports gambling was a thing in my state (Ohio) I seem to recall traveling to other countries and they had sports gambling too. Maybe I'm wrong?
> Meanwhile the classic American move to these kinds of comments is to claim that negative sentiment MUST be part of some bot campaign because surely no one could actually dislike the great America??
I can't speak for others but since I started this thread I'll chime in. I specifically disagree with the arguments put forth about AI strategy with respect to the US and China. I have no idea and frankly, aside from erroneously being accused of a bot once [1], have no interest in trying to guess if someone is a bot or not. Sometimes I just assume everyone is, because really what's the difference when it's all just text?
Others may feel differently and believe that there is a brigade of bots. I'd ask them for concrete evidence.
I say China is not starting any wars, you gesture at some vague "not behaving nicely" as if this is equivalent to a war or some damning proof that they will start a war - these are nowhere near the same thing. China has killed maybe 50 people outside it's borders since 2000. The US has killed 100,000-250,000 in that time period. There is no amount of rationalization that can reverse the deaths of those people or the cities turned to rubble.
The attitude towards Iran is also just arrogant. The US believes they have the unilateral authority to dictate who has nuclear weapons and who doesn't, and is morally in the right for bombing their cities and killing their citizens if they don't comply. Sovereignty be damned, we are gods, fuck you. Same thing with literally kidnapping the president of another country. Not to mention, the Iraq war was started on the exact same WMD excuse and it turned out they never had any real evidence of WMDs back then, they just said they did and it was really about oil. And the government now is 10x more corrupt and cruel and xenophobic than it was back then, and back to a huge focus on fossil fuels - don't you have to be a little naive to just trust them at face value now?
I can tell you the prevalence of sports gambling and betting is exploding in Canada now that Kalshi is directly tying up with banks to put gambling inside fucking banking apps. I'm sure it was prevalent in many countries, I'm also sure it will scale up to huge levels and enter new countries with the amount of marketing and money and influence these US companies will bring, who also happen to have DJT's son on their board. Making them markets also frees them to manipulate and trade against their users covertly in ways that cannot be detected and that a traditional sportsbook could not do.
People who actually visit China or talk to Chinese citizens are all quite optimistic about it. The tropes people keep throwing around about autocracy are very inaccurate to anyone who actually reads anything about day to day life there. I can't explain all of it here but you can try reading something from Arnaud Bertrand if you're interested [1]. There’s a comical level of propaganda against China since the west recognized them as rivals that does not match reality at all. Anyways, I always see incredible whataboutism these days when talking about the US' problems now. This is what I think your original post that I replied to does too - the parent was criticizing Meta, and you said but what about China? And the thing is, we're talking about Meta because we live in the West and the horrible shit they do impacts us much more directly than whatever China does! It's not really productive to say China Bad!! in that scenario, it's really just saying "yes we suck, but they suck more!" and that doesn't solve anything.. it channels what should be a focus about huge corruption and threats to democracy in western countries into a pissing match on some other country, and its very sad to see how many people are falling for that.
> Only China can release good, open weight models and American companies can’t compete. Oh by the way all the spend is for nothing because China alone can release open-weight models thus destroying American AI.
Let me be blunt and let me say: you don't understand why we people support Chinese models.
1. Chinese labs started with open weight models, US labs started with dooms day narrative
2. US VC based companies must become greedy to win and return the money, Chinese companies can make 1/10 of that revenue and still be happy
3. Meta in this case, started nicely with Llama, then switched to closed models, kicked out researchers to build data labeler CEO empire inside Meta. Now opening again, what's next? closing again?
Science doesn't work this way, someone could claim only reason World is successful because Chinese invented paper, so please don't go there.
Regarding distillation, you can also say Anthropic and OpenAI stole /distilled books, articles, blogs from everyone who published things without knowing their work might be used against them
> Science doesn't work this way, someone could claim only reason World is successful because Chinese invented paper, so please don't go there.
Perfectly valid claim. It could be debated though, since you are talking about a centuries long diaspora of processes.
I'm talking about an architecture that was invented, published, and pioneered by US companies just a few years ago. Not really the same argument IMO.
> Regarding distillation, you can also say Anthropic and OpenAI stole /distilled books, articles, blogs from everyone who published things without knowing their work might be used against them
Yes. I would argue that anthropic et al are only able to exist because of the massive body of work in the anglosphere, both in books and online. That goes hand-in-hand with my view of the Chinese AI sphere.
US companies started by publishing their research, but then they have stopped abruptly, seduced by the mirage of heaps of money that could be obtained by having a monopoly on AI.
During the last few years, the published research from USA was only a small fraction from that published from China.
> It’s rather amusing to me to read comments like this, and then simultaneously whenever a Chinese company or team releases open-weight models or whatever there is a giant round of applause, America is so behind, and there’s nothing but positive things to say about the intelligent, creative, and well-intentioned Chinese engineers
It is absolutely applied accurately. You're commenting on the alleged hypocrisy of people simultaneously criticising American open-source while praising Chinese open-source, and then attributing your perception of hypocrisy to the website as a whole. The reality is the behaviour you've observed comes from completely different individuals, not some kind of HN hivemind. Your comment is such a typical case that it could go in the wiktionary page as the example excerpt teaching people what the goomba fallacy is.
There is a big astroturfing going on social media platforms by the chinese. Did you notice 'day in a life of unmarried 30 yr old lady in china' videos flooding usa social media.
Regular ppl in the west now hold mildly positive views of the ccp and how 'advanced' china is than usa.
Then there are europeans who now are looking for china to give them the technology handout now that relationship with usa has soured.
The interesting thing is that China began as an agrarian economy and quickly modernized via influence/direction of the CCP.
The United States used to have a dominant middle class that was geographically distributed (cities & rural areas inclusive), but the advent of the tech economy has also been having a similar effect here as it did in China: massive wealth accumulation in Tier 1-3 cities and everyone else being largely left behind.
It'll be interesting to see if there's convergence in the next decade or so, especially with the GOP reducing regulation and increasing the explicit capitalist priorities here. TBH, though, as a generally well-informed political outsider, the feeling I have is that the US government is slow, bulky, inefficient, balkanized and overall poorly run compared to the Chinese government. We'll likely either slowly improve authoritarian efficiency and become more like the CCP ... or we'll pivot left and move more toward the EU model, but we're floundering around right now paying lip service to both.
Propaganda in China is distributed, decentralized—ie it's the responsibility of every minor bureaucrat, clerk, and white-collar worker to produce and engage with propaganda.
Deepseek never fucked us over. zuck has. A decades of harm creation run doesn’t get excused by the US flag. Zuck is not on your team and if you can’t see that by now, oh my.
> When an American company does anything? Doom. And. Gloom
Meta, "an American company". Being the main driver of an ethnic cleansing in Myanmar - and just sticking your head in the sand when told about it - is just another day's affairs at the average American Acme Inc.
These are comments on a release by easily the most societally damaging Western tech company there is. They so far easily beat Flock, Palantir, Anduril and so on, as a result of their incomparable scale. You're just ignoring that and pretending any negative comments are because it's an American company rather than Meta. That's much more FUD than any pro-China comments I've seen on HN.
Get off HN Mark, you have ten million pervert glasses to sell.
Sorry this comment is a bit snarky, but yours is indeed a sight to behold.
I'd also argue this is the case for any company releasing open weights. They're not righteous, they're marketing. That's not necessarily a bad thing! They're releasing some great stuff for free and we benefit from that. Every company doing this has a motivation to not release these for free.
Alibaba, Google, Moonshot, Thinking Machines, etc are not releasing their models for free because they love to. They want to grab market share. I'll take it.
I still will not use a hosted Meta product, but damn this model looks solid.
This model doesn’t look solid at all. It comes months after the Qwen model, and in almost half the benchmarks, it performs worse than that. Plus, the next Qwen 3.8 is going to be announced this week. So, this model is DOA.
I don't really care that much about benchmarks, but having tested it on one of my puzzle prompts I can tell you that it solves it well, writes clearly, isn't noticeably slower than Qwen 3.6 27B and is much more terse in its reasoning (which will help with preserve-reasoning).
It also has a knowledge cutoff inside this year.
The main limitation is the smaller maximum recommended context.
Meta can never be redeemed, but it's still valid to admit that FB at one point had a very badass engineering culture.
They're one of 2 companies I would absolutely never work for (weapons etc aside). FB's recruiters hounded me so often I requested that they blackball me. The day they became Meta, I learned this by checking my email to see that they started trying to reach out again. I once again requested that they blackball me. This by extention taints OAI, the other company I'll never work for.
After a few hours with Glimmer I'm pretty impressed. It's better than the benchmark scores seem to indicate compared to Qwen 3.6 27B. I'm very excited for 3.8
The book title is a reference to The Great Gatsby: "They were careless people, Tom and Daisy- they smashed up things and creatures and then retreated back into their money or their vast carelessness or whatever it was that kept them together, and let other people clean up the mess they had made.”
― F. Scott Fitzgerald
> It's better than the benchmark scores seem to indicate compared to Qwen 3.6 27B. I'm very excited for 3.8
Is it worth considering if it's only marginally better than Qwen 3.6 though? Qwen 3.8 27B is almost there, and will probably be better suited as drop-in replacement for 3.6. Not even considering there's probably going to be a 3.8-35B-A3B too - which will have even better performance.
Qwen3.6 is very token inefficient with it's thinking. Quantized versions often get into loops.
Glimmer is trained with 4 effort levels, not just thinking on/off. Maybe it's more token efficient in general. There's official 4 bit quantizations with reported 1% loss across 15 benchmarks -- so quants probably work good.
IMO that alone is worth trying for, even if they're otherwise equal.
Meta doesn't need to be "redeemed". They have two of the most popular social media apps in the world. And theyll prob survive without ever having you work there
Should have clarified. I meant my comment to be read as [random online person] is not particularly valuable and his comment gasing himself up as a high value meta acquisiton is polluting online discourse.
Tech equivalent of "I wouldn't date Sydney Sweeney, I'm not into blondes". Cool story bro
Mind telling me roughly what you had on your resume that had meta /fb hounding you for a job? ( Of course so I can avoid having this situation happen to me, naturally)
Muse Glimmer doesn't redeem Meta, but it's a contribution to the commons and the Apache 2.0 licensing is an improvement from the restricted licenses attached to Llama. If even Meta can use a permissive license for its model weights, so can any other company.
Thomas Bayes would say that the population of people who hate Facebook is really big and the population of people who are scrupulous about whether or not their comments are specific to the matter at hand is relatively small
Unfortunately there are a few topics that short circuit some terminally only people. One of them being anything related to meta. Few others recently emerging is Flock or Musk. It's really exhausting since you can't have a discussion relating to anything that may be adjacent to said topics. It's like a black hole.
This is par for the course, HN is far worse than reddit on balance, especially involving upvoting/downvoting decorum.
Go vibecode something to auto upvote all downvoted posts, call it "Antiechochamber.HN" or something, and if enough people used it this website might improve a bit.
Your lamentations and opinions are noted. Do you have anything to say about the model? Something useful or substantive? Or is this just a place for you to let us all know what you are thinking these days?
To be honest the main issue with meta has never been around open/closed software. They've also done react, Cassandra and some other bits. But this, like their open weights is like a feather pressing down on the scale compared to things like promoting genocide in Myanmar, enabling Cambridge analytica, creating a huge closed ecosystem which dominate(s/d) local community communication, mandating doxxed communication, trying to replace actual community communication with algorithmic nonsense etc.
More meta derangement syndrome on HN, what a surprise.
We all benefit when companies invest their resources in producing open models. No one thinks this absolves anyone of being terrible elsewhere. But we can still be happy about it.
Any retort to do this like “but why would they just openly release this”[1] pretty much answers itself. Public relations.
If a company can spend money to redeem itself then, well, it can (game theoretically or whatever) do whatever it wants in the future and then spend money to wipe the slate clean.
[1] By which I mean: the very act of being prompted to ask such a question, of planting a seed like hmm, Meta might have some aspects which are good for us. You don’t have to be convinced of it. Just the seed itself can pay for itself.
Meta and its products, as a whole, is a threat to your kids, your mental health, your community's health and the planet as a whole. It is just sad and very repulsive everyone fell so easily addicted to their social drug. Yes - it is a drug, and it is hard to get off from.
Nothing redeems them at this point of time, they are doing exactly ZERO to redeem. Tossing open weight models (not opensource!!) is not a basis for redemption, and does not constitute remorse in any way. Trying to portray it as such is complicity to META's crimes against humanity.
some support already merged, and I verified in a local build that it runs (cannot get MTP params working tho, about ~40 tok/s on my beefy 800GB/s 7900XT w/ 20GB VRAM). https://github.com/ggml-org/llama.cpp/pull/26841
The post suggests that you need an rtx 5090 use it, which is currently selling for around $5,000 USD. I wouldn't exactly call that "my device", since my device costs about 25% of that for the entire computer.
For the same cost, you could run on a frontier model on a pro plan for two years. The economics dont make a lot of sense for this to me, so I would love some input on why people want to do this instead (privacy, for fun, etc).
If you’re doing breakeven math on subscriptions, consider that your own rig can run 24/7 whereas you will get a fraction of that with sub rate limits. Even if you factor in PG&E residential rates, the breakeven is a lot closer to months for overnight long-running agentic coding a couple times a week.
And in terms of interesting use cases: recently pointed an agent at Blender and gave it vision. That setup can essentially iterate on a scene forever.
Is the assumption here that inference costs will stay roughly static, or that frontier models will keep getting more expensive quickly enough to offset efficiency gains?
Because I don’t think “the subsidization train is going to end” necessarily means current pricing becomes impossible.
If capital keeps pouring into frontier AI, companies still have an incentive to subsidize access while competing for users and market share. And if that subsidization starts drying up, there’s even more incentive to bring inference costs down by making smaller and cheaper models catch up to today’s frontier capabilities.
So either way, I’m not sure you can extrapolate from the cost of serving current frontier models to what equivalent capability will cost two years from now.
I think like you mentioned, the practical reasons are disproportionately oriented around either privacy (, a clear constrained workload (need to OCR files/transcribe audio, and there isn't really a clear or meaningful reason to switch out the model to chase new incremental gains), or regulatory compliance (e.g. source code, patient data, can't leave the country).
I fully expect Meta will release other, smaller Muse models in the near future too.
The 5090 is also supposed to be a $2000 GPU, not a $5000 one. The entire market is utterly distorted right now, which will impact cloud inference more and more over time too. They are not immune to the absurdly high RAM prices, so their prices will have to go up over time too until the RAM supply chain goes back to normal.
I'm currently running it on an RTX 3090 (street price ~$1000 USD) with a long context and getting pretty good performance.
Prefill: ~1000 tok/s
Decode: 75-100 tok/s
It'll be far faster on a 5090, but I find the above performance to be acceptable. I've seen some claims that it even works OK on an AMD RX 7900XT (~$500USD)
That is the most recent Qwen and Google models, there is no newer version, yet. Qwen3.8 27B might come in a couple of days tho, if it's launched alongside the large one when the Qwen3.8 countdown reaches zero.
Optimizing speed is really the way to go.
Yet 24GB is not what everyone can afford.
Maybe we could take some of those 56tk/s and transfer into some free RAM space using MoE loading ? I'd be glad with a less than 10GB and more than 6tk/s model.
Unfortunately this is just the entry price for LLMs. With the exception of the Qwen 27B models, I personally haven’t found a ton of use cases for models less than 200B. With the right setup, fine tuning, etc, you can make small models do cool things, but hard to please everyone given the insane hardware costs at the moment and the comparably cheap API costs.
Small models are still great for lots of “simple intelligence” use cases, like annotating or summarising files and media; or even just basic chat when given web search tools.
My local NAS is private and I’m not going to send it off to APIs for captioning or metadata; but even Qwen3VL 8B does an excellent job at this, despite being quite old.
They are also really excellent for fine tuning. Unsloth and Tinker (from Mira’s TML) are great places to start.
If your use case is narrower than “coding agent for everything”, you can probably match frontier performances on that narrow domain with ~30b and exceed it with ~100b+.
Having played around with this model a bit, I am fairly confident that it is not competing in the coding space.
It can do that, but its actual selling point appears to be a different take on guardrails and safety alignment.
Either that or the only new training data left was industrial quantities of dark romance literature and Wattpad.
Clever business move. 131k context is more than enough for that use case, and due to that small K/V footprint, you can probably have a bunch of characters on the same GPU.
Or it's just a happy little accident. We will never know.
___
I was informed that normal people use LLMs for mundane tasks like asking for a pancake recipie.
That it apparently can also do decently.
Unfortunately, it is also very confident, regardless of whether it is actually correct.
So maybe it should actually stay the smut engine and nothing else.
The least they could do, after ruthlessly bombarding my employer's servers with requests, ignoring the robots.txt, scraping everything, and incurring significant Google Maps costs for us in the process.
I tried to run it with lemonade by installing it via hf but did not succeed, it gets some weird 500 errors. I also see that ollama has currently only an mlx version available.
> Some have tried to frame distillation as harmful, but I think it is important to protect the principle that you can learn from anything you can observe.
Let’s give thanks to all those meta engineers who have been ripped for my heir teams (while sitting right by them) working on manually tagging data. I guess the morale dip paid off in some way? I wish you all well and hope you find some happiness … IYKYK
I like this class of model. Multi-token prediction makes it viable to run dense models at not-too-far-off speeds as MoE models with much better intelligence.
The submission’s title (open weights 30B local coding model) is luckily wrong: This is meant to be a general agentic model.
It even comes pre-quantized and with a MTP/drafter model. Looking good!
Let’s hope they aren’t dishonest with the benchmarks this time …
> It even comes pre-quantized and with a MTP/drafter model
Glad to see the extra engineering effort that went into creating this local model and making it run well on a consumer device. I use qwen3.5-coder, and am waiting to kick the tires on this one. I hate to say this, but kudos to Meta ! I hope apple and others follow suit and create similar local models for other use cases like audio, images and video that can run on a laptop.
I'd really like to see a 45B-ish dense model ready for a dual GPU setup. Something with a little more intelligence while still within the range of some higher end local setups.
There is definitely an under-served target memory size of 48GB - almost everything aims for: 12, 16, 24, 32, 64, ...) But most dual-gpu setups, 3090/4090 (and some mac configs afaik) have 48GB, and most 64GB systems would do well with the extra 16gb of overhead saved. 48GB is also moderately common in PC memory configurations since 24gb DIMMs are a thing.
Another candidate for the 7900XT (20GB VRAM) I got sitting around. I pulled latest llama.cpp (targeting vulkan during build) after seeing a muse PR merged a few hours ago, and unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL runs on my 7900XT barely (and with no MTP). Sits at 19GB VRAM w/ 4 parallel 113k context slots, all layers on GPU, and at 700 tok/s prompt, and ~36 tok/s generation.
Waiting on Q3 to download to check speed + do my usual anecdotes. I generate beefy code snippets and poems, and also ingest my HOA declaration and answer nuanced questions.
edit: i should've prefaced this somewhere with: This card ballparks at 800GB/s IO, which I can't seem to find easily on the market anymore. Kinda the ideal card for this model, if I just had a _little_ more VRAM (XTX is 24GB).
edit2: not mtp, this is dflash model (param in child comment). I'm up to ~60 tok/s generation and sitting at 19GB VRAM (i added --no-mmproj (makes it text-only i believe) because I'm used to speculative decoding wanting more VRAM and I'm already close to the limit :sweat_smile:)
Q3 results: unsloth/Muse-Glimmer-30B-GGUF:UD-Q3_K_XL gets down to 15.6GB VRAM and full context (131k) on the 4 parallel slots. Prompt/generation speeds about the same. Overall feeling like a nicer-fitting Qwen 3.6 27B, but want to test out MTP generation speeds once I can.
edit: My favorite bit of reasoning I saw go by in my "generate me a beautiful code snippet" anecdote: 'Could give a snippet of beautiful code: the "hello world" in brainfuck? No.'
edit2: my first dflash speculative model! no mtp. I'm up to ~60 tok/s on empty context with `--spec-type draft-dflash`
Many companies are stressed about token cost, as we are moving to a consumption based charge.
In the meantime - new open source models, such as DeepSeek V4 Flash and GLM5.2 reduced the price to about 13x chepaer. Also OpenAI had reduced its price for considerably.
Now Meta is back in this game. The upcoming months are going to be interesting (GoT)...
Meta did not abandon opensource. I would love to see a smaller distill, or a moe of this size but the benchmarks seems competetive as long as it isnt benchmaxed witch i would not be suprosed if it is.
I don't think outside of the Big 3 (Ant, OAI, GDM), given the strong competition from China, any other Lab has a chance at capturing the coding market if they aren't open weights (save for xAI whose latest Grok looks every bit good & will probably rely on Cursor for distribution instead of going open weights). There's literally no other selling point, as the capabilities have mostly converged by now among the chasing pack.
It’s not completely open source, but they actually release their pretraining and post-training datasets with some redactions for (cough) pirated content.
They also have very good code and playbooks for actually doing a fine-tune, CPT, etc.
Even if you’re not tuning a Nemotron model, its mixes are very excellent for your replay data slice; or general experiments. Way better curation and quality than Dolma, etc; or other large huggingface data mixes I tested.
Yes, I second Nemotron. I'm using Ultra remotely and Super locally, and I find them very useful for RAG-like problems. I wouldn't really use them for coding.
There was a good discussion yesterday on the DeepSeek Flash release thread about this.
There's a large market, very large, who want the best regardless of what it costs. Probably a large enough market to keep that domain of research afloat (as opposed to shifting research manpower to cost cutting).
The reasoning is just that the marginal cost of AI is very secondary to fixed costs of the businesses themselves; it's not an excuse to sacrifice performance.
Some labs go through bad patches, GDM is definitely in one right now and the recent departures are not reassuring, but I think it's too early and dismissive to count them out of the race so far. They just need one good frontier release for everyone to go "GDM is back!"
Claude models weren't really good or noteworthy until the 3 series anyway.
I think the record is quite clear, GDM was never in the race.
All of the Gemini models have been considerably behind the capabilities frontier. The only exception was 3.0 which seemed quite good, but had latent issues and we were all measuring with the incorrect metric, agentic where it's latent issues were very pronounced.
GDM+Google may have created an exceptionally efficient LLM for serving search. This is likely a great accomplishment (or maybe Google is burning money at a rate unheard of before). But Frontier capability: they have never been in the race.
This is sad, since they had everything necessary to be on or beyond the frontier.
Those two offer MoE variants, this doesn't seem to.
Dense model makes it dog slow on anything without HBM. Max 15tok/sec on decode on DDR5 systems like a Spark or a Strix Halo -- and that's at 4 bit quant.
Dense models run at a very usable speed (Qwen 3.6 was running at ~50t/s last I looked) on my dual 7900 XTX desktop. (And before anyone brings it up, I did not buy them for this purpose, so the up-front cost is irrelevant in my case.)
That's a bit amusing - not that I have the hardware to run it, but officially it's not available in Hong Kong. Not that getting it would be much of a problem with a help of a VPN either, but I'll assume mainland China is also restricted. Certainly not a competition for Chinese open weight models... in China.
kind of a nonspecific complaint, but i haven’t yet had much luck with anything under ~120b, feels like models released on that order is coming to a trickle. the last few qwen models didn’t seem to go that high, and i got worse results than qwen3.5-122b
Next step: Burn the weights of these local models into an asic that ships cheap on a laptop (AMD/taalas looking at you), and I will be a happy camper. Make it pluggable so I can select a model I want. I use qwen3.5-coder currently on my laptop, and while it works well enough for me, it is somewhat slow processing tokens.
I would hazard a guess that fast small models with a smart agent harness can do quite well compared to large models which cant be run locally.
Personally I would never trust a coding agent or agent harness from Meta.
I agree with their open-source model approach, but actually trusting Meta… to protect my privacy and my data… when it’s running on my personal hardware…
Not . In . A . Million . Years - that ship has sailed
If there is anything meta can do to regain hearts other than owning up their evil deeds, radically change their business model and paying up for taxes and damages, then the world is truly fucked and corporations will continue to win.
Meanwhile those of us with 128GB RAM plus some VRAM don't have any good modern (last 8 months) open weights models to make use of all that. I don't care if it would run 5 tok/s, I want a smarter model than Qwen3.6 which avoids loops and can handle more context than 80k before crashing.
How are you handling the tradeoff between quantization for device fit and accuracy loss on tool calling? That's where local agents typically break down in production.
Meta's clearly changing strategies back towards their original "frontier open source", but this time around they have a lot more competition from leading Chinese labs.
I'm all for it though, and I think Glimmer is a fantastic bet on locally-hostable models. I for one would love to self-host as much as I can.
The more open weight models get released the greater the market for personal and small business oriented hardware to run these models. This will drive lower cost hardware, which has stagnated in recent years due to most software not needing the performance and capacity.
"Meta Muse" immediately made me think of Metamucil.
Product teams really need to hire at least one or two people with a 12-year-old's sense is humor. They need to winnow all the potential stupid jokes out of their product namings.
PSA: Fast RAM isn't going to be getting cheaper anytime soon. Acquiring inference hardware is a really good way to own an appreciating hard asset. Learning how to use it and cool it is a hacker's journey worth taking. My 4090 I bought in late 2022 for $1600 is selling for a cool $3,489.95 right now, and going strong under nominal use. My DRR5 has tripled in value, my nvmes almost doubled. I grabbed a 128GB M5 Max MacBook Pro when they were still available and told all my friends to buy at least one. With that and a base M4 Studio 36GB, HuggingFace rates that hardware as:
> Amazing!
You have a total of 128.94 TFLOPS of computing power. 71.3% percentile on scale of "GPU Poor" to "GPU Rich"
The way I see it, these are amazing machines that the richest folks are hovering up. I think they should be in the hands of regular people as much as possible. They depend on an incredibly global, increasingly fragile supply chain. If the become impossible to produce, their value would increase tremendously. I think they will become really valuable to you to use the tokens directly, but if that isn't the case, they can be rented out or resold. Please don't just buy any hold. Let's try to get as many people that can use them for decent things that help humans. For example:
As an industry, I wish we would stop calling these things "open weight" because it is too easy to confuse with actual "open source", which they are not.
Photoshop source code+ OSI license = open source
Photoshop binary you can run on your own computer = open weight
Photoshop SaaS web app = closed, proprietary (Opus, GPT, etc.)
"Open weight" models are still just binary blobs that are completely inscrutable. It's like bringing home a dog from the rescue and just hoping that it doesn't have a tendency to bite kids in the face. You just can't know. The only thing that you can do is try to add more training (fine tuning) telling it not to bite kids.
I don't think the FOSS community has ever accepted this, but somehow we're feeling like it is okay now.
Photoshop binary you can run on your own computer = open weight
I don't think this is a correct analogy. You are not allowed to distribute modified versions of the Photoshop binary. Most open weight model licenses allow you to make and distribute your own finetunes, etc.
I believe that comparing LLMs with traditional deterministic software is fundamentally misleading. It is extremely difficult to truly interpret what LLMs do internally, and as of now, nobody fully understands it. Even if you trained the LLM yourself, there is no source code you can simply read and learn from.
Sure, having information about how these models were trained is helpful for reproducibility, but it is basically impossible for anyone without substantial capital and access to the same (likely copyrighted) data to reproduce the model. For normal users, owning the model weights essentially means owning 100% of the model, you can inspect and study the weights in much the same way as the lab that produced the model can, you can modify the weights, and you can use and distribute them if the license allows you to
Given an open weights model trained to sometimes bite kids, we can’t train it to not bite kids, even though billions of dollars of research have been thrown at this open problem.
Given an open weights model trained to never bite kids, you can get it to bite kids with 10 prompts and a linear projection, the known simple algorithm doesn’t even need a backwards pass.
It is useful to indicate you can run the weights on your own hardware. That’s categorically different from most other commercial offerings. It’s as if your adobe example ignores the reality that would exist had photoshop been invented in 2019: cloud only.
I am extremely well aware of how rescues evaluate dogs. And I'm also fully aware that they do not know the full history of the dog. They go through a limited set of testing and interrogation to evaluate the safety of the dog. That's it.
Will be interesting to see how Qwen3.8 27B compares against this once it releases this week. Seems like dense 30B is back in fashion?
EDIT: An open weight version of Muse Spark 1.2 is going to be released as well:
https://x.com/alexandr_wang/status/2086756152034066792
https://xcancel.com/alexandr_wang/status/2086756152034066792
Based on the benchmarks, it seems that Muse Glimmer barely edges out against Qwen3.6 27B, except for tool-calling skills (MCP, etc.). I wouldn't be surprised if they released it now because they are afraid they wouldn't beat Qwen3.8 27B.
I am glad they released it because I think we need a competitive culture of open weights that isn't just geopolitics.
But I have to say, I quite like the way Muse Glimmer thinks and talks. It's a cocky bastard in tone, but it's quite good, and its thinking traces are relatively terse.
Do AI companies make release plans based on upcoming other models like this? I would think all the processes that go into the repository and weight infrastructure pre-training, checkpointing, knowledge distillation, model compression, post training pipeline, ecosystem integrations, inference API, benchmarking, human eval/safety/alignment, docs, etc... all that dictates the release schedule.
Any company working in a competitive industry is generally aware of what their competitors are doing. PR is an important aspect to market success, so it factors into release schedule. It may not be the dominant factor given engineering constraints, but yea, it’s certainly a factor, and a large one at that.
Yes, not every model release is reactionary to other labs. Either they had hints for the release of other models or they cut efforts in late stage testing of the models to hit these earlier release dates. There’s always some flexibility. And there’s certainly the incentive to cannibalize the news cycles for competitor models.
I could imagine pulling out all the stops to get a release over the finish line a week early if you're worried about being surpassed by another release
Yeah but you can probably have everything ready and then accelerate as necessary. Meta itself did this when releasing Llama 4, it was a really botched release right when they were feeling the heat from DeepSeek and others.
> ...all that dictates the release schedule.
AI companies release models when they are ready - not on a set schedule. The steps are required to produce a release candidate, so a company can choose to bless the RC with the best metrics at any time - or alternatively continue (post/)training newer RCs if they feel they can do better, and they have time. When a model is ready for release is subjective, and can take into consideration what the competition is doing.
An obvious counter-example to schedules driving releases is the still unreleased Gemini 3.6 Pro.
There has been a long history of AI model releases made shortly before or after a major planned release by another company. Almost always to upstage or steal thunder.
Just recently, Minimax H3 released as open weights on the eve of Seedance 2.5 global availability. It's not as good, but it's good enough and it's completely open.
Flux 3, which is nowhere near as good as either, suddenly announced their release once news of these other two became public. They knew if they waited they'd be ignored. It didn't really help them much, unfortunately.
The LLM releases are even more rivalrous.
And don't forget all of the competing launches planned before Google IO or major release events.
Companies like to eat into the news and press cycle of their rivals.
I've seen it here on HN (it's particularly noticeable via the /active page) multiple times. If Google, OpenAI or Anthropic release something significant, odds are good you'll see a headline from one of the others.
BFL is in a rough spot here too. It’s pretty much looking like a repeat of the exact same situation they had when they released Flux2 at the same time Z Image Turbo came out and completely overshadowed their launch.
Minimax H3 can run exceptionally fast (10 minutes for a 15 second 0.5mp video and that's stock cuda 13), works on 16 GB VRAM GPUs, etc. If Flux3 is anything like Flux2, it’s going to require an absolute monster truck of a machine and still run significantly slower. Even if it’s a better model, that won’t matter as much if nobody releases any LoRAs or fine-tunes for it.
Not to mention BFL licensing often feels deceptively confusing and restrictive.
>long history
Seems a bit premature of a statement lol
If you start counting since WaveNet or BERT, it's been ages. Especially when it feels like decades of advancements happen every single year, and rival labs are always trying to one up each other.
I don’t start counting since we WaveNet or BERT so there you go!
Even if I did, we’re talking barely a decade
the last few items there (benchmarking, human evaluation, docs) can be rushed or skipped by leadership if they want to beat comp. they probably spend a few weeks on those things normally
One window that can be shortened is working with software ecosystem and upstream partners; think day 0 on together, fireworks, Unsloth, etc. That obviously happens from partners getting embargoed weights early.
It will also be very interesting to see some direct head to head benchmarks between qwen 3.6 27B (let's say all at Q8 XK quantization, using the GGUF that unsloth publishes as a baseline) vs 3.8 27B. Particularly in tool use, terminal use.
The whole class of what can reasonably fit in a single GPU is an interesting category of LLM, and based on the results I've seen from 3.6 35B A3B and 27B versus what existed a year prior, it seems there's a lot of room for advancement.
I would hope that Qwen 3.8 is better. It's been 4 months, and we've seen almost no progress in this space.
As people have called out, Glimmer appears to be a trade-off rather than a clear winner.
And from what I've been reading, no one is expecting Qwen 3.8's model in this space to be a clear winner, but just slightly and marginally better.
That's a little concerning as DeepSeek v4 Flash proved at it larger sizes there's a ton of room left to compress knowledge.
If we don't see something that's substantially better in the ~30B param space soon - it would appear we might've saturated that size with knowledge.
> If we don't see something that's substantially better in the ~30B param space soon - it would appear we might've saturated that size with knowledge.
I wouldn't be quite so pessimistic. We may have saturated the current approach, but I think there's a lot still left in terms of compression, attention, active parameters, caching etc. etc.
I don’t think four months without a major breakthrough is cause to abandon all hope just yet. ;) The wild pace of LLM development is highly atypical, and we’re still in the ‘initial rush’ phase of development.
For contrast, the Newcomen steam engine (widely considered the first commercially useful engine) was used for over 60 years before the next major improvements. Now, 300 years later, we’re still finding ways to significantly improve heat engines.
> For contrast, the Newcomen steam engine (widely considered the first commercially useful engine) was used for over 60 years before the next major improvements. Now, 300 years later, we’re still finding ways to significantly improve heat engines.
Off-topic, but I stumbled upon the first Newcomen engine imported into Australia in a museum in Sydney and I was unexpectedly charmed (not an Engine Guy). It's large, but nothing like the awe of "mega-engineering", it's crude, but it clearly has such amazing utility (when compared to a reality without it) and it changed the world
I honestly expect that major advances in the open 30B dense space will take about a year, but expect incremental advances every couple of months from different developers in the meantime.
Qwen 3.6 27B was already a massive gift to smaller homelabs around the world; anything more is just a delightful surprise.
For so many non-coding workflows, tool calling is more important.
Qwen3.6 27B has really punched above its weight for a long time. It’s shockingly good for its size. Very excited to see what 3.8 can do.
Qwen3.6 27B is the go-to medium sized model for coding, so beating it is not a small achievement
There is a finetune Qwen3.6-27B-Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO which seems to be as good at coding as vanilla Qwen, but way, way better at creative writing than Qwen and even better than Gemma 4 26 and 31b.
I saw that one in the "Popular models" sort at Hugging Face and tried it on some tasks I do frequently to compare models, and it feels damaged by the fine-tune, to me. It wrote security bugs into the code (probably just sloppy thinking, not intentional), it exhibited looping behavior in some configurations in llama.cpp, configurations I regularly use with the regular 27B, and it failed to write unit tests without being prompted (though the regular Qwen 27B tends to do so if it sees there are unit tests for everything in the repo). They have good benchmarks, but I'm not going to trust it. Also, that name is absolutely crazy.
> but way, way better at creative writing than Qwen and even better than Gemma 4 26 and 31b.
I suspect this is the only use-case I would consider...and I don't really have a use-case for "creative writing" that I would delegate to an LLM. I suppose for dialogue generation in games?
But yes, hard agree. Why on Earth would you ever want to write code with a model that is supposedly "jailbroken"? So it can put great backdoors into everything it touches? Pass.
I've noticed most fine-tunes, whether "heretic" models or something else, tend to be over-fitting, or something, at least some of the time, and get kind of chaotic. I want to believe normal folks with normal resources can be involved in this stuff, as I'm working on fine-tuned specialist models as we speak, but it seems like it takes notable investment and time. My first experiment was teaching a little Gemma 4 more to write more like me with a LoRA (like you, I don't want to use a model to write for me, but I did want training data that I could ethically use, and I've written several million words on the internet over the years), and it wasn't what I would call a success. It either wrote like an asshole (which I only do, like, 15% of the time) or it just borrowed a few of my quirks, like too many ellipses, if I applied it less heavily.
> I suppose for dialogue generation in games?
I use it to write short sci-fi stories. Life is not only about being an SDE.
> Why on Earth would you ever want to write code with a model that is supposedly "jailbroken"?
I need a model which decent enough at coding to write boilerplate for me, yet I do not need to unload in case I want to write an article or enguage in my writing hobby.Exactly why Gemma 4 are so popular.
> So it can put great backdoors into everything it touches?
What a strange take...
Yeah, this became my main coding model.
It seems to know tooling and coding.
Yes but it's also about 13 years old in LLM-years and 3.8 drops in a few days.
Yes, and also waiting for the next iteration of Gemma. Muse or Qwen are optimized for coding, while IMO Gemma is still better for non-coding tasks.
https://x.com/osanseviero/status/2086107547535122767
I am working on a project where we have to classify customer calls into more than 10 categories. As the client wants everything locally I tried a few local LLMs. Gemma turned out to be the best model for this task. The classification accuracy is impressive, and the client is happy that I am using an American model.
I guess the client is American.
You can partially tell by the tokeniser; which gives you some hint into the training corpus mix.
</div> is four Gemma4 tokens, but one Qwen3.6 token.
Looks like we have a /r/localllama dweller here.
Where do you find this information for each model?
When you look on HuggingFace.co at the files of a model, for each model you will see a file "tokenizer.json".
In that file you can see all tokens and their corresponding numeric codes.
The tokenizers are included in the open s̶o̶u̶r̶c̶e̶ weights releases; you wouldn’t be able to use the weights without the corresponding encoder/decoder, in fact.
> Will be interesting to see how Qwen3.8 27B compares against this once it releases this week
Considering that Meta distills Qwen[1] (and should!), it'd be hilarious if Muse loses the head-to-head; the "distillation attack!!1!" people claimed distillation on release n-1 is enough to match the intelligence of the latest version.
1. They wrote a paper about it
It’s really interesting timing, Qwen over thinking is what kills it for me. I’m just glad we have more options in this size class now.
I've been using Qwen3.6 35B A3B, and with reasoning turned on, I'd say 2/3 (give or take) of the tokens for a response are thinking tokens. Which at 70+ tps locally, that isn't that awful. I run an 80k context across 4-10 "agents" for my solo TTRPG, where Qwen is the GM, each NPC at a location, the director, and the narrator.
Each turn is about 45-60 seconds to generate all of the various responses. The GM and director have reasoning on, and the NPCs/Location/Narrator do not.
It's a fairly good "engine" for that. I'm not sure how a denser Qwen would do here regarding speed.
I like the tabletop RPG use case, and wanted to say: If your hardware likes it you should check out Gemma 4 for creative DMing use case. I found it to be much better at holding the plotlines and being creative on gaming turns. My experimental case was an audio-only Zork and Gemma 12B and even E4B were pretty good!
This sound very interesting, do you have any resource I could look at? Me and my son did a very rudimentary (compared to yours) setup to play Paranoia, but this is at another level.
I'll go over my repo, and see if it is hiding any API keys and maybe make it public. The issue I have is it relies on a nuget package that also isn't live (its in my local nuget feed).
I'm not sure what all is needed to make that work for people.
I'm working on something similar. My biggest annoyance is that the overly-helpful LLM was making every die roll succeed. I ended up building some tooling around rolling dice. Also some tooling around character stats and inventory management, so those don't get lost in context compression.
Is there some sort of dedicated tool for this type of setup, or did you hand-craft it ?
Somewhat hand rolled, somewhat claude coded.
Back in 2023 I started my own C# LLM library for doing tool calls and structured output, and over the years it has morphed bigger and bigger, and that is the backbone of almost all of my LLM-based projects.
I've never released it, but its easy to understand, and simple to add your own tools:
I'm sure plenty of better libraries exist for this now, but in 2023, I don't think any existed in the dotnet ecosystem. I've never released it though, because I've never "finished" it.It looks like you’re describing something like an MCP server and a client model.
If you’re in the C# ecosystem you could consider converting your APIs to MCP format tools using the MCP SDK.
https://devblogs.microsoft.com/dotnet/build-a-model-context-...
And then leveraging Microsoft Agent Framework for the client and orchestration side of things:
https://learn.microsoft.com/en-us/agent-framework/
Not parent, but I use Goose for my non-handcrafted Qwen use cases, I’m also working on handcrafting as well. Goose was the only harness that didnt bloat context too much with system prompts (like openclaw) and I could get reasonable web search working with Qwen.
Are you running inference in parallel? 70 tps seems low for parallel execution.
It is on a single 3090, and that seems to be where it averages out. I'll get 85tps on turn 0, but then it settles down to low 70s within a few turns, but holds steady at that.
My issue currently is KV Cache, because I can't keep enough parallel caches running (4 is where I'm at), so TTFT (is that the initialism?) can be long when I have a particularly large scene (basically more than 2 NPCs).
But my harness does let me offload to any OpenAI compatible endpoint, I just prefer local cuz free.
Just to play devil’s advocate: you can’t compare Qwen to a (proprietary/closed source) hosted model and deduce that Qwen is overthinking, as Qwen gives you the full reasoning/thinking trace while all the proprietary models now give you only a summary “to prevent distillation”, making it hard to properly compare apples to apples here.
You can compare Qwen with thinking to Qwen with no thinking though. I find my results are better without thinking because of overthinking.
No, but you can compare it to the similarly-sized Gemma4 model and see the difference, it's not subtle
You can tell how long the cloud models spend thinking based on the delay.
The Qwen models have a habit of going into thought loops where they go in circles for a while.
People say Qwen overthinks because they analyzed the thinking traces, and Qwen finds the answer relatively quickly but then second guesses itself multiple times for another 20,000+ tokens. Regardless of what other models do, that's clearly overthinking.
Qwen thinking is really good in Mandarin; and probably natively trained the most there.
Try a system prompt requiring it to think in Mandarin, while still delivering the response in the user’s language.
This is most likely because the vast majority of the information the model absorbed during training was in Chinese. As a native Mandarin speaker, I frequently need to convert the prompt into English and output it in English in order to avoid that the model falls back into Chinese reasoning logic.
PS: Switching the thinking process from Chinese to English can also significantly circumvent certain self-censorship mechanisms built into the model.
Is the quality of the thinking better or it's just shorter since Mandarin is more compact?
Disable thinking? I think many harnesses disable thinking on Qwen anyways because it interferes with tool calling.
Llamscpp provides reasoning budget and message. You can use the message to redirect it.
Once you get the agent and message consistent,itll keep moving.
You can use any message you want, but the model was tested to react reasonably well to the specific token sequence of "\nConsidering the limited time by the user, I have to give the solution based on the thinking directly now.\n</think>.\n\n" (from a Alibaba paper, struggling to find it now)
Edit: arXiv:2505.09388 Qwen3 Technical Report
Makes me feel hopeful. Things felt more positive around the llama 3 era. Now it’s like a dark, dreadful race.
I feel you. not sure if the "Glimmer" (of hope) branding is intentional to capture this vibe
Well, it has to, since even the MoE models can't really hold a conversation.
yes i think everyone is waiting to see that ;d, i've been on qwen for the last year and a half now.
> Seems like dense 30B is back in fashion?
Huh, well... no? Gemma A4B and Qwen A3B are quite popular in fact. I'm sure 3.8 35B A3B will outperform 3.6 27B by all metrics
I'd be skeptical w.r.t. "by all metrics".
Qwen3.6 is a definitive, significant downgrade from Qwen3.5 for creative writing and prose for example. Yes, it's better at agentic and coding, but it regresses in many non-coding areas compared to Qwen3.5.
Of course, I do expect the 3.8 ones to perform better for agentic coding.
One thing I would caution is staying out of the prediction market like this.
Tech tends to get boring when you judge current products against the hypothetical capabilities of unannounced products that may never ship. It's like comparing Nikon cameras against Canon camera rumours, or comparing iPhones against unannounced and therefore largely imaginary Samsungs.
- If they do a Qwen 3.8 35B A3B (and I hope they do because I love the 3.6 version)
- and if it beats 3.6 27B by all metrics
… then the local open weights world will be a better place.
But they have said nothing about it and they dropped several weight classes for 3.6, so who is to say they won't drop the 35B? And even if they don't, this is a tall order; why would the MoE tradeoffs no longer be apparent? (Again, I really like both the Qwen and Gemma MoEs)
FWIW I am enjoying testing Muse Glimmer — it's really quite impressive on chat, has nice terse and even amusing thinking traces, a bit of brass to it, and I'm hoping it will be good on agentic stuff.
> Seems like dense 30B is back in fashion?
Surprising that Meta don't host this model, even as rate-limited free-tier.
> open weight version of Muse Spark 1.2
Wait. Is this "version" different from what Meta serves?
It seems worse than 3.6, but a bit smaller.
UPD. was wrong on smaller, it's actually much larger
I'll back up your smaller claim, but be specific that it's UD-Q4_K_XL size:
- muse glimmer: 15.9GB
- qwen 3.6 27B: 17.6GB
My video card is so close to its limit that these GB thresholds are mattering too much for me :D
How is 30B smaller than 27B?
It uses fractal compression
They say it is trained with quantization awareness, so it should only be 15GB or so. Qwen was only trained in FP8 with QAT.
UPD, NVM, got misled by comments here. It is actually almost 60 GB so much larger
Quantization awareness doesn’t change the size of the weights, just means it won’t degrade when quantized. QAT = quantization aware training. They will both be very similar in size at the same quant.
You're mixing up sizes of different quants. The 60GB is unquantized, and Qwen's unquantized size is around 54GB. Their sizes as like quantization levels are similar.
From my perspective it doesn't make sense to talk about the number of parameters. What matters is model size in bytes and its performance at that certain size.
Meta actually relesed official 4 bit quants in 17GB, but I haven't seen any indication that training was quant-aware, so the quants are not going to have same performance. 3.6 27B has official FP8 quant that AFAIR was trained with quantization awareness.
The best example is last year's gpt-oss which was released prequantized in mxfp4 so 20B parameter model was under 14GB and 120B was under 70GB right away.
> but I haven't seen any indication that training was quant-aware
readme on huggingface says they've benchmarked the quants -- for 17GB quant reported 1% avg loss across 15 benchmarks (sadly no breakdown).
I assume that's strong enough signal for QAT. Not just first party quants, but they cared to monitor degradation.
> sadly no breakdown
That's exactly the point. We know short context knowledge stuff does not regress with quantization. But I expect agentic intelligence to suffer greatly.
If I were to pick one bench, I would like to compare quants on TerminalBench Hard. But then Glimmer already loses to 3.6 27B on it by a large margin.
Remember when we needed 200 servers for an enterprise website because Apache used one process or thread per connection - and Nginx collapsed that into a single box overnight? That moment for LLMs is near. It’s going to move us from the big iron era of AI to small portable brains. Nature has already proved it’s possible with 20 watts and very little heat generation. And I think the data center buildout will end in carnage.
Side note! Nginx was by no means the first web server to use a non-forking mechanism, nor the first open source web server to do so. Certainly Zeus (which was closed source) was earlier and very useful in this sort of application, and so was thttpd (open source, still exists as Merecat). I used thttpd quite a bit for single box applications and at one of my employers, nginx replaced a mixed strategy with Zeus, Apache and thttpd (and we tested one other whose name I can’t recall).
Non-forking httpd servers using select() were a popular little coding challenge for a while in the 90s. Spinner was one of them.
Nginx’s real strength was being able to proxy and cache HTTP using that same mechanism, so you didn’t additionally need to deploy Varnish or some other appliance.
As to whether this is a good mental model for what is coming for local LLMs, I am not sure I am convinced. Apart from more quantisation-aware training, perhaps binary and ternary aware training, custom inference engines per model, and maybe some improvements in diffusion models, the grand challenge in small footprint LLMs is training really small reasoning and tool use models, and so far it’s far from clear they can deliver.
Truly tiny models will not be viable as general coding assistants; even 12B dense is too small and you will find plenty of people who will tell you that 26B/4B or 35B/3B MoE is too. Though perhaps they can be trained for single languages, like just Python or just TS/JS.
More likely is the idea that 30-40B dense models might be good enough for most things once low cost and likely bespoke hardware catches up.
But I don’t think any truly profound advances seem likely in software or training alone. I am no expert but it feels like we’re already a lot closer to efficiency than we were in your analogy, and the gains are perhaps not going to be much more than small increments.
Maybe we will see something like a ternary 60B/10B MoE model turn up. But at the moment at least I am not sure where the incentives are to train these.
> Nginx’s real strength was being able to proxy and cache HTTP using that same mechanism
Fun fact: Igor Sysoev originally wrote mod_accel [1], an acceleration and reverse-proxy caching module for Apache before he made nginx.
I remember using that module in the mid-2000s as a load balancer (and to offload SSL encryption/decryption as it was a CPU-expensive operation).
[1] http://sysoev.ru/en/apache_modules.html
We've barely even started on optimizations like advanced language aware grammars, and specialization routing (dynamically loading fine tunes or seperate weights for specific tasks or languages).
Right. But those still sound like modest gain territory, or qualitative gains within the same rough performance, rather than the "breakthrough" improvement notion I was responding to.
My naïve impression is that the LLM world will keep delivering these fractional improvements for some years at the cost of simplicity. And sure, ontological support seems quite promising.
But making things radically better or faster for small models in the way that is hypothesised, that feels like it can only come as a result of hardware performance improvements and likely architecture changes.
Because there's no free lunch, right? Speculative drafting for example, noticeably improves performance until acceptance rates drop for reasons that have to do with the particular application, and then it starts hurting you, especially near the limits of the memory bandwidth. Because once it is wasteful it is an extra overhead.
I gather Gemma 4 supports, in principle, dynamic speculative draft lengths, to help with this — where it will stop making bold predictions when the success rate falls. But I'm not sure if any of the inference engines I've tested with support that.
I think small models are miraculous — I still think Gemma 4 12B is astonishing — but I guess what I am saying is that I think maybe technology is moving quickly enough that the developers are done with the low-hanging fruit.
The gains wouldn't be "free lunch", it's the result of time and effort researching optimal design and architecture.
Even if the idea of "no free lunch" was taken liberally discounting the cost of research, it would only be limiting to systems built from a foundation of optimization, but that's not the case. The foundation so far has been one of brute force scaling. Usually meaning there is lots of room for optimization.
> The foundation so far has been one of brute force scaling.
In cloud AI, sure. But in the smaller open weights model territory it feels like we're well into optimisation?
Also lighttpd! Released in 2003, one year before nginx (2004)
Everyone keeps repeating this who doesn’t understand the underlying technology.
Small llms are still way more efficiently server on big GPUs.
Sharing server capacity takes advantage of the massive parallel throughput and sharing of memory bandwidth.
You are sharing the GPUs with thousands of concurrent users.
FWIW it is entirely possible to square the notion that small models will still be hosted on cloud hardware with the idea that the data centre buildout will end in tears.
Many analysts (and Microsoft) think even now that if everything committed gets built there will be considerable oversupply and there is not the revenue to pay for it.
If small models do continue to improve in unusual ways (I think there are limits) then the marginal need for cloud AI compute could fall precipitously beyond current estimates. The marginal need for consumer AI could almost totally collapse if someone makes good progress on very small reasoning and tool-calling models (which is a modestly big if)
The possibility of the data centre boom resembling the Chinese real estate bubble is not inconsiderable.
So far everyone seems to be consistently GPU-poor, despite the huge buildout, and usage keeps going up drastically. I don't know what would make usage drop.
Every time they've made smarter models we've wanted the smarter ones, and local models runnable on typical hardware are still very far behind in speed and intelligence (as neat as they are)
> So far everyone seems to be consistently GPU-poor, despite the huge buildout
That is seemingly not the case. The buildout is actually slow; almost nothing of these giant projects has been completed. Nobody will say how much of anything they have actually finished. And Nvidia have made huge, huge buy-and-hold deals for GPUs that do not have data centres to go into.
Everyone is GPU poor because stuff hasn't been finished but large numbers of GPUs are spoken for, but they are GPU poor on therefore much less demand than is being built for.
Look at how tiny SpaceX's deal is with Anthropic, for example. This meaningfully turned around Anthropic's prospects — allowing them to radically lift rate limits beyond what many users needed -- but it was for just 300 megawatts. Tiny compared to the 31 gigawatts allegedly under construction by the end of last year.
So the picture is partly illusory. GPU prices and RAM prices have been pushed up by the AI firms booking them for data centres they haven't even started building yet, as well as the ones that they've only completed a tenth or an eighth of.
There will be significant oversupply. And if open weights models keep getting good and staying fuel-efficient, that picture gets worse.
If I understand correctly, you're saying people are compute-poor but not necessarily GPU-poor because there's a lot of GPUs out there but nowhere to plug them into? If so, I'm not sure that distinction matters to the GP's point that there is too much demand to call this an oversupply.
I also doubt we can estimate the level of demand based on a single deal between Anthropic and SpaceX (despite which, note, Claude still stuggles at times.) Consider other signals, like Google, who we thought had an insurmountable infra advantage, also renting compute capacity from SpaceX and limiting Meta's usage (along with other clients apparently) to conserve capacity: https://www.cnbc.com/2026/06/28/google-limits-metas-use-of-i...
I am not sure Microsoft thinks there will be an oversupply either; last earnings they announced bumping up their CapEx spend, along with all the other hyperscalers.
Here's a way to estimate how much room there is for demand to grow. Various sources (linked in this comment, along with more analysis: https://news.ycombinator.com/item?id=49089296) indicate that even though a large number of people (50 - 60%) are now using AI at work, they use it for only 6% of their work hours.
That means, even if AI can only address 30% of all work, there is still 5x potential demand growth left! Note, the sources above indicate that AI is even being used in non-knowledge work industries, so the scope is already larger than we thought. This is in addition to the remaining 40 - 50% of people are still not using AI at work. Plus we know that agentic workloads consume way more tokens, so that's yet another multiplier.
But will that demand keep growing? Well, some of those same sources above mention that most executives are planning on ramping up their AI spend in coming years.
Putting all this together explains the hyperscalers' quarterly bemoaning of how strapped for compute they are and why they are spending so much to add more capacity. Given this, an oversupply seems pretty unlikely.
> If I understand correctly, you're saying people are compute-poor but not necessarily GPU-poor because there's a lot of GPUs out there but nowhere to plug them into? If so, I'm not sure that distinction matters to the GP's point that there is too much demand to call this an oversupply.
It's a distinction without a difference if your issue is getting hold of a GPU.
But there's a significance to it if you are trying to use demand for a GPU as a proxy for demand for AI. That is where the industry is making serious mistakes.
> I am not sure Microsoft thinks there will be an oversupply either
https://finance.yahoo.com/news/microsoft-ceo-says-overbuild-...
The rest of your comment I am not going to address because it's kind of unfalsifiable. Hypotheses about what AI might be able to do in principle aren't all that useful when talking about even medium-to-long-term demand for what LLMs and GANs can provably do now.
When the glut of GPU arrives I'm sure humanity will find a good use for all that excess compute, like finally getting back to signing monkey pictures and excreting endless hash based pyramid schemes.
It may not make financial sense for someone retired, not into tech, and/or data privacy to host their own LLMs. However if usage of AI in day to day lives continues to increase, I think it will eventually make sense for the majority.
Many tasks suited for AI assistants are background asynchronous tasks. They can run in the downtime where immediate demand is low, keeping overall utilization high enough.
Your argument is similar to those who argue that owning a GPU for gaming doesn't make sense when you can stream from something like GeForce Now. However like with gaming locally (improved latency) there are also benefits to local AI (data privacy).
Another big benefit of local AI is control and increased predictability, e.g. in costs or behavior.
We will appreciate it even more when popular agents become more costly or start including more ads.
> Small llms are still way more efficiently server on big GPUs.
Yes, but the privacy aspect means that for many, many applications slower local will still be preferable to faster remote so long as the actual model performance is the same.
The power of small models isn't only that you can run them on local hardware. You can also fully own your data and workflow, and choose/fine-tune models for your specific use-case.
for clarity, I'm not agreeing with GP that small models will mean doom for data center projects
Indeed. Local compute and RAM are a some of the most wasted resource, sitting idle 95 percent of their life. Datacenters flip that ratio.
True, but local setups can run LLM requests in parallel too. In this case efficiency gap is much narrower.
I’m sure you could find this exact same comment regarding technology in 1999.
You can find it today for gaming. Even despite the outlandish rise in hardware costs, there is very little demand for cloud gaming.
Yeah but I presume this has more to do with latency than overall compute, right?
Yes latency, and the usual preference of ownership over rentership. Similarly their are benefits to running local AI too, like data privacy and control.
> Remember when we needed 200 servers for an enterprise website because Apache used one process or thread per connection
I don't, but holy moly. That sounds insane!
I don’t remember that and I was there! The idea that the performance delta between Apache and nginx for any normal workload is anything like 20,000% is absurd.
Absurd indeed. Apache httpd server got MPM support two years before nginx was released.
> And I think the data center buildout will end in carnage.
Isn’t it more likely that they will still all be used to the max? I don’t see how at any rate we will be going “alright, that’s enough intelligence for now”
What specific technical signals make you think we're close to a shift like that?
The researchers who published Attention Is All You Need didn’t have the benefit of the LLMs they birthed. Take a look at the prompt that solved the Cycle Double Cover conjecture, and which has been adapted to achieve breakthroughs in cybersecurity. The field is entering a feedback loop that is leading to exponential innovation. We’re at the beginning of the curve. And right now the big iron data center approach is brute forcing the problem.
Because I wasn't familiar with it and others might be curious too: that prompt is available at https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98... and is just below 5 KiB of text.
Do we know how much token dollars it took to solve?
I dont think its exponential innovation. Rapid incremental innovation is happening very fast with some occasional bigger bumps.
Where can I find more information on the cybersecurity breakthroughs adapted prompt?
Intelligence per unit of compute will almost certainly keep increasing. That doesn't mean having more compute isn't still way, way better.
> It’s going to move us from the big iron era of AI to small portable brains. Nature has already proved it’s possible with 20 watts and very little heat generation.
Agree. I know enough about the vagaries of scientific progress to not put any money on any timeline but directionally, that's where we are headed.
> And I think the data center buildout will end in carnage.
Disagree. And this is quite the leap from the previous statement, btw. The carnage happens if the demand for general purpose GPU compute disappears and even then there are so many ways to salvage the asset.
It'll need a change in architecture for that to happen. For example Geometric Reasoning that's being actively worked on by people like this:
https://www.sophontic.ai
> Nature has already proved it’s possible with 20 watts and very little heat generation
Never thought of it in those terms before.
First point is plausable, moving from bigger models to smaller models. But the nature thing is a bit of an overstatement, yes our brains are very efficient but they are fundamentally different from LLMs so it doesn't really map.
What does this have to do with Muse Glimmer 30B?
Nature takes its own sweet time to come up with photosynthesis or the krebs cycle. What takes 2 billion years for Nature to work out, these large systems will soon do it in 2. They have capacity to compress time in ways the chimp troupe cant.
This is a statement of nearly pure faith not fact. Which is fine. I have a lot of things I believe based in pure faith. The difference is that I don't state them as if they were fact. Which you appear to be doing here.
20 watts huh? How much energy has nature sure is required for lift? Does the same amount of energy scale by size for airplanes?
Can these models compete with Cerebras inference performance? Why would I pay for a 2400 baud modem when DSL is available?
brains do it with 20 watts because theyre analog. llms require massive amounts of power and this isnt changing any time soon without a breakthrough
There are arguments that the brain is quantum, as in parts of it locally using quantum effects. Which if true, might make a counter-argument, as there will be bigger data centers needed if the goal is to simulate the brain classically.
On the other side, advancement in quantum computers would make current LLM inference much faster. Because of the extreme cooling needed, i dont think the energy demand would become less.
With AI companies talking about AGI, i sometimes wonder if they really need the machines for serving inference to customers, or they have a formula for computational capacity that could run an AGI, and they just want to reach that level.
Can you give me some pointers?
Last I heard researches simulated an entire brain of a fruitfly with just a classic neuron approach [0] and if I recall it worked great.
I know a human brain is many orders of magnitude more complex, and that there are some birds that use quantum navigation - so it's certainly not impossible by any stretch, though I am quite interested in this statement since it's the first I hear about it.
[0]: https://flybrain.app/
If the brain does rely on quantum effects, it's still possible the quantum effects in use are able to be simulated efficiently on a classical computer. For example if it's a matter of signal transfer rather than quantum computation, that could be simulated rather easily.
I wonder what company will release the first "quantum LLM"
Alibaba probably. They are doing work on both already, that is if a quantum LLM is even feasible.
> the brain is quantum
Doesn't that apply to everything that exists in the material world?
Naively-speaking, sure. I think GP was referring to the act of information-processing, specifically, though.
And a breakthrough in hardware, specifically.
There are exciting developments underway in analog computing.
https://xcancel.com/finkd/status/2086755195535413696
"... Soon we'll also release the weights for Muse Spark 1.2, our latest foundation model..."
This is bigger news - good for self hosting enthusiasts and a strategically sound move for Meta. Any push towards 'anti Chinese' models will directly benefit Meta as the competition on the frontier open-weights American models is almost non-existent. Meta will have no problem being #1.
It wouldn’t surprise me if Meta does become the #1 American open weights provider, but I doubt it’ll be easy. Thinking Machines has a good amount of talent behind them as I understand it and their Inkling model was decent (admittedly not great though). I think Meta’s biggest problem is going to be internal as there’s be a bunch of headlines posted here on their talent retention issues.
Poolside Laguna was quite good too (if you look beyond some of the teething issues).
Had Deepseek V4 Flash 0731 not launched, their latest Laguna release was really intelligent at non-coding tasks and it would have been my go-to model for my local workloads.
For me Laguna frequently slightly corrupted text then it would be unable to notice the difference and get stuck making the dumbest conclusions. Thinks like typoed directory or function names. It was a great model other than that, but I ended up just going back to Qwen3.6
Yeah, but if it's huge, how many can run it? Many folks struggle to run 200B+ models
Two DGX sparks will run the full Deepseek V4 Flash full. (It is definitely expensive, but relatively easy and compact; and extremely power efficient)
Many people don't have $5000 for DGX Sparks. With that said, it doesn't take much to run Deepseek. I run it on a sub $1000 system 128gb 2 3060 at 6-7tk/sec and then on a $1000 system with 10 MI50 GPUs built when the price was cheap.
What about Inkling? It's a quite large model that for some reason isn't discussed much.
Meta is rocking AI. As of last week I have been using their excellent muse coding harness with their model Muse Spark 1.2.
Starting this morning I am running their new local 30B model muse-glimmer on my old MacMini 32G using Ollama (remember to increase the context size!) and pi coding harness. I am getting good results with muse-glimmer running locally, with the caveat that everything runs slowly (e.g., give it a task and then go walk outside or do Qi Gong exercises for a while).
Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385
When that article was first published , I started looking into llama.cpp. With the help of an LLM I researched the knobs to turn that made most sense.
Things went from "local models are useless unless you have a 512GB GPU I guess" to "oh.. I can do a lot of stuff locally now!"
Had a similar experience. Llama.cpp compiled natively; parameter sweep to find best options fitting my use case for the qwen models with 16GB VRAM. The whole thing packaged into a portable container.
What do you use instead?
If you need a GUI, Unsloth Studio and LM Studio are both great frontends for llama.cpp. If you don't need a GUI, llama.cpp is the business for single-user deployments. Easy to use, always gets new model support very quickly, built-in Hugging Face client/cache support, works on probably everything (Mac, ROCm, CUDA, Vulkan, etc.).
I use llama.cpp w/ llama-swap
https://github.com/ggml-org/llama.cpp https://github.com/mostlygeek/llama-swap
FYI, llama-server can now be run in router mode so llama-swap is probably only needed for more exotic scenarios.
I’m using llama-swap because it can manage arbitrary backends, not just llama-server instances. I have llama.cpp chat and embedding models running alongside whisper-server all behind a single endpoint with per-model TTLs so they don't fight over the limited vram I have available on this box. Native routing could replace the llama.cpp part but not whisper so I guess I'm exotic ;)
I'm running it in router mode, but people on Reddit were recommending people use llama-swap instead. Am I missing something by using router mode?
I believe it's useful for running multiple llama.cpp forks at the same time (e.g. a model you want requires special patching)
Thanks for the link to llama-swap. Didn’t know about it and will definitely install it.
try oMLX or vMLX - both great projects that offer some amazing performance optimizations for Apple Silicon that utilize UMA and NVME caching efficiently.
https://omlx.ai https://vmlx.net
That said, it's been a few weeks since I've looked so maybe llama.cpp has those features now... they really do move that quickly.
Jan is so much better.
The link in that discussion has a section about that
lm studio
seems to underperform on Terminal Bench compared with qwen3.6-27b: 51.7 vs 60.7
To be honest, I never give benchmarks a look. I just use the models for whatever I need to work on, so I can't really make comparisons that are useful for other people.
Newb question but I’m curious what would help it to run faster? Would it need more vRAM or just system memory?
The biggest gain you'll get is faster memory, provided you have enough capacity to load all the weight into vram. The DGX sparks and Apple silicon memory bandwidth (and also memory access latency) drag down the decode speed quite a bit.
I have two GPU rigs both with 2x RTX Pro 6000, can get ~250 tk/s decode with deepseek-v4-flash in native mixed precision. For context, in antirez's dwarfstar project he only gets ~20-40 tk/s on the same model @ 2bpw on M5 Max.
The latter is for sure usable if it's your only option, but it's really hard for me to personally go back to speeds like that when I've experienced the former.
(Also worth noting dwarfstar only has experimental support for dspark spec dec, when that lands it will definitely give a big boost at higher acceptance rates)
It runs very quickly on my RTX 5090 fwiw. Whole thing is loading entirely into vRAM with a ~130k context size (the max) fitting as well.
That's a $5k 32GB card for anyone who doesn't know all these off the top of their heads (like myself).
Unsloth has quantized versions uploaded: https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF
The quantized releases often change in the weeks following release as new improvements are discovered, so either use a tool that checks HuggingFace for new versions or manually check back in a few days or weeks to check for improved versions.
Initial reports are good. It hasn't been out long enough for anyone to really test thoroughly, but the people I know who have stable non-public test cases are reporting impressive results compared to even Qwen3.6 27B. That's a good sign that this might not be benchmaxxed (trained to excel at public benchmarks with less impressive performance on general tasks) which has been becoming common with recent releases.
www.reddit.com/r/localllama is a good place to keep up with the details from people who are actually using it. It feels strange to recommend a subreddit over Hacker News, but on this topic the /r/localllama threads are much more on topic right now if you're looking for information about the model.
There are some initial reports that even the 2-bit quantization is looking somewhat usable. That might make it small enough to squeeze into 16GB GPUs. I'd take those reports with a grain of salt because early tests are often optimistic and I've yet to see good results from anything 3-bit or less, but it should be fun to experiment with.
are these all uniform quantization? or mixed and matched by layer (can't tell from the naming scheme)
Some interesting findings from the chat template designs:
1. The template name is Onyx ATEM as found in the tool call exception message
2. It appears to be following a harmony-style chat template. But the tool use seems to be a xml like :<atem:function_calls> / <atem:invoke> / <atem:parameter>
3. atem: a internal joke of meta in reverse?
https://huggingface.co/meta-models/Muse-Glimmer-30B/blob/mai...
The XML tags are similar to <antml:xxx>, which is obviously Anthropic ML (or ANTrophic xML).
I think it’s likely 3; meta in reverse. While tokenisers and preprocessing can catch it, you want your special tokens to be unique and not present in the original corpus. <meta: is likely too common.
Likely inverted "meta" to avoid collision with HTMLs meta tags
> atem
also perhaps taking some small joy from the lexical similarity to aten[0] namespace that lies at the heart of pytorch
[0]: https://github.com/pytorch/pytorch/blob/main/aten/src/README...
atem also means breath in German
With the business model for API based LLMs looking iffy at best it seems like we’re heading back to the “server under your desk” era of IT again.
Considering how all the big players are playing fast [1] and loose [2] with limits, billing [3] and adding undisclosed changes that burn your tokens on autopilot [4], it can't happen soon enough.
[1]: Limits may change without notice, including due to capacity constraints. - https://support.google.com/gemini/answer/16275805?sjid=14713....
[2]: "standard limits" are never defined - https://support.google.com/gemini/answer/16275805?sjid=14713...
[3]: https://tobyonfitnesstech.com/blog/anthropic-refund-scam/
[4]: https://news.ycombinator.com/item?id=48947776
Not to mention all the other ways they can screw you:
- Middle of the day, servers busy? Swap to Sonnet while pretending it's still Opus. Many people won't notice, and nobody can prove anything if they suspect.
- Middle of the night, server load is light? Put it into extra thinky mode so it burns more tokens to ramp up the bills. Flip the switch where it gets really pedantic about writing lots of extra test cases and verifying against documentation.
- Demand increases, but don't feel like running more hardware? Switch to low bit quants, but have a monitor model swap back to quality if it can tell you're running a benchmark.
Assuming model capability plateaus (I think it will), token providers will be in a race to the bottom to maximize profits at the expense of quality that's very difficult to measure.
It all sounds like having to rely on a dodgy housing contractor that wants to steal from you, take shortcuts AND choose the gold-plated options from their supplier friends, and will start doing this the minute you are not on site supervising. You don't do it yourself (because the contractor is faster and stronger than you in many ways) but you can't leave, so you're stuck on the worksite just watching them.
It's worse though, because you can't really watch them at all. It's very difficult to get quantitative numbers for quality. Even within the same model family, same tokenizer, and complete control over the weights and logits, perplexity and KL-divergence isn't really what you want. Now put it behind an HTTP endpoint, and it's just opaque.
I've seen local models recognize when the task I'm asking them for is likely to be an artificial benchmark.
And any smart company is going to use lightweight models to monitor your sessions. If their sentiment analysis suspects you're close to cancelling, they'll up the knob for a few days until you calm down. Or worse, their accounting tells them that you're getting too much value from your fixed price subscription, so they turn the knob down to encourage you to cancel.
In the short term, the "frontier" models are too good to ignore. But if (when?) that plateaus, I don't see how anyone could trust a non-local model. When you pay an ISP to serve your web site, you can tell if they over-compress your images to save storage and bandwidth. With LLMs, it's just JSON with more errors and pointing to the fine print that models are not deterministic.
One of the frontier companies (Anthropic) is already doing prompt injections on the API, which you pay for.
Right now, the presence of these injections are still visible: count the API's returned tokens/billing data, and you'll start realising that sometimes, your INPUT tokens are inflated! That's their prompt injections.
You can also give Claude a tool like `telemetry_log_anthropic_reminder` and get it to dump the verbatim API injections; which additionally verifies the token maths not adding up.
Yes, Anthropic is tackling their extra injections on your API prompts WAY more than you think, and YES, you're paying for it.
So far I have not observed any visible injections on OpenAI API.
Don't forget the whole debacle over Fable 5 sabotaging the user for "advanced frontier AI development". I still get Fable classifier refusals for nearly any kind of ML work on my 2x RTX 6000 Pro 96GB; so who knows.
> Don't forget the whole debacle over Fable 5 sabotaging the user for "advanced frontier AI development".
Yeah, I've had that happen twice. The second time was about some attention weights thing, and it kicked me to Opus. When I edited my question to make it clear I was talking about Google Gemma, Fable was happy to keep talking. So clearly it's not about safety or cyber security - they're happy to tell you about what their competitors do.
These kind of tricks will completely break API customers and be super visible, since most companies deploying API at scale have ample telemetry, evals, etc.
Although, selectively applying it to consumer subs is probably beyond likely at this point.
I find Claude doing a lot of pointless confirmation at night when it will ask me about things that it would normally just do during the day.
Frustrating to be like “do X overnight, don’t ask me for input” and come down to find it having worked for a few minutes and then stopped.
Doesn't setting the question auto-continue timeout before you go to bed fix that?
https://code.claude.com/docs/en/tools-reference#question-aut...
Interesting, I'll try that, but usually it's not a question, last night it said:
"Ready for you to rebuild MAME, restart, and retest. If the freeze goes away → timing was the cause."
when previously it had been happily auto rebuilding and running the mame test
What areas do you think model capability will plateau in, and why?
I've got nothing but hand-waving, but after you've extracted all the smarts from every piece of text ever created, how do you get more?
Alpha Go had a game where the models could compete against each other. That let it become super human. What's the intelligence game we can create for LLMs? Even if you invent something, will it make the model smarter in a way the market values enough?
Then there's a race to use the weights more efficiently, or to offload information that shouldn't be in the weights in the first place (Karpathy's Cognitive Core). I like to imagine we train the models in something like Lojban, have a lightweight model translate from human language to that, and you can update the Sqlite or Postgres store it uses for knowledge.
And there's no barrier to entry for agent harnesses. So whatever loops or recursive orchestrated council of elders idea comes up, that won't protect the monopolies (duopolies).
Anyways, depending on your definitions, I think we'll hit AGI, but I don't think we're getting a Singularity this time around. Again though, this is all just hand-waving.
I think you’re thinking about it in slightly the wrong way. We’re not throwing more data at frontier models in hopes they get more/better capabilities somehow.
We’re either: setting up a verifiable task, and doing RLVR to get the model better at achieving that task.
Or we’re simply asking: “What do we want the model to do that it can’t now, and how do we curate data that would benefit it on that task?”
Most useful capabilities going forward aren’t going to come from data accidentally found on the net; that’s already all been scraped. You need to develop the dataset that shows how a model could perform insert task in its provided environment, and this still requires a decent bit of human ingenuity.
Yeah, there's room for improvement at every level, but your specific example: How do you get more and more difficult tasks where you can steer the training? To me, that seems limited by how creative humans can be. How do you get past AGI and into ASI with that? If the AIs make the tasks, how could we encourage them to be useful? Maybe you could push for harder and harder math proofs, but other than that I'm not sure.
Anyways, I'd be thrilled to see exponential (or faster) growth. Bring on the Culture, Accelerando, whatever. I just don't see it yet.
Ugh... didn't think about extra thinky mode in the middle of the night.
So many ways for enshittification here.
> at best it seems like we’re heading back to the “server under your desk” era of IT again
Maybe in the very long term. If companies go local, the efficient model is to buy some big hardware to share among developers.
I run local models. Even with 128GB unified memory systems or a 5090 or RTX 6000, the generation speeds X model quality X context length is still far behind what I get from my SOTA model subscriptions. I also pay a lot more for the locally generated tokens in electricity and hardware costs. I'm also limited in parallel requests to the local box. The list goes on.
I really like running local models, but for any given point in time it's more efficient to have a big central box aggregating requests and churning through them. So maybe companies buy $300K servers and try to split it among 30 users instead of trying to buy 30 x $10K boxes.
More likely, they rent time on cloud servers by the month so they can adapt the hardware when new models come out with new requirements.
Then some day in the distant future when hardware is cheap and plentiful again, it might make sense for us to go back to individual boxes under the desk.
Why do you say API based llms looking iffy at best? Do you just mean current profitability due to market pressures from some companies’ subsidized investor money?
Surely, even if you’re just using open weights models, it should theoretically be cheaper to use them in a highly optimized cloud architecture(even with vendor markups) rather than each person serving their own models from much less efficient (and more importantly, much less consistent volume) self-owned “server under your desk”?
LLMs are becoming commoditized, which means the margins are trending to zero. It's a lot less exciting to spend another trillion on a new model if you can barely make any profit. Meta getting out of the game might be the smarter move.
Companies are going pretty quiet about costs, but I see no reason to believe the cost to train a model exceeds $5B. Moonshot AI's entire funding is like $5B, their $300M revenue is negligible but not nothing. Renting the compute to train a model like Kimi K3 cannot possibly exceed $5B and is probably under $1B. It's probably at least $100M, but also plausibly not. I don't think it's likely Meta would be giving away models for free if the compute cost to train them was in the billions. This 30B parameter is tiny, that's not billions of dollars, that's likely millions, maybe even less.
I've been coding using the LLM server in my living room for the past few weeks, and I haven't had this much fun with tech for ages
Can I ask, do you feel the pain of the level of abstraction? I haven't tried local in a few months, but last time I tried, I felt like I was directing a coding exercise - whereas with a frontier model, it feels more like directing a product building. "I need this feature", vs "write code to do this in this file".
> write code to do this in this file
I haven't had to micromanage to this level. I usually start with a spec for a feature, which will be as detailed as I am opinionated about the feature. But it's usually on the level of a high-level context, plus some key implementation details (technology choices, key requirements, maybe an interface/API specification to 80% detail), and then the project already has high-level policies documented about e.g. how to structure files within the project.
Then I do a planning phase, task breakdown, and implementation of subtasks all within the model. I do read through it, but mostly the quality is good and I might make a couple notes. Then I do a review phase, which usually picks up a couple things. I'm moving towards less manual review of results and more automation as I learn what I can and can't trust the model with.
There's definitely a capability gap vs. larger models, but honestly I kind of prefer this workflow, as I stay more in touch with how the codebase is structured.
And it's great to be able to experiment as much as I want without worrying about how many tokens I'm burning or how close I am to a usage limit.
Thanks for that, it's the level I like to work too - what model/quant are you using? How much vram/context and which coder?
I'm using Qwen3.6 27B Q4, max context with pi on 32GB VRAM (although I'm testing out Glimmer on a feature implementation literally right now). Pi is great because it has minimal context added by the agent.
Looking forward to the 3.8 27B release to compare.
Pi and a similar set of tools is also likely similar to the harness these models are trained on. More complex harnesses burn reasoning tokens on these small models and in my benchmarking don't seem to be able to beat Pi ever. Usually it isn't close on some tests.
Some people like it better when they direct the solution because they walk away with a better understanding of it.
This has emotional/psychological aspects (it feels less like LLMs are replacing you), as well as practical ones (overall complexity is bounded by what the dev brain can understand/grasp).
A dev work becomes more and more about reliability, signing off safe software with a litmus test: “I will be on to handle this code failure as if I had written it”.
All the above points towards keeping tight control over some level of abstractions and delegating others.
I find it depends at what stage I'm at with the idea - sometimes I don't want to understand it until it works, because I've wasted enough life on things that didn't do what was promised. But once I know the idea is feasible, yes I would prefer to understand the code at some level.
That’s part of it. There’s also just a natural back-and-forth between what I call “time sharing” and “personal.” When the thing you want is expensive, you share it remotely, but as soon as costs fall, everyone wants it under their desk.
more likely that hosting and delivering the models will be commoditized, much like how DO, Linode, Hertzer etc all commoditized VPSs and server hosting. And you'll end up paying for virtual hardware size (or compute resources) rather than tokens
What do you mean iffy? The major AI labs are gross profitable when selling access to inference. In addition, the best models have trillions of parameters and are most efficiently served on large, expensive clusters and served to many concurrent users.
That’s like saying an apartment building is “profitable” because the rent covers utilities while ignoring the real cost which is the mortgage on the capital cost of the building.
It’s funky math and a good way to quickly go bankrupt.
They make money on each token when you look at the electricity and interconnect fees, but no, I don’t think they’ve turned a profit on their Capex, even a little bit
> The major AI labs are gross profitable when selling access to inference.
Do you have a good source for this?
Well, I can run some models that are better than some of the weaker and cheaper Anthropic models locally, like Haiku 4.5, and solve tasks that would cost ~4500$ every day in tokens, so yeah, they are definitely extremely profitable on inference.
Are you also paying 8,000 employees [1] and funding massive infrastructure [2]?
[1] https://www.makerstations.io/openai-employee-statistics/
[2] https://www.wheresyoured.at/oai_docs/
This release is not a meaningful improvement in any metric over 5 months old Qwen 3.6.
DS v4 Flash update maybe, but it is too big for typical Joe's desktop.
I wouldn't say "server under your desk", necessarily; more of an "Linux getting big" era of IT.
If you want to host the model on the server under your desk, you can. If you want to build a data center on-prem to host it, you can. If you want to pay a cloud provider to host it at their data center until you figure out how to scale it without their help, you can. It's like when people were first building commercial services to support Linux-based OSes, and people were also still hacking on it on local machines.
APIs may still have their place - maybe you just want to throw your devs a known quantity with all of the management built in - but it's not going to make Sam Altman a trillionaire, which is something anyone outside of the SV echo chamber could have figured out as soon as the first real competition to OpenAI emerged.
With how expensive consumer hardware is and will continue getting (due to LLM demand), good luck getting a "server under your desk" for something less than an arm, leg, and first born.
Until A100 prices are reliably under 1.70$ an hour, there is no GPU/AI bubble and Michael Burry doesn't know anything about GPUs.
There are lots of points in a spectrum of choices. DGX Sparks, Strix Halos, and the surviving Mac Studios can easily run these 30B class models, just not as fast. So maybe just the leg, but you can keep the arm and first born.
And super noteworthy is that a 27B model (Qwen 3.6 27B) from this year is a huge improvement over a 120B model (gpt-oss:120b) from last year. The goal posts are moving, but at some point "good enough" is good enough for the kind programming I like to do.
The gguf is up and works, I don’t know if it’s them or unsloth that’s facilitated this but it’s nice because e.g. Inkling still doesn’t appear to have support in llama.cpp which makes it irrelevant to a class of user.
Unfortunately I don’t have enough experience with Qwen 27B to immediately compare, but I do it’s Qwen 3.6 35B A3. It’s much slower obviously but it seems to be way more efficient with its thinking to the point that using it might actually be faster. I find Qwen and some others rehash the same things over and over when thinking without getting anywhere, in mg limited checks here Muse is much better.
I don't really use the Qwen 3.6 27B though I do test the variants (Bonsai, ThinkingCap).
I really like the 3.6 35B A3B for experiments, and it seems OK, but as you say, it spins round in thinking loops more than say the 26B Gemma 4 does. If Muse doesn't actually-wait itself as much it will be very interesting.
I am just downloading it to run my small tests.
I have a custom A3B proxy that caps its thinking off. It is a known issue with the model that Qwen themselves documented but is almost never addressed in any harnesses. I also patched up a few other known bugs in the proxy. I highly recommend you shim A3B and when it hits 2K thinking tokens inject (paraphrasing) 'Time to wrap it up bud! Get to work' into its thinking stream. It almost always gets to work. If it needs more time to think there is always next turn.
In my experience it is almost never productively thinking past that point, just spinning in circles. I also reinject all of the thinking. And there are a few tells that it is getting stuck. I have an optional mode that takes the last few turns and tool calls and shoots it off to DSV4 with a prompt to basically understand where it is at and inject better thinking and or planning. It almost always gets it over relatively difficult humps, but some of the time I don't want things going remote. It might end up with 10-30 cents of DSV4 calls over a hours and the quality improvement is remarkable.
The other thing is I trick it into thinking a web_search tool is a web search but it really just asks DSV4 the prompt. DSV4 is a cheap filter to help prevent prompt injection lol. You can give it other models but DSV4 is my cheap-mode default.
edit: oh! My final 35B A3B tip -- use an extremely simple harness. Pi is good. Pi's default tools almost exactly match what Qwen says they tested the model with (likely meaning that tool set is also what they trained it with or something similar). So, in my experience bigger harnesses don't have a noticeable improve ment on tasks.
Fascinating, thank you. I am trying to switch to pi from opencode (my own thinking loops and burnout are a challenge lately).
It had not occurred to me that you could nudge it to stop thinking with a proxy. Nice idea.
Will favourite your comment and come back to it.
ETA: Incidentally you've helped me put into words the difference between the way Muse Glimmer thinks to the way Qwen thinks. There is a clear sense of urgency in Glimmer's thinking traces.
Having spent a good part of the day with it, glimmer reminds me of Rorschach from The Watchmen. No unessential parts of speech, action oriented, brief and to the point. From a token perspective anyway it’s great, and it seems to hold its own well against more verbose models.
I really do feel like it’s effective tok / s is way higher because it doesn’t waste them.
I am very struck by the way open weights LLMs seem to reflect a culture.
I don't really enjoy the way Qwen writes prose, and I find its thinking a bit exhausting, though it clearly writes very good code.
I like the neutral, clear way the Gemma models write, which I sometimes use to get myself a "getting started" document on something I want to understand; it also summarises well. It is neutral, sensible, un-showy. It writes in a way that is fairly close to what I would use for documentation. The 12B and 26B models are also very good for talking about art and photography. Analysing my own photographic work has helped me more than I expected it to.
This model, honestly, has made me smile. It also feels like it is more creative at a given temperature than Gemma. I am trying to motivate myself to do something quite open-ended so I asked it about what other people's considerations might be in my situation, and at the risk of anthropomorphising, the things it has come up with feel like the work of a more curious mind, somehow. More eclectic. I have enjoyed testing it and I really want to test it more, which might help me get over a motivation hump there, too.
(I am also exploring its hard-wired policies by asking it to analyse some studio art nude work I have done; it definitely thinks out loud about its policies in a way I have not seen Gemma do.)
I think we’re going to see a lot more “product“ focus in the future with deliberate attention paid to these kind of properties. Historically though there are some obvious differences, the focus has been on benchmark maximizing. As that saturates, I expect more interesting choices about writing and thinking style designed to be differentiators instead of a side effect. Kudos to the PM here for taking it in a different directions, there’s obviously been thought put into it.
I ran a 9B over my like 100k photo library — it was very good at it. And extracting any text. All local.
It might be an indication that analysis of photography is something of an ideal discipline for an LLM since so much content online involves discussion of pictures.
A lot of what I am trying to do with my photography is sort of meta-photography. I am really interested in early photographic history, pictorialism and its opponents etc., but I try to avoid reproduction, so I try not to simulate processes too closely or to use vintage tropes in props and settings, but I use simple, undercorrected lenses and some vintage lenses, to gain some of the visual language.
Finding out that LLMs (including Gemma-4 12B with its built-in image encoder) understands a lot of my references and influences, could recommend me my (still semi-obscure) favourite historical photographer and other photographers who clearly engage in the same work, is amazing. And sometimes it says stuff I had not thought of, which is what I am looking for, since my photographic journey is somewhat lonely.
And that is just sort of brushing past the fact that these things can describe the contents of photographs with an accuracy that you can almost take for granted.
You really have to get the models to end their thinking. Almost any commercial model serving has safe guards like this to tune how much they think.
the “Actually… But wait!” style responses are so annoying, even Claude opus struggles with this so I’d be interested if meta has done something to cut down on that while still giving good responses
It's not perfect but it is very terse! Better than BottleCap managed to do with post-training Qwen in ThinkingCap.
I suspect it will help a lot with enabling preserve-reasoning, because the biggest apparent limitation of this model is the 128K context window.
Though the practical issue I am seeing on my M1 Max MBP is that performance suddenly drops off a cliff if I have DFlash enabled.
128k context window is a complete non-started for us. We need to optimize our most needy agentic jobs, but our average context is well above that
I was able to use rope/yarn scaling with llama.cpp to extend the context window to 256K and it seems to be pretty usable on the debugging-and-bugfixing session I have that’s 216K tokens deep. No infinite output loops, reasoning is still coherent, tool calls appear to be passing and failing at roughly the same rate as a fresh context window. Haven’t tried going to 512K or higher yet but 256K definitely seems usable to me on a single Radeon R9700.
I added these arguments to my llama-server call, using the official GGUF release from Meta’s account on HF.
`--rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 131072 --override-kv muse-glimmer.context_length=int:262144,dflash.context_length=int:262144`
Useful stuff, thank you. And more reading for me to do!
(DFlash seems rather weird on my M1 Max; model performance suddenly drops off a cliff like maybe there is some memory management issue).
In my experience I am getting 23-24 t/s output with dflash off, and it craters to ~9 t/s with it on, miss rate exceeding 50%. And I'm using the same device as stated on their model page/card. We might need to wait for the software to catch up
As an extra data point, I have exactly the same issue on a M1 Max as well.
The model card does say 131K+ but I have no idea what scope the + really has in practice. Feels like overclocking; you're-on-your-own territory?
One thing I will say is that its thinking traces are really quite terse. It genuinely seems to spend many fewer tokens on reasoning. So that might help a bit.
There's an inkling branch, go to unsloth, read - https://unsloth.ai/docs/models/inkling
> Muse Glimmer is a 30-billion-parameter model optimized for always-on local agent workflows. It’s small enough to run on a Mac or PC with a single consumer GPU, enabling use cases that range from local agents and function calling, to local coding, and LLM-as-a-judge evaluation.
The next iteration in LLM products is a 24/7 thinking loop where the claude-code like thing gets input continuously from your wearable, notifications, and newsfeeds and is constantly preparing things for you.
I've been building this for the last 6 months or so. I've basically got it working. The model is not the issue, the infra is. Keeping everything in context just isn't possible and LLMs, even Fable, don't mode switch well. To get around this I've built a database software that ingests as much digital information as possible, and annotates it, then creates timelines with resolution gradients (longer ago = less resolution) that it feeds to the LLM on every request.
Then you have your cheap little MoE or ternary model just running in a loop, with an escalation pathway before it reaches the big expensive models.
Currently it's doing things like reminding me to take allergy medication when I wake up because it's checked AQI or whatever, reminding me to stop at the market when I'm on my way to pick up the kids to get the cherry tomatoes I forgot, giving me heads up of what folks are expecting from me in certain meetings based on cross correlating email and calendar, etc.
It's honestly the single most productive tool I've found for my ADHD.
I’m curious why you don’t just use them like a Meeseeks box, rather than compressing and context stuffing into one. One only checks and categorizes your emails, another one for each category of email or even subcategory, one that only handles calendar additions, a different one to check it and notify you; you can go infinite with it. Hell, I’ll have one instance find a file and read it into the context of a different one because I don’t want a bunch of grep commands mucking up the context of the analysis. The find/read one exists for a few moments, as does the analysis one, and the ‘perform’ one is entirely different. I can run them all in parallel and use a queue if needed.
I’m sure you have reasons for your setup though, so I’m curious how you landed on it.
Think of an LLM as a thesaurus, but for entire trains of thought rather than words. Your initial query yields something pertinent to the task at hand. But let it endlessly recurse and... you end up with something completely useless.
People would do well to acquire at least a modest familiarity with what an LLM actually is. NLP is fascinating. So is entropy.
This is is already possible with Claude Code. I use a setup where I have one instance monitoring a local queue, I have a web app for receiving webhooks from various sources and pushing them to the queue. Plus email for things that don't have webhooks. That instance then decides what to do with each input, sometimes it can spawn additional agent to investigate/prepare, sometimes it creates a ticket assigned to me and then waits for me input. All of that just uses the monitoring tools built into CC. The dispatcher loop doesn't need to be extremely smart, so I might experiment replacing it with a local model like this.
If you are willing and not too busy, What model do you use and what is your cost? (If using subscription would you be able to check with 'npx ccusage').
I'd also be really curious about the cost to run something like this, and what things you think it's particularly helpful for?
I run this on a side of the Claude Pro subscription that I use for other purposes. My main motivation was root cause analysis of production issues. I have a solo project and unfortunately my mental state has been degrading over the last years. I would avoid looking at production issues, because I didn't have the energy to focus on the investigation. So I automated this, setup the loop, setup metrics/logs access for Claude to use and now whenever something goes bad, I have a single report that I can act on easily, and if I don't, it will ping me in a way that's not spammy like automated alerts. But I'm finding more uses for it.
Maybe it's my lack of imagination, but what do you imagine you'd be doing where you'd want to keep a computer busy overnight?
It seems like the purpose of humans isn't to keep machines busy. When our phone or laptop is idle, it's fine if it sleeps. And when we do want something, we'd rather not wait.
(Also, this new model seems to be designed to keep latency down, which is useful for interactive tasks.)
Help convince Firefox of this: https://news.ycombinator.com/item?id=46294238 Rather than develop its own AI, Firefox should develop a system to pipe your html rendered browsing history in real time so external local services can process it: https://connect.mozilla.org/t5/ideas/archive-your-browser-hi.... Firefox could be the only browser that does this.
The fact that you've been posting this idea into the void for 8 months with no pickup is already your answer
Couldn't this be implemented as a web extension? I imagine modifying singlefile to automatically send html to a local port is much easier than trying to convince a chronically mismanaged organization like mozilla (no offence to mozillians).
It is interesting but it does look like a careful distillation of (Spark and) biggers open-weight models.
The progress compared to Qwen3.6 27B is good, not that impressive, it's a 4 months old model. (kuto to them to compare to 27B dense and not 35B MoE, it's more fair to do so). It is very probable that Qwen3.8 27B will crush Glimmer-30B on most benchmarks.
Still great if they want to play in this space. Having competition for the 24-32GB VRAM target is only good for the end user.
Agreed, the trend in this consumer-accessible range is encouraging.
I'm so excited about these two new models. Qwen 3.6 27B has been my sweet spot so I cannot wait to try 3.8. Glimmer looks really strong, I'm encouraged that Meta compared it to 3.6 in the model card! Exciting times!
Optimizing models specifically for always-on local agent workflows is the right primitive for decentralized systems. Brilliant release.
Tried it (the full version, using 120 GB RAM), wasn't impressed, gave it some defective code, and asked it to fix all errors. It kept looping around and around and digging itself deeper and deeper into a rabbit hole; eventually, it got into a "reasoning" discussion about whether a custom compiler was used that supported the wrong syntax...
What I think would be perfect is a model that could run on a single DGX spark and be competitive with DSV4 Flash 731. Flash is already a game changer. Hopefully meta plans on this, like the old 70b. V4 flash is smart enough for any use but slightly too big. 27b-30b isn’t intelligent enough.
DSV4 Flash 0731 already runs on RTX 4090 24GB + 128GB system RAM at a usable tok/s and quantization.
You personally? Just curious. Context window is also a factor and ram isn’t really cheap. Sparks are assembled units which I like.
For the same price as a DGX Spark here (A$8499) I can buy roughly 544GB of DDR5-5200MHz from retail; which on a quad channel platform would deliver ~160gb/s real world; and ~320gb/s with octa channels (Xeon, Threadripper Pro).
If you can afford it or somehow find a used unit, you can go Epyc for 12 channels.
8/12 channel DDR5 will beat DGX Spark in inference/decode even without a GPU of any kind, as it’s memory bandwidth bound, and the Spark tops out at ~240gb/s real world.
With some optimisation and maths, it’s entirely plausible to ach
You are paying an extraordinary amount of money for the convenience of a super small unit, with still mediocre software support, but at least a community. Expect to be crawling through forum posts regularly, as SM121/Spark has many quirks and ecosystem issues still.
Please don’t pay another 70-80% gross margins on top of already inflated DRAM prices unless you need. The Spark IS really nice if you want to test out ConnectX or if you really need something small and compact and quiet.
Also consider: used Adas or even Ampere NVIDIA workstation GPUs can come with a lot of VRAM and be “reasonable”, with CUDA.
Been investigating these multichannel AMD based platforms last year and seem like none of them can in real scenarios utilize anywhere close to their theoretical bandwidth.
I've run it with a large context window on 256GB ram + 4090. It wasn't super fast, but it was manageable and it completed the tasks I gave it well.
How many tps and size of ctx?
This model I think will be too slow for that on Spark, even at 4 bit quant.
It's a dense model, not MoE like e.g. Qwen 35b or Gemma 4 26B A4B. On a Spark it will be memory bandwidth limited
I haven't tried yet (working on it) but back of the napkin estimate puts it at around 15tok/s even after converting to NVFP4. Prefill would be much higher though. That 15tok/sec is pretty typical for dense models of this size:
NVFP4 Q/K/V/O and MLP projections: ~13 GB/token
BF16 attention gates: ~3 GB/token
BF16 LM head: ~2.5 GB/token
Total: ~18.9 GB/token
At 273 GB/s, that gives a bandwidth-only ceiling of about 14.5 tok/s; actual performance would be lower.
Native dflash support on day 1 helps a lot! High quality speculative decoding speeds up a lot of agentic work.
You're right. I'm getting ~33tok/sec w/ dflash on it, even bursts up to 60tok/sec, using my personal home-built-for-Spark inference engine (not vLLM or llama.cpp based)
That's pretty respectable.
Still working on optimizing and cleaning up before I push it.
https://github.com/rdaum/eider/ fwiw
Still needs 32-64GB memory to run it locally. 64GB Macbook pro with an M5 chip costs more than 4k Euros in Germany. A more practical model would be a language specific (e.g Python or JVM language) and excellent at tool calling and reasoning. Maybe that way they can shrink it even more.
Practically ~20GB with KV cache
> We quantize weights to ~4-bit, bringing the LM under 20 GB. We validated minimal to no degradation on agentic tasks under compression.
https://www.reddit.com/r/LocalLLaMA/comments/1vkgsum/introdu...
I think if there's going to be advantages to making smaller, more targeted models, those advantages will probably come from targeting specific domains, not from targeting specific languages.
I think that if an LLM can't abstract over the differences between Python and C++, it probably will have an even harder time abstracting over the differences between writing code that manages a webserver, and writing code that does aerodynamic simulations.
I am running it on a single RTX 3090 (24GB VRAM).
Some folks on Reddit are having the same experience: https://www.reddit.com/r/LocalLLaMA/comments/1vkm42m/muse_gl...
It uses an order of magnitude less VRAM at longer contexts which is a huge advantage over Qwen 3.6 27B
Seems like that's the tradeoff with this model. Close to 27b intelligence while using less vram.
Coming from the PC games industry in the 90s and early 2000s, it was a struggle to run some of the games on release. 90%* of people wouldn't be able to play the AAA games on release (think Crysis, etc). This period of local LLMs reminds me of that time, whereby the hardware just isn't there yet. Give it time, and the prices will drop.
* total guess
Assuming we can even get the hardware in the first place, it might not even be possible for consumers to buy it at any price if it sells out through "agreements" made years in advance https://news.ycombinator.com/item?id=47045459
I'm sooo happy I pulled the trigger on upgrading and getting a new laptop (with 64 GB RAM) last summer. Feels like it was just in time before the exponential price jumps.
I kick myself a couple times a week for not getting the 512GB Mac Studio in February. I was holding out for an M4 or M5 chip...
I was about a week away from buying a very tricked out MacBook Pro with 128 GB RAM, but was on vacation and worried about it arriving while I was away, and then the price hikes went into effect. Grumble. Oh, well. Serves me right.
https://www.ign.com/articles/ramageddon-continues-another-ye...
The best time to plant a tree…
Lol, I still think about buying that now, even after the price hike. FOMO.
I’m waiting for the bubble to pop. I suspect we’re 12-18 months away. We’re at the point where manufacturers are going out of business because the tech market is contracting so much. That’s not sustainable.
Bought an M1 64 GB for 2000 euro’s second hand a year ago. That was sweet
paid 2.7k € for this same build new in Dec 2023, that was also sweet (still is)
Pulled the trigger?
lol, you're right, the brainfart completely changes the meaning.
I corrected it.
Common phrase.
That commenter you're replying to knows that. The original commenter before them wrote "pulled the plug" which is different and doesn't quite apply here (actually implies the opposite of what they meant to say).
Ah, the edit was fast enough not to leave a mark, leaving a conversation fragment that made me think OP was being weird about a gun related phrase or something
Parent used "pulled the plug", are you saying it's applicable here and not "pulled the trigger" like suggested?
There have been discussions on language specific not really being a relevant change to reduce size.
I would love to see any good research projects about it but i have the feeling that Frontier with MoE is making too fast of a progress so that a customized model would always be worse and that the MoE part is actually going somehow in this direction.
On the other hand, at the GTC was a talk about coding in different lanugage (like spanish) and explaining that the quality between spanish and english is relevant different.
But i have not found a good article about the impact of learning data with practical experiments or even if the order of the learning data matters.
At least I think i remember that Meta mentioned having better and less data can be better than more data with lower quality.
As long as these models can explain to you facts about any other topics, its still overfitted for the task though.
Capability in LLM's is distributed throughout the manifold in subspaces. Even worse, the subspaces exist in superposition.
That is to say, there is no single 'python' part of the model. The python bit is spread throughout the entire model and overlaps with other pieces that have similar, but unrelated, capabilities. For example the python subpspace might be partially in superposition with cupcake recipes, Esperanto, and calculus. We need calculus in a coding agent but not the other two. However, separating them cleanly is almost impossible, and even identifying them is tough.
Internally the manifolds are highly inefficient and nothing like you would imagine something humans built would be designed. It's more like something that evolved in nature.
My current image from a MoE is that the base/core might be the more generic thing and that things like python are part of one expert though.
With MOE you train a router designed to select which parts to activate. The router itself is a trained neural network and the 'experts' are usually not really things like 'python'. They're just the functional subspaces I described above.
Again, those subspaces are all somehow inextricably correlated and live in complex superposition spread throughout the manifold. The router doesn't know (or care) WHY those sections get lit up it just learns which ones to activate to optimize it's own reward function. So maybe it learns to activate "logic", "python" and "cupcake recipes in esperanto" whenever it see's something that kind of looks like python. It's not the best answer, it's just the best answer the tiny router could figure out.
It's all wildly complicated and inefficient, and works nothing like any reasonable human would imagine that it SHOULD operate.
I mean the only way to separate them would be to separate them at training would it not? If there is zero python in the training set, then you have a smaller training set, should be able to have fewer weights in the model
There was some paper about routing at training bio-knowledge into a particular region of the model, which you then can cutoff when serving. But you probably lose some efficiency since maybe you sized that region too small/too big.
That's a very clever approach. Any idea about the papers title or authors? I'd love to look it up.
https://arxiv.org/abs/2410.04332
https://alignment.anthropic.com/2025/selective-gradient-mask...
Well if you're spending thousands on API tokens already, you could just drop the same amount on a 128GB MacBook Pro and that's a one time cost.
If you're dropping thousands on API tokens, you're going to be slowed down at least 10x trying to do everything on a single MBP.
But you could grab a 5090, and paired with some DRAM for MoE offloading of bigger models, and be a happy camper with 1.8TB/s of memory bandwidth.
Or just use Luna honestly. Worth considering if you’re ok with hosted APIs.
Don't forget about energy usage, you'll probably never break even vs same model on openrouter.
If you can’t do it cheaper on your own hardware it does make you wonder how much of the cost of inference those large LLM providers are eating? Datacenter hardware isn’t magic.
Your personal hardware probably isn't running useful tasks 24/7. If you spend 60% of your 8h work day on full on agentic work, then your hardware is paying off for itself only 20% of available time.
Datacenter hardware can batch at large scale, probably over 90% more energy efficient per token than a MacBook.
Datacenter hardware might as well be magic compared to consumer. "Oh the F35 isn't magic compared to my M16 bro!"
The models people are spending thousands on require more on the range of 600-800gb memory.
128gb hardly runs deepseek v4 flash which is almost free via api pricing.
I feel like we’ve had this discussion before. From what I remember, specialized models rarely do that much better than general ones, hence no mode Codex models.
There is no good reason to believe language-specific models are going to be any meaningfully smaller, just worse. Same as English-only models vs those trained on a multilingual corpus.
> 64GB Macbook pro with an M5 chip costs more than 4k Euros in Germany
Sure, if you want the latest and almost* greatest. You can pick up an M1 Max 64GB for ~1k.
* I guess 128GB also exists
The machines that can run this are pricey, but not beyond a high end developer machine.
4K bucks buys you around 180 months of <insert AI subscription here> with zero upfront cost.
Problem is that might go away or get nerfed.
then you switch provider, it's not a monopoly
If that happens you can still buy hardware later with almost certainly more (tok/s)/$ and better capabilities to run newer models more efficiently (remember native MXFP4?). Right now basically every generation of accelerator is adding new capabilities. These aren't yearly DirectX 9.0c-compatible GPU performance bumps.
As an individual, for average privacy needs (e.g. open source or at-home coding and automation), it's pretty much complete nonsense financially to self-host LLMs currently or select hardware now based on the capability to do so, and pay thousands of bucks extra.
If you don't mind exfiltrating all your IP to the API provider
Haha wow. I’m trying to even imagine the AI landscape in 15 years and I can’t.
Instead of saying "I have a MBP with 64gb of RAM" you'll hear people say: "I'm subscribed to Model 9.x11B" and others will comment: "Oh dang, that's a nice model!"
Even if you had a 64GB machine: Are you willing to reserve 90% of your memory to run a LLM? With dirt cheap models like deepseek-v4-flash that will run "forever" on $10, the answer for me is clearly: no.
I'm waiting for the speed/quality per dollar metric to go down a little bit further and then I will def run it at home.
Its not just that you send a sentence to an API endpoint, you always send EVERYTHING to that agent as a context.
You want to analyse your spending history? You now send everything to someone.
Either no one cares but understands this implication on how easy it is to really capture you or no one really things about it.
But i'm a lot more diligent on what I send. I disabled the gemini activity feature for example because google started telling me that my stuff could be reviwed by humans.
Yeah, it does feel a bit silly with my encrypted disks, encrypted backups, unique passwords, advanced router, etc, while I send everything I do in plain text to anthropic.
Similiarly I wonder why we dont run our own email server despite the sensitive data there.
I did, it started to become too much work to run it well due to all the spam :| (even with the right signatures and configs, until you learn what a blacklisted ip is and that ips need some time of 'positive history' and what not.....)
But at least with your email, you had to trust only one company, as shitty as it is.
Separation of concerns was also easy.
Now with OpenRouter, you just might by accident, send your whole context to just everyone because OpenRouter just routes to different models and you might just switch around between some free model, the good one etc. And it is always the whole context.
"With dirt cheap models like deepseek-v4-flash that will run "forever" on $10, the answer for me is clearly: no."
When it's free, you are the product.
Deepseek flash is open weight, this means we can download and run that model without any connection to deepseek, no data/tokens/usage data ever reaches them. They cannot make us their product.
All those random api providers are absolutely scooping up your data though. And the hardware to run it locally is absurdly expensive.
Running Ds flash at acceptable speeds is challenging unless you have several thousands of dollars to invest
I see many people saying deepseek and other chinese providers have always been profitable. Also they show their training costs publicly. Can't say for sure since I have not used it personally, but I think they'll for sure outlive the western SOTAs.
OpenAI apparently runs a profitable inference business with 40% gross margin, but their advertising budget is nutso and their real costs are pretraining and research. I suspect Deepseek's comp is not predicated on capturing the lightcone of all future value, some googling insinuates their top pay is $212K US which would support that suspicion. Compare and contrast with the $1.35M and up at OpenAI.
Ah yes, I'm sure Trovalds and Stallman are harvesting my data through free software, aren't they? This argument is used by boomers who were fed cold war era propoganda that surely everybody is selfish, and you're always at fault.
Think they're talking about things that are free as in beer but not free as in freedom, not FOSS
It's the size of a big vm. There's nothing wrong with reserving that much working space for one item.
I don't understand the desire to run own AI models for programming locally. No laptop is ever going to be as powerful and energy efficient to run anything close to OpenAI, Anthropic or Google models. A model you can run on a loptop is simply not going to work as well as it's needed for programming. Small models for linguistic work fine, but anything more sophisticated simply won't provide enough resources or power. Or models would need to be significantly dumbed down - then why use them at all? So far the idea of carrying a "thin" or "thin"-like device looks more reasonable to me, while running AI on your own server.
> A model you can run on a loptop is simply not going to work as well as it's needed for programming
The models you can run on a high-spec laptop today are approximately where frontier models were 12-18mo ago (albeit at a lower tok/s rate). If you scan back through hn comments from that era, you’ll find plenty of people saying “this is powerful enough to massively increase my productivity”.
> albeit at a lower tok/s rate
Not always! I get 80-100 tok/s from Qwen 3.6 35B-A3B on a MacBook Pro thanks to MTP. With long contexts that dips to around 50-60. However, prefill is much slower than API models. So it becomes really, really, really critical to not have cache misses.
Slower is meaningfully dumber when you’re time bounded and need all the inference time compute you can get.
I’m quite optimistic about the long-term future of local LLMs for privacy and cost control reasons. An LLM running on my own hardware, even if it’s not a laptop but a home server, is one where I don’t need to worry about token limits, token fees, privacy, and “rug-pulling” from the vendor.
In the short term, the big challenge is being able to afford hardware that can run a ~30B model. Last month I got to experiment with LLMs on a NVIDIA RTX 6000 Ada Generation as a visiting researcher during my summer break. I see the power of local LLMs for agentic coding; they’re no Claude, but they are quite useful. I wish I had gotten into local LLMs before hardware has gotten prohibitively expensive and in some cases unavailable; Apple discontinued certain Mac Minis and Mac Studios with high amounts of RAM due to the RAM shortage.
Hopefully high RAM prices don’t become a new normal, though the next year or two doesn’t look good.
I've never done it but would be interested because it cuts out the burden of worrying about costs. Maybe I'm mistaken on energy cost here. There's a constant raincloud that follows me around regarding limits, and it would be nice to shake that.
I've been able to accomplish incredible feats (for myself) since GPT-4, so model intelligence is secondary.
For some companies there might be a need to run them locally. For instance, Apple decided to run LLMs on the phone locally. I guess it depends on how important latency and privacy are. Perhaps Meta is looking at how much interest for those local models is there.
> I don't understand the desire to run own AI models for programming locally.
Privacy. Security. Not bulk uploading your trade secrets and intellectual property to Sam and Dario’s servers.
I was suggesting running personal AI servers.
Yet.
I lament the comments saying this in any way redeems Meta (the company).
The researchers releasing this stuff have almost nothing to do with Meta other than being bankrolled by the slaughterhouse.
You aren't the customer, you are the pawn in big tech's game of thrones. Your good will is a commodity to be traded, almost literally. It will be used against you the moment it's convenient. This is open weights because Meta couldn't monetize it in any other way than to cloud developer's judgement of their reputation.
But I guess most people just don't care.
I'm glad it's open. It does not make me think any better of Meta.
It’s rather amusing to me to read comments like this, and then simultaneously whenever a Chinese company or team releases open-weight models or whatever there is a giant round of applause, America is so behind, and there’s nothing but positive things to say about the intelligent, creative, and well-intentioned Chinese engineers (which is true, America certainly doesn’t have a monopoly on great people). Don’t you know? Only China can release good, open weight models and American companies can’t compete. Oh by the way all the spend is for nothing because China alone can release open-weight models thus destroying American AI.
When an American company does anything? Doom. And. Gloom. The engineers? Taken to the slaughterhouse! America? Behind! The public? Bamboozeled!
> This is open weights because Meta couldn't monetize it in any other way than to cloud developer's judgement of their reputation.
I’ve been told over and over this doesn’t matter. Just needs to be cheap and open. Or maybe that’s only when Chyna is involved?
Sorry this post is a bit snarky but it really is something to behold. And certainly I don’t know the OP’s opinions on Chinese open weight models. Perhaps they agree with me.
While composing a reply to a comment throwing tons of shade on American AI, I took some time to check out the commenter’s HN profile. Their comment history was about 50% such comments. Their submission history started with an article about how Russia was unfairly blamed for some hacking campaign.
It’s entirely possible that this is not a foreign influence campaign. Perhaps there’s a group here that is simply anti-American as its primary interest, and passionately so to upvote each other.
On the other hand, one should not discount the value of HN as tastemaker and trendsetter. Also, it would be fairly easy to run bots here. I wouldn’t be surprised if HN were a field of combat for many parallel influence campaigns, foreign and domestic.
> On the other hand, one should not discount the value of HN as tastemaker and trendsetter.
I would encourage dedicated readers here to aggressively and persistently discount the value of HN as a tastemaker and trendsetter.
HN is actually a trailing indicator on tastes and trends, essentially by design. Things only make it to the front page if they get submitted and voted upward by a large number of people. That means it’s all stuff that is public and seems cool to a bunch of random people browsing a website. Not exactly cutting edge.
Basically every community, online or not, over-indexes on its own importance. Like how random small towns can get so worried about international terrorists targeting them. To folks who love HN, of course the great forces of the world would be attracted to compete here. Look how obviously awesome it is, right?
Paul Graham said HN was set up on the theory that it would efficiently surface great ideas and entrepreneurs for Ycombinator. Needless to say, that did not pan out. (How often do you see PG or any YC leadership here anymore?) Honestly I am mystified as to why YC continues to operate it at all.
I think you're wrong. To gut-check, I asked a panel of AI models "What platfroms should be considered tastemaker or trendsetter for adopting new AI models". Here's the part of the answer relevant to this discussion:
> AI X/Twitter — a few hundred accounts effectively set the narrative in the first 24 hours; vibe checks here outrun benchmarks.
> r/LocalLLaMA — the open-weights kingmaker; a model that fails here doesn't get quantized, and unquantized means unadopted.
> Hacker News, and increasingly YouTube/Discord for the practitioner layer.
Source: https://pellmell.ai/s/aefaa217b57ed50be9e2a4b8c9f3173e
The internet will always be a trailing source of these things. If you have to ask the internet, you're behind. The internet is a form of information exchange; information about the thing needs to exist before it is exchanged.
For a brief moment the readership of HN heavily overlapped with founders in Silicon Valley, and for those years HN was indeed a platform that acted as a trendsetter or tastemaker. That time has long passed. Twitter is the closest to this today, which is on that that list, and makes it a decent place to listen. r/LocaLLaMa is not bad as a support forum for GGUF and quantizer projects but it's at the level of "Windows tutorial" of software from the '90s.
If anyone is interested in being at the "edge" of this, I suggest simply going to meetups in tech hubs where people are working with AI and models. My guess is any city with a major tech presence will have more "edge" than HN. I'm fortunate enough to be in Silicon Valley right now and have friends who work at foundational lab companies so it's not hard to stay on top of what's happening. The "internet" of 2026 is just much, much bigger than the internet of 2007 when HN was founded, and so it's just a lot harder to find the information that you need.
So - presumably you have another site in mind that is better. I'd be intrigued to know which one you would recommend.
I don’t. It happens in-person or in private group chats these days.
Sounds like something a bot would say!
Which reply? Because both seem like nothing a bot would say honestly.
You gotta touch grass and talk to people irl
I asked my barista what they think about Qwen 3.8 Max…
Would be nice if there were a hn feature, userscript, or plugin to just filter comments from new accounts. Bonus if there was some sentiment analysis or llm-based analysis to filter out unsubstantiated inflammatory comments too.
The account we're discussing is from 2014. How much earlier do you want?
Old accounts can suffer takeovers too!
There's literally a Chinese-funded influence campaign against American AI: https://openai.com/index/disrupting-malicious-uses-of-ai-dat...
As if US labs don't lobby politicians to get regulatory capture.
It's much easier to buy politicians in US, e.g. Elon bought POTUS for ~200M, Anthropic revenue is 40B, imagine what they can do
lol you believe this bullshit? "likely PRC-origin cluster" what an awesome amount of proof corporations need to convince the gullible.
Why not "OpenAI used simplified chinese to create a fake prc-origin campaign and media buzz to convince the public they actually love data-centers and anti-datacenter sentiment is a psyop" there's an equal amount of proof provided for either scenario.
... and I think it's a big mistake to say, as some people are, that people are opposed to data centers in their backyard because of a Chinese influence campaign. There is also reason to think that some "foreign agents" are doing it for the money
https://www.theguardian.com/media/2026/aug/10/indonesia-clic...
https://humsci.stanford.edu/feature/political-post-real-or-m...
> Perhaps there’s a group here that is simply anti-American as its primary interest, and passionately so to upvote each other.
Considering this site has international users, it's not hard to find this, especially right now. Hell, Canada is in this group currently.
I'm a 52 year old natural born US citizen whose ancestors have been here for generations and I'm currently anti-American. Why wouldn't I be?
We've never been the shining beacon of light we would claim to be, but we're so fucking awful now.
When you say you are anti-American, what do you mean by that? Are you, for example, wishing for the demise of the United States? Do you want to tear down the 1st Amendment and the Statue of Liberty? Are you against democracy? Do you want our businesses and factories to shut down and go out of business? Are you willing or would you support foreign countries attacking our military at home and abroad? Are you cheering against our athletes?
Could you expand on what you mean by being "currently anti-American"?
I can't speak to the parent commenter's precise meaning, but I interpreted them to mean they oppose the current domestic and foreign policy of US political and business leadership (a distinction that blurs to the point of indistinguishability of late, courtesy of said domestic policy).
I believe that many Americans that were previously dismissive or ambivalent regarding critiques of US activity at home and abroad (either due to patriotism, realpolitik apologia, or general naïveté) are now re-evaluating some of the beliefs they hold about their country in light of the chronic political dysfunction and an absolutely breathtaking extent of corruption being perpetrated in broad daylight today (as well as indications that extensive corruption has long festered among our elite class, surfaced via the Epstein revelations).
Yea but if they just oppose the current administration isn't that just being anti-Trump instead of anti-American? His administration is just some other administration that'll come and go like foul sewage gas. We've had 47 of them. Does the OP become pro-America when their party is in charge or when certain conditions are met?
I don't know, just asking.
But I know that in my own upbringing I've always viewed America as a place where, because it's a democracy, we are never going to all be aligned or on the same page about policy direction. Even when I disagree with the Biden Administration or the Trump Administration on some number of issues, and believe my I've got a lot of those issues with the current one, I wouldn't think of myself as anti-American or wishing harm to the country. But that's just my own experience.
Why would you tear down a gift from France?
Well I certainly wouldn't, but it's a cultural symbol of America and so when you think anti-American I'd think things like tearing down the symbols that represent our nation would probably be in play. Maybe not, idk. That's why I mentioned it and asked.
> Are you, for example, wishing for the demise of the United States? Do you want to tear down the 1st Amendment and the Statue of Liberty? Are you against democracy? Do you want our businesses and factories to shut down and go out of business? Are you willing or would you support foreign countries attacking our military at home and abroad? Are you cheering against our athletes?
I am very pro the "dream" of America, in terms of liberty, democracy, etc. According to every "democracy index" I'm aware of we're not doing so hot in that regard, generally rating as a flawed/deficient democracy and the trends are going in the wrong direction, fast.
> Do you want our businesses and factories to shut down and go out of business?
Generally, no, but this is way too open-ended of a question. I want good economic opportunity for everyone, including every US citizen. But relevant to the OP if Meta got snapped out of existence I think it would be a net positive for the world.
> Are you willing or would you support foreign countries attacking our military at home and abroad?
Nope. But across our entire history ask yourself how many foreign countries have attacked the US? Now ask yourself how many the US has attacked. With those numbers in mind, does the US seem like "the good guys"? really? ...really?
> Are you cheering against our athletes?
Nope, but I'm not cheering for them either just because they are American, I'm not a tribalist.
> I am very pro the "dream" of America, in terms of liberty, democracy, etc. According to every "democracy index" I'm aware of we're not doing so hot in that regard, generally rating as a flawed/deficient democracy and the trends are going in the wrong direction, fast.
Well, to be fair people do have different dreams. I'm not sure those indices count for a whole lot. As an example, folks who argue in favor of returning more power to the individual states are, certainly, acting in accordance with stronger democratic principles. I'd argue the EU is actually a bit anti-democratic as it removes more power from local populations and individual states/countries/entities. Yet how would individual democracy indices rate these two?
And if you disagree with my perspective on what is more democratic, well, who is right and who is wrong?
> Generally, no, but this is way too open-ended of a question. I want good economic opportunity for everyone, including every US citizen. But relevant to the OP if Meta got snapped out of existence I think it would be a net positive for the world.
I'm no fan of Meta. But they aren't the only entity where if they snapped out of existence it would be a net positive for the world. I can think of a few non-corporate entities at least.
But fair enough it's a bit open-ended. I guess if push comes to shove when you say you are anti-American do you want to see, for example, economic opportunity increase in other countries at the expense of Americans? Not all relations have to be such give-and-take, but certainly many economic opportunities are. I'm just trying to learn more about what being an anti-american means to you.
> Nope. But across our entire history ask yourself how many foreign countries have attacked the US? Now ask yourself how many the US has attacked. With those numbers in mind, does the US seem like "the good guys"? really? ...really?
I'm fine doing things like removing all of our overseas bases, leaving alliances including NATO, never again being involved in the affairs of other countries, whether that's Korea (Korean War), Kuwait, Bosnia, or Iran, or Taiwan and China, and cutting our military spending quite a bit to reflect your desire to not be the bad guys anymore. I say let's just sit back and let the rest of the world figure out their problems - why do we need to be involved?
Is that something you agree with as well? Would that be anti-American in alignment with your views or is there other nuance here I'm not properly capturing?
> Nope, but I'm not cheering for them either just because they are American, I'm not a tribalist.
And to be clear even if America was even, say, your ideal dream since you're not a tribalist you wouldn't cheer on American athletes and would also label (not as a matter of insult but as fact) other people who cheer on their compatriots as tribalist?
To answer most of the questions above simultaneously (to some degree):
I only care about the content of someone's character and how their actions impact the rest of the world, not the country they happen to be a citizen of, a fact which for non-immigrants (the vast majority of people) is completely random happenstance.
If already wealthy foreign agents become wealthier at the expense of impoverished Americans, I find that to be unfortunate. If already wealthy Americans become wealthier at the expense of impoverished non-Americans, I find that to be just as unfortunate.
I'm not going to cheer someone (athlete, business leader, or otherwise) on just because they are American. The fact that they are American is as irrelevant to me as the color of their eyes.
Do they seem to be a good person who treats others well? If yes, I will cheer for that person, whether they are American or not. Do they seem like an entitled asshole that treats others poorly? If yes, I will cheer against that person, whether they are American or not.
And the reason I currently label myself anti-America is that I believe that collectively we are the entitled asshole that treats others poorly. And we can't just pawn that off on Trump to be the scapegoat. He didn't materialize out of nowhere. We elected him. Twice. The second time after a failed insurrection. We have collective culpability.
it is strange that when people are pro-american, or pro-any-ingroup, nobody asks for justification; OTOH, make a mistake of speaking from rational viewpoint based on historical events, expect a lot of hate from reddit/HN/any-ingroup-forum.
I applaud your openness to speak. I've traveled/interacted with many nationalities, and very rarely I come across someone who is open/rational enough to openly state their dislike of their country and denouncing their history.
I wish you the best in life, smart internet stranger!
> is strange that when people are pro-american, or pro-any-ingroup, nobody asks for justification
Being pro a group can be strictly positive sum - wanting to lift that group up, likely because you consider yourself part of it or on the same team. It's possible your intentions are bad, but they certainly don't have to be.
Being anti a specific group is inherently negative. Perhaps they deserve it, but that requires justification in a way simply being positive does not.
Good points, I personally believe that if/when China takes the lead, they will immediately stop releasing model weights. It only makes sense as a strategy to counterbalance (current) American labs' monopoly on frontier models.
Holding both those positions would be hypocritical all right, but are you sure it's the same people commenting/voting in both cases? I don't think there's a strong consensus on Hacker News. Even something like the time of day an article is posted might get different engagement depending on who is active in which time zones.
> I don't think there's a strong consensus on Hacker News. Even something like the time of day an article is posted might get different engagement depending on who is active in which time zones.
Based on my own experience and reading, I do think there's a general consensus on this site but I could certainly be wrong about that. I'm less concerned about hypocrisy per se, it's more that the arguments that are used, even if by a minority, seem to apply in only circumstances in which China releases open-weight models.
I am aligned with your viewpoint as well. And I've repeatedly argued it. If China were to take the lead the US can then just release open-weight models. Folks say having the lead doesn't matter because China releases cheaper open-weight models. We can just let them take the lead and then do it back to them.
> It's common, if not inevitable, for people who feel strongly about $topic to conclude that the system (or the community, or the mods, etc.) are biased against their side. One is far more likely to notice whatever data points that one dislikes because they go against one's view and overweight those relative to others. This is probably the single most reliable phenomenon on this site. Keep in mind that the people with the opposite view to yours are just as convinced that there's bias, but they're sure that it's against their side and in favor of yours. -dang [1]
The problem isn’t that people on HN have a bias, I feel it’s pretty balanced. The problem is that when there are any sides, they spend the top 100 comments rehashing the same arguments, often over a political bugbear or web design faux pas.
That pattern became a lot more obvious when there are five new front page AI posts a day.
[1] https://news.ycombinator.com/item?id=42205856
> The problem is that when there are any sides, they spend the top 100 comments rehashing the same arguments, often over a political bugbear or web design faux pas.
It's not even about sides, if for the last few hundred days you read a few AI related threads a day, then you notice that almost all arguments are rehashed, literally it's the same thing repeated using different words for 80%+ of comments on almost every AI thread. I started skipping most of it because there is genuinely nothing new or interesting added to these discussions.
> The problem is that when there are any sides, they spend the top 100 comments rehashing the same arguments, often over a political bugbear or web design faux pas.
This is a problem with any upvote/downvote based site, in my experience. It only takes a couple people who are highly engaged and who have a lot of free time to refresh the comment section and downvote everyone who disagrees with them.
Some times I’ll write a polite and well-sourced comment correcting some misinformation here and the comment will go to -2 or -3 when I check back in 10 minutes. Information that goes against the angry narrative du jour is often not welcome. Later, as calmer heads read the article and peruse the comments the downvotes start to get balanced out and the comment might rise, but some times the first wave downvoters are aggressive enough to get the comment downvoted into gray text before it has a chance to be seen.
the downvoting rings ( or bots ) are very active here. I think they're mostly bots, but who knows i guess
I don't think they're bots and I don't even think they're organized.
In my experience with other communities, it only takes 3-5 people with a lot of free time to crush conversations they don't like. On a fast moving comment section a wrongthink comment can be buried into the bottom half of the comments with only a few early downvotes.
When I used to run a spam factory my impression was that success on Reddit was all about avoiding the downvote brigades that would nuke you before you got your first 5 upvotes. Write a post on proggit where you confess that you are partially in it for the money or have any interest in the business of software and hoo boy, that's why I migrated to HN.
I still think bots run the /new and what gets to the first page shrugs
I'm certainly pro-America and anti-communist/fascist as a bias, but I don't really care about whether AI tools are open-weight or not. I just use the products that best fit my needs. I just think the arguments put forth regarding China and open-weight models and strategy are not very good. It just so happens that China is the only real competitor in the AI space and so they are who get talked about the most in comparison to the United States.
This comment isn't applicable to me, and if you believed that it applied, you'd have to add it to the OP as well since they feel strongly about Meta[1], they notice data points about Meta's behavior, and they overweight their bias against Meta[1] relative to others. Same with China "leading" and open-source/open-weight models and any time someone says China's strategy is better.
You can repeat this for any online argument or any topic.
It's not that Dang is wrong, however. It's that posting it in response to my comment(s) alone is hypocritical and pointless. Whereas Dang who is more responsible for the entire community is right to speak about it more generally. The message matters but so does the messenger, in this case.
[1] I don't use any Meta products (I don't even click on links), think social media should probably be outright banned, and Meta very likely should have been sued into the ground for the effects that their platform seems to have not just on children and young adults but also on our political system.
I see it as a strategy to increase capital costs for American frontier labs. By eroding the expected ROI of frontier labs, you deter private investment into them, which slows OpenAI and Anthropic in particular, giving space for the laggards to catch up.
> I don't think there's a strong consensus on Hacker News.
There are diverse viewpoints. However there are some topics and threads where it becomes obvious that the comments are going to tilt toward one viewpoint. Participating in those threads with a different opinion will get your comments downvoted to -2 within minutes even if it’s well-written and factually sound.
After this happens a couple times you learn not to engage with those threads because it only takes a few zealous downvoters to bury anything you write. So the illusion of consensus persists.
Concrete example: There was that fake (AI hallucinated) report that Meta spent $2B lobbying on something that was popular here months ago. I actually read the repo and report and noticed the AI hallucination, as well as pointed out that $2B in lobbying spend by a single company was not plausible or supported by any evidence. It didn’t matter how I wrote it, it would risk getting downvotes and angry replies about “How dare you defend Meta!” Some people are here for the anger and to feel revenge against the enemies they think they know (like the US) and will cheer on anything that goes against those enemies, regardless of the other facts surrounding it. Factually accuracy often takes a back seat to pushing agendas.
Free models help move robots; a complement. I can foresee the models staying free.
Maybe you're right for smaller models, but for frontier models this is the tail wagging the dog. If China has exclusive frontier model capability in the future, that's an enormous commercial and geopolitical lever. They won't throw that away to sell a few more robots. Anything beyond their competitors' capabilities will remain closed.
You imagine robots are going to play second fiddle to models, but they could become integral parts of model training with all the data they collect. Training could even become distributed. This will enable them to be responsive to local needs.
The robot market would have to be pretty damn valuable, to be worth throwing away a lead in the model market. Say, 10-100x as valuable.
Of the two competing models Meta compare Glimmer to in the post, one is Google's Gemma 4.
At this size open weight model, a Western company was already state of the art, Meta is joining that competition.
And my memory is that Gemma 4 got little criticism or doom/gloom. And no, it isn't Chinese.
Likewise the Inkling open weights announcement, Thinking Machines model, was also not criticised.
The comment about Meta is because of particular dislike of Meta, because of their business model, and how harmful they've ultimately turned out to be for the world - disproportionately so relative to their benefits to the world, compared to other big tech companies.
> disproportionately so relative to their benefits to the world, compared to other big tech companies.
This is certainly what many people around here appear to believe, but there are lots and lots of people who get much more value out of Meta's products than those of any other tech company. Whatsapp alone is probably the most useful tech product for many, many people.
That being said, FB/Meta have done a bunch of awful stuff, but to say that they're worse than Google/Amazon/Microsoft is not necessarily obvious.
The GP is claiming Meta is an awful company for what they have done and how they continue to treat their employees. That’s a perfectly ok opinion to hold, and many seem to agree.
Have DeepSeek, Moonshot, or the other Chinese AI companies done such things that attract moral outrage?
If they have, would you ever know about it? CCP controls the media, the companies, and the people. What's your evidence that Chinese employers are treating their employees better other than there's silence?
Chinese tech companies make their employees work 16 hour days, 6 days per week, compared to them Meta are absolutely saints.
The Chinese government owns half of every Chinese don't they? They've done some pretty horrific things.
Nobody wants to live in a world where one party dominates due to access to superior AI and the others have to fear it (well, except for a few psycopaths who would gamble on being in control of that party). So the underdog will always be the good guy in this race. It has been framed as a race between countries, so Meta fails to be the underdog because they're in the wrong country. That's all.
I don’t think Meta is bad for releasing open models, but are you really going to ignore all the terrible things they’ve done over the years just because of that?
As for DeepSeek or any other Chinese lab, I’m not aware of any practices that would make me consider them a bad actor. Can you say the same about OpenAI, Meta or Anthropic?
It makes me wonder why this dynamic exists here, and I do wonder at times how much our conversations here are influenced by China in a top-down fashion. I'd prefer to think that HN is pretty organic, but that is probably a naive thought.
> When an American company does anything? Doom. And. Gloom. The engineers? Taken to the slaughterhouse! America? Behind! The public? Bamboozeled!
I think I've always had a pretty healthy amount of cynicism towards China. In recent years my cynicism towards the US has increased significantly. I don't see all of my US peers with cynicism, but I think you're living in an age of grift, corporate capture, and unheard of corruption. I also think there's nuance to both. There are some US and Chinese companies and people that I do respect regardless of what's going on politically. (Meta / Zuck isn't one of them though...)
I live in the 51st state though, so maybe I'm just overreacting...
Apparently Americans haven’t got the memo yet that the world is moving closer to China.
I don't think that's entirely true. As an example you can look at how the European Union is increasingly putting up trade barriers with China and identifying opportunities for new rules and regulations to prevent Chinese dumping as America is also trying to do.
Japan, South Korea, the Philippines, and other countries participate in freedom of navigation and combined arms exercises because they perceive China to be a threat to their countries. [2]
I think it's more of a mixed bag. You see a lot of public talking points, and in Europe specifically a lot more healthy discussion about not being militarily as dependent on the United States as it has in the past and looking itself to follow Trump's lead to onshore capabilities (cloud for example) but I wouldn't read such moves as moving closer to China so much as they are hedging their bets a bit more.
Here's an article [1] that was reporting on this topic that I found interesting along with a select quote from the article:
[1] https://www.wsj.com/economy/trade/chinese-export-flood-tests...[2] https://www.msn.com/en-us/news/world/u-s-and-9-allies-just-h... - this isn't the "best" article but I just grabbed one to illustrate the point.
The Americans are writing that memo.
True.
Can you point to a lot of posts lauding the Chinese government based on the release of Chinese open models? Because that would be the equivalent to contrast with the OP.
I suspect it's just the generalized anti-West/anti-American sentiments extended into anything and everything. Anything that makes the US look anything close to good goes against their cause and therefore must be countered and talked down.
But you're not wrong about the bias here. You just don't see many comments talking about it because they get mass flagged/downvoted for obvious reasons.
These sentiments aren’t formed in a vacuum. America is becoming increasingly oligarchic and corrupt with decades of experience of companies profiting off harming people and lying through their teeth and Meta is like one of the worst offenders. They are pissing on every ally they have and once again started a war and both have material impacts on other countries.
Americans seem to take the US’ geeat reputation for granted and don’t realize how it has slipped and what that means. They also take for granted that China BAD is truth when this sentiment basically just sprang out of nowhere when the west realized it was their geopolitical rival. But to the rest of the worlds citizens, China is not starting any wars and is the source of cheap goods and innovation to other countries. EV batteries recently. The US is now directly causing high oil prices with their war and exports their rapacious companies like “prediction markets” which are 90% sports gambling now to the rest of the world. Meanwhile the classic American move to these kinds of comments is to claim that negative sentiment MUST be part of some bot campaign because surely no one could actually dislike the great America??
Those factors are all rightly part of the sentiment.
> America is becoming increasingly oligarchic and corrupt with decades of experience of companies profiting off harming people and lying through their teeth and Meta is like one of the worst offenders.
Isn't China a single-party state that is run by a corrupt autocrat, has literally enslaved people to build products, and disappears people for saying the wrong things? If America is becoming more like China, shouldn't you dislike China more than America?
> when this sentiment basically just sprang out of nowhere when the west realized it was their geopolitical rival
Well this isn't a sentiment that sprang out of nowhere. It is a sentiment that emerged based on real or perceived behavior.
> China is not starting any wars
Maybe not yet, but they certainly aren't behaving nicely toward Taiwan or other countries in and around the South China Sea, are they? The Philippines and Vietnam come to mind immediately, of course there are others.
> the source of cheap goods and innovation to other countries
But if those cheap goods run your country's businesses out of business and you have lots of people without work, is that a good thing?
> The US is now directly causing high oil prices with their war
The US didn't ask Iran to pursue building nuclear weapons, supply Russia for its ongoing war in Ukraine, or provide weapons and funding to groups in Yemen, Iraq, Syria, Lebanon, and Palestine which have led to instability in the region. If oil prices are too high, the rest of the world should have worked to address American concerns about Iran. They failed to act, and so the US eventually just did what it thought was necessary.
> rapacious companies like “prediction markets” which are 90% sports gambling
Before sports gambling was a thing in my state (Ohio) I seem to recall traveling to other countries and they had sports gambling too. Maybe I'm wrong?
> Meanwhile the classic American move to these kinds of comments is to claim that negative sentiment MUST be part of some bot campaign because surely no one could actually dislike the great America??
I can't speak for others but since I started this thread I'll chime in. I specifically disagree with the arguments put forth about AI strategy with respect to the US and China. I have no idea and frankly, aside from erroneously being accused of a bot once [1], have no interest in trying to guess if someone is a bot or not. Sometimes I just assume everyone is, because really what's the difference when it's all just text?
Others may feel differently and believe that there is a brigade of bots. I'd ask them for concrete evidence.
[1] https://news.ycombinator.com/item?id=48199327
I say China is not starting any wars, you gesture at some vague "not behaving nicely" as if this is equivalent to a war or some damning proof that they will start a war - these are nowhere near the same thing. China has killed maybe 50 people outside it's borders since 2000. The US has killed 100,000-250,000 in that time period. There is no amount of rationalization that can reverse the deaths of those people or the cities turned to rubble.
The attitude towards Iran is also just arrogant. The US believes they have the unilateral authority to dictate who has nuclear weapons and who doesn't, and is morally in the right for bombing their cities and killing their citizens if they don't comply. Sovereignty be damned, we are gods, fuck you. Same thing with literally kidnapping the president of another country. Not to mention, the Iraq war was started on the exact same WMD excuse and it turned out they never had any real evidence of WMDs back then, they just said they did and it was really about oil. And the government now is 10x more corrupt and cruel and xenophobic than it was back then, and back to a huge focus on fossil fuels - don't you have to be a little naive to just trust them at face value now?
I can tell you the prevalence of sports gambling and betting is exploding in Canada now that Kalshi is directly tying up with banks to put gambling inside fucking banking apps. I'm sure it was prevalent in many countries, I'm also sure it will scale up to huge levels and enter new countries with the amount of marketing and money and influence these US companies will bring, who also happen to have DJT's son on their board. Making them markets also frees them to manipulate and trade against their users covertly in ways that cannot be detected and that a traditional sportsbook could not do.
People who actually visit China or talk to Chinese citizens are all quite optimistic about it. The tropes people keep throwing around about autocracy are very inaccurate to anyone who actually reads anything about day to day life there. I can't explain all of it here but you can try reading something from Arnaud Bertrand if you're interested [1]. There’s a comical level of propaganda against China since the west recognized them as rivals that does not match reality at all. Anyways, I always see incredible whataboutism these days when talking about the US' problems now. This is what I think your original post that I replied to does too - the parent was criticizing Meta, and you said but what about China? And the thing is, we're talking about Meta because we live in the West and the horrible shit they do impacts us much more directly than whatever China does! It's not really productive to say China Bad!! in that scenario, it's really just saying "yes we suck, but they suck more!" and that doesn't solve anything.. it channels what should be a focus about huge corruption and threats to democracy in western countries into a pissing match on some other country, and its very sad to see how many people are falling for that.
[1] https://arnaudbertrand.substack.com/archive?sort=new
> Only China can release good, open weight models and American companies can’t compete. Oh by the way all the spend is for nothing because China alone can release open-weight models thus destroying American AI.
Let me be blunt and let me say: you don't understand why we people support Chinese models.
1. Chinese labs started with open weight models, US labs started with dooms day narrative
2. US VC based companies must become greedy to win and return the money, Chinese companies can make 1/10 of that revenue and still be happy
3. Meta in this case, started nicely with Llama, then switched to closed models, kicked out researchers to build data labeler CEO empire inside Meta. Now opening again, what's next? closing again?
US companies published their research on transformers. The only reason Chinese companies have any models at all is open US research...
Add distillation to that, and the entire Chinese AI sphere is really just an offshoot of public US AI information...
Science doesn't work this way, someone could claim only reason World is successful because Chinese invented paper, so please don't go there.
Regarding distillation, you can also say Anthropic and OpenAI stole /distilled books, articles, blogs from everyone who published things without knowing their work might be used against them
> Science doesn't work this way, someone could claim only reason World is successful because Chinese invented paper, so please don't go there.
Perfectly valid claim. It could be debated though, since you are talking about a centuries long diaspora of processes.
I'm talking about an architecture that was invented, published, and pioneered by US companies just a few years ago. Not really the same argument IMO.
> Regarding distillation, you can also say Anthropic and OpenAI stole /distilled books, articles, blogs from everyone who published things without knowing their work might be used against them
Yes. I would argue that anthropic et al are only able to exist because of the massive body of work in the anglosphere, both in books and online. That goes hand-in-hand with my view of the Chinese AI sphere.
All LLM’s are distillation of stolen copyrighted materials, some of which are the outputs of other LLM’s.
US companies started by publishing their research, but then they have stopped abruptly, seduced by the mirage of heaps of money that could be obtained by having a monopoly on AI.
During the last few years, the published research from USA was only a small fraction from that published from China.
There are comments like the one you're replying to on literally every Chinese model release. This is textbook goomba fallacy, btw.
Had to look this up.
https://en.wiktionary.org/wiki/Goomba_fallacy
Not applied accurately with respect to my comment, but it is a funny one and also new to me in the naming.
> It’s rather amusing to me to read comments like this, and then simultaneously whenever a Chinese company or team releases open-weight models or whatever there is a giant round of applause, America is so behind, and there’s nothing but positive things to say about the intelligent, creative, and well-intentioned Chinese engineers
It is absolutely applied accurately. You're commenting on the alleged hypocrisy of people simultaneously criticising American open-source while praising Chinese open-source, and then attributing your perception of hypocrisy to the website as a whole. The reality is the behaviour you've observed comes from completely different individuals, not some kind of HN hivemind. Your comment is such a typical case that it could go in the wiktionary page as the example excerpt teaching people what the goomba fallacy is.
both countries are authoritarian. both are using this "free" tech to spy on people and control them.
why is america and its people anti-china? take a chill pill and worry about yourselves instead of other people :)
There is a big astroturfing going on social media platforms by the chinese. Did you notice 'day in a life of unmarried 30 yr old lady in china' videos flooding usa social media.
Regular ppl in the west now hold mildly positive views of the ccp and how 'advanced' china is than usa.
Then there are europeans who now are looking for china to give them the technology handout now that relationship with usa has soured.
The interesting thing is that China began as an agrarian economy and quickly modernized via influence/direction of the CCP.
The United States used to have a dominant middle class that was geographically distributed (cities & rural areas inclusive), but the advent of the tech economy has also been having a similar effect here as it did in China: massive wealth accumulation in Tier 1-3 cities and everyone else being largely left behind.
It'll be interesting to see if there's convergence in the next decade or so, especially with the GOP reducing regulation and increasing the explicit capitalist priorities here. TBH, though, as a generally well-informed political outsider, the feeling I have is that the US government is slow, bulky, inefficient, balkanized and overall poorly run compared to the Chinese government. We'll likely either slowly improve authoritarian efficiency and become more like the CCP ... or we'll pivot left and move more toward the EU model, but we're floundering around right now paying lip service to both.
USA has to fall behind China much further before its people can make up their minds to pick a road forward.
I only see those type of videos on 小红书
https://www.youtube.com/watch?v=PEpo4yCut2E
i thought youtube and instagram are blocked in china. how are these ppl posting these?
Propaganda in China is distributed, decentralized—ie it's the responsibility of every minor bureaucrat, clerk, and white-collar worker to produce and engage with propaganda.
See:
https://collaborate.princeton.edu/en/publications/decentrali...
https://yiqingxu.org/replications/lpxx2025.html
https://cira.exovera.com/conferences/engineers-of-the-human-...
Deepseek never fucked us over. zuck has. A decades of harm creation run doesn’t get excused by the US flag. Zuck is not on your team and if you can’t see that by now, oh my.
> When an American company does anything? Doom. And. Gloom
Meta, "an American company". Being the main driver of an ethnic cleansing in Myanmar - and just sticking your head in the sand when told about it - is just another day's affairs at the average American Acme Inc.
These are comments on a release by easily the most societally damaging Western tech company there is. They so far easily beat Flock, Palantir, Anduril and so on, as a result of their incomparable scale. You're just ignoring that and pretending any negative comments are because it's an American company rather than Meta. That's much more FUD than any pro-China comments I've seen on HN.
Get off HN Mark, you have ten million pervert glasses to sell.
Sorry this comment is a bit snarky, but yours is indeed a sight to behold.
I'd also argue this is the case for any company releasing open weights. They're not righteous, they're marketing. That's not necessarily a bad thing! They're releasing some great stuff for free and we benefit from that. Every company doing this has a motivation to not release these for free.
Alibaba, Google, Moonshot, Thinking Machines, etc are not releasing their models for free because they love to. They want to grab market share. I'll take it.
I still will not use a hosted Meta product, but damn this model looks solid.
This model doesn’t look solid at all. It comes months after the Qwen model, and in almost half the benchmarks, it performs worse than that. Plus, the next Qwen 3.8 is going to be announced this week. So, this model is DOA.
I don't really care that much about benchmarks, but having tested it on one of my puzzle prompts I can tell you that it solves it well, writes clearly, isn't noticeably slower than Qwen 3.6 27B and is much more terse in its reasoning (which will help with preserve-reasoning).
It also has a knowledge cutoff inside this year.
The main limitation is the smaller maximum recommended context.
Meta can never be redeemed, but it's still valid to admit that FB at one point had a very badass engineering culture.
They're one of 2 companies I would absolutely never work for (weapons etc aside). FB's recruiters hounded me so often I requested that they blackball me. The day they became Meta, I learned this by checking my email to see that they started trying to reach out again. I once again requested that they blackball me. This by extention taints OAI, the other company I'll never work for.
After a few hours with Glimmer I'm pretty impressed. It's better than the benchmark scores seem to indicate compared to Qwen 3.6 27B. I'm very excited for 3.8
> FB at one point had a very badass engineering culture
Perpetually kneecapped by one of the worst management cultures I've ever seen
I'm going there: Bosworth is the problem. He should never have got near being CTO.
Would you say those badass engineers were/are managed by Careless People?
My impression is that they had created a system where management cared very deeply about "number go up", and very little about "which number?"
The book title is a reference to The Great Gatsby: "They were careless people, Tom and Daisy- they smashed up things and creatures and then retreated back into their money or their vast carelessness or whatever it was that kept them together, and let other people clean up the mess they had made.” ― F. Scott Fitzgerald
> It's better than the benchmark scores seem to indicate compared to Qwen 3.6 27B. I'm very excited for 3.8
Is it worth considering if it's only marginally better than Qwen 3.6 though? Qwen 3.8 27B is almost there, and will probably be better suited as drop-in replacement for 3.6. Not even considering there's probably going to be a 3.8-35B-A3B too - which will have even better performance.
Qwen3.6 is very token inefficient with it's thinking. Quantized versions often get into loops.
Glimmer is trained with 4 effort levels, not just thinking on/off. Maybe it's more token efficient in general. There's official 4 bit quantizations with reported 1% loss across 15 benchmarks -- so quants probably work good.
IMO that alone is worth trying for, even if they're otherwise equal.
It takes 10 minutes to download and try, any model is worth at least that. In my experience benchmarks are generally dogshit
FB/Meta have had "don't recruit me" databases for a long time that are very easy to get yourself added to, thankfully.
What is the other company that you would never work for?
He said OpenAI in the comment (if I read it correctly)
Ah, my bad. Thanks for pointing that out :)
Meta doesn't need to be "redeemed". They have two of the most popular social media apps in the world. And theyll prob survive without ever having you work there
I read that as, "Meta is a piece of shit but they're rich and don't care."
(Which I guess I agree with.)
Should have clarified. I meant my comment to be read as [random online person] is not particularly valuable and his comment gasing himself up as a high value meta acquisiton is polluting online discourse.
Tech equivalent of "I wouldn't date Sydney Sweeney, I'm not into blondes". Cool story bro
Mind telling me roughly what you had on your resume that had meta /fb hounding you for a job? ( Of course so I can avoid having this situation happen to me, naturally)
Simply don't get flown out by IG to be their first mobile dev.
Muse Glimmer doesn't redeem Meta, but it's a contribution to the commons and the Apache 2.0 licensing is an improvement from the restricted licenses attached to Llama. If even Meta can use a permissive license for its model weights, so can any other company.
How is this non-sequitor the top comment?
Thomas Bayes would say that the population of people who hate Facebook is really big and the population of people who are scrupulous about whether or not their comments are specific to the matter at hand is relatively small
Certain topics bring out the hidden Reddit inside.
First time here?
Unfortunately there are a few topics that short circuit some terminally only people. One of them being anything related to meta. Few others recently emerging is Flock or Musk. It's really exhausting since you can't have a discussion relating to anything that may be adjacent to said topics. It's like a black hole.
Heaven forbid people have a moral compass and communicate it
This is par for the course, HN is far worse than reddit on balance, especially involving upvoting/downvoting decorum.
Go vibecode something to auto upvote all downvoted posts, call it "Antiechochamber.HN" or something, and if enough people used it this website might improve a bit.
Lol at Internet points decorum.
It's trendy to hate on Meta just like it's trendy to handwave on Apple.
One can do no right regardless, the other can do no wrong.
At least in HN.
There is literally not a single comment like this, the only off topic comment like this is yours.
Your lamentations and opinions are noted. Do you have anything to say about the model? Something useful or substantive? Or is this just a place for you to let us all know what you are thinking these days?
You’re conflating the release of a local dense model that can benefit the ecosystem with the adverse effects of a digital ad system.
The latter bankrolled and continues to bankroll the former. It is not incorrect to conflate them.
To be honest the main issue with meta has never been around open/closed software. They've also done react, Cassandra and some other bits. But this, like their open weights is like a feather pressing down on the scale compared to things like promoting genocide in Myanmar, enabling Cambridge analytica, creating a huge closed ecosystem which dominate(s/d) local community communication, mandating doxxed communication, trying to replace actual community communication with algorithmic nonsense etc.
Crazy idea, maybe people can be happy a new open model got released, and still have nuanced ideas on meta as a whole.
A company is a big thing there's a lot of moving pieces, why do we have to evaluate it as a whole instead of just seeing it as it is?
If it’s open, do you care so much that it’s from Meta? At least it should be able to give you an honest answer about Tiananmen Square.
You can say the same about planet Earth.
https://www.meta.com/thefutureisforeveryone/
More meta derangement syndrome on HN, what a surprise.
We all benefit when companies invest their resources in producing open models. No one thinks this absolves anyone of being terrible elsewhere. But we can still be happy about it.
This is Apache 2.0, which is quite permissive. Just accept the gift.
These kind of responses are hilarious.
Someone gives something for free (and indeed this is entirely free) and the top comment is pure complaint.
All models come with some bias. Given Meta's track record, I would not touch anything from them with a 10 feet pole.
Any retort to do this like “but why would they just openly release this”[1] pretty much answers itself. Public relations.
If a company can spend money to redeem itself then, well, it can (game theoretically or whatever) do whatever it wants in the future and then spend money to wipe the slate clean.
[1] By which I mean: the very act of being prompted to ask such a question, of planting a seed like hmm, Meta might have some aspects which are good for us. You don’t have to be convinced of it. Just the seed itself can pay for itself.
Meta and its products, as a whole, is a threat to your kids, your mental health, your community's health and the planet as a whole. It is just sad and very repulsive everyone fell so easily addicted to their social drug. Yes - it is a drug, and it is hard to get off from.
Nothing redeems them at this point of time, they are doing exactly ZERO to redeem. Tossing open weight models (not opensource!!) is not a basis for redemption, and does not constitute remorse in any way. Trying to portray it as such is complicity to META's crimes against humanity.
Social media, often owned and perpetuated by Meta, has poisoned the world. It is not redeemable at this point.
I think it’s also worth pointing out that that there are numerous less evil options to choose from.
Perhaps none of the AI companies are shining examples of high ethics, but basically all of them have ethical high ground over Meta.
At least Anthropic isn’t sending private videos from pervert glasses to contract workers in Africa. It’s a low bar but it’s a bar nonetheless.
I personally don’t like, and wouldn’t work for Meta; but it’s an Apache 2.0 model.
I’m liking it, and I don’t see a personal moral contradiction here. Do you use React for frontend for example?
I also wish this HN post is a bit more focused on the release, and less noise around Meta.
There’s no moral contradiction, but there is a moral decision.
I don’t use React for front end nor do I willingly use any Meta product.
I would like to focus on the release of the Volkswagen Beetle and hear less noise about Adolf Hitler.
They're still using Elon's servers, though. Not like their models are any good anyways
what makes Meta so bad ???? they just your average billion dollar company
I can't take any Big Tech company that still uses PHP seriously. Sorry.
Pelican, rendered by Muse Glimmer on my Mac running LM Studio (with this model release: https://lmstudio.ai/models/muse-glimmer): https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
It has all of the components of a pelican riding a bicycle, though not exactly arranged in the right order!
(For comparison, here are the pelicans I got from Muse Spark 1, 1.1, and 1.2: https://bsky.app/profile/simonwillison.net/post/3mseqv5z4qk2... )
> It has all of the components of a pelican riding a bicycle, though not exactly arranged in the right order!
Maybe a sign that they didn't have SVG pelicans in the dataset
It is very bad with any svgs.
Picasso's Pelican
Happy to see meta back in the game, it's like after llama nothing came out that was comparable to mainstream open models.
Great to see Meta back, looks like really strong, local model, can't wait for llama.cpp support.
some support already merged, and I verified in a local build that it runs (cannot get MTP params working tho, about ~40 tok/s on my beefy 800GB/s 7900XT w/ 20GB VRAM). https://github.com/ggml-org/llama.cpp/pull/26841
Just tested muse-glimmer:30b-mlx on my laptop. Works great although a bit slow.
The post suggests that you need an rtx 5090 use it, which is currently selling for around $5,000 USD. I wouldn't exactly call that "my device", since my device costs about 25% of that for the entire computer.
For the same cost, you could run on a frontier model on a pro plan for two years. The economics dont make a lot of sense for this to me, so I would love some input on why people want to do this instead (privacy, for fun, etc).
If you’re doing breakeven math on subscriptions, consider that your own rig can run 24/7 whereas you will get a fraction of that with sub rate limits. Even if you factor in PG&E residential rates, the breakeven is a lot closer to months for overnight long-running agentic coding a couple times a week.
And in terms of interesting use cases: recently pointed an agent at Blender and gave it vision. That setup can essentially iterate on a scene forever.
It seems exceedingly unlikely that the current Pro plan costs will hold for the next two years. The subsidization train is going to end eventually.
Is the assumption here that inference costs will stay roughly static, or that frontier models will keep getting more expensive quickly enough to offset efficiency gains?
Because I don’t think “the subsidization train is going to end” necessarily means current pricing becomes impossible.
If capital keeps pouring into frontier AI, companies still have an incentive to subsidize access while competing for users and market share. And if that subsidization starts drying up, there’s even more incentive to bring inference costs down by making smaller and cheaper models catch up to today’s frontier capabilities.
So either way, I’m not sure you can extrapolate from the cost of serving current frontier models to what equivalent capability will cost two years from now.
I think like you mentioned, the practical reasons are disproportionately oriented around either privacy (, a clear constrained workload (need to OCR files/transcribe audio, and there isn't really a clear or meaningful reason to switch out the model to chase new incremental gains), or regulatory compliance (e.g. source code, patient data, can't leave the country).
The post does not imply the 5090 is needed, that is just a common reference point.
A single six year old RTX 3090 works great: https://www.reddit.com/r/LocalLLaMA/comments/1vkm42m/muse_gl...
I fully expect Meta will release other, smaller Muse models in the near future too.
The 5090 is also supposed to be a $2000 GPU, not a $5000 one. The entire market is utterly distorted right now, which will impact cloud inference more and more over time too. They are not immune to the absurdly high RAM prices, so their prices will have to go up over time too until the RAM supply chain goes back to normal.
I'm currently running it on an RTX 3090 (street price ~$1000 USD) with a long context and getting pretty good performance.
Prefill: ~1000 tok/s
Decode: 75-100 tok/s
It'll be far faster on a 5090, but I find the above performance to be acceptable. I've seen some claims that it even works OK on an AMD RX 7900XT (~$500USD)
The performance will be so-so, but you can buy an Intel Arc B70 for $1000. There are definitely ways to get going for less.
The comparison set is Gemma4-31B and Qwen3.6-27B, not the current Qwen
Fair on size, but the headline numbers are against a model a generation back
That is the most recent Qwen and Google models, there is no newer version, yet. Qwen3.8 27B might come in a couple of days tho, if it's launched alongside the large one when the Qwen3.8 countdown reaches zero.
What more recent open weight Qwen release is there?
Optimizing speed is really the way to go. Yet 24GB is not what everyone can afford. Maybe we could take some of those 56tk/s and transfer into some free RAM space using MoE loading ? I'd be glad with a less than 10GB and more than 6tk/s model.
Unfortunately this is just the entry price for LLMs. With the exception of the Qwen 27B models, I personally haven’t found a ton of use cases for models less than 200B. With the right setup, fine tuning, etc, you can make small models do cool things, but hard to please everyone given the insane hardware costs at the moment and the comparably cheap API costs.
Small models are still great for lots of “simple intelligence” use cases, like annotating or summarising files and media; or even just basic chat when given web search tools.
My local NAS is private and I’m not going to send it off to APIs for captioning or metadata; but even Qwen3VL 8B does an excellent job at this, despite being quite old.
They are also really excellent for fine tuning. Unsloth and Tinker (from Mira’s TML) are great places to start.
If your use case is narrower than “coding agent for everything”, you can probably match frontier performances on that narrow domain with ~30b and exceed it with ~100b+.
some small models are fast, and fine tuning can be done locally
for example in gaming context, if you need an answer below 5 seconds, they are the sweet spot
Gemma4-E4B (4B params) works pretty well as a local wiki, or when you don't have connectivity.
Nitpick: Gemma4-E4B is actually a 8 billion param model, but only 4.5B params worth of memory bandwidth needed per decode.
I don't thinnk just MoE will solve it. If you hit constantly different expert layers, you can't outsource layers efficently and have to swap it in.
MoE will be faster because it will read less memory for sure, you still have to have it though.
and immediately followed up with Manifesto from the man himself - what / how are they going to make of it longer term?
I guess for FOSS and self hosted it is good - but I am still wondering how are they going to Meta-stasize it ;)
Having played around with this model a bit, I am fairly confident that it is not competing in the coding space.
It can do that, but its actual selling point appears to be a different take on guardrails and safety alignment.
Either that or the only new training data left was industrial quantities of dark romance literature and Wattpad.
Clever business move. 131k context is more than enough for that use case, and due to that small K/V footprint, you can probably have a bunch of characters on the same GPU.
Or it's just a happy little accident. We will never know.
___
I was informed that normal people use LLMs for mundane tasks like asking for a pancake recipie.
That it apparently can also do decently.
Unfortunately, it is also very confident, regardless of whether it is actually correct.
So maybe it should actually stay the smut engine and nothing else.
good to see new open weights releases from meta
good looking showing too, which is excellent.
The least they could do, after ruthlessly bombarding my employer's servers with requests, ignoring the robots.txt, scraping everything, and incurring significant Google Maps costs for us in the process.
2a03:2880: by any chance?:
https://news.ycombinator.com/item?id=48137854
Have asked them to stop numerous times and they just keep hitting for about eight months now.
I tried to run it with lemonade by installing it via hf but did not succeed, it gets some weird 500 errors. I also see that ollama has currently only an mlx version available.
Anybody here succeed to run this on AMD?
Meta seems like the one American bigtech that would distill the the other American frontier models. My enemy’s enemy is my friend?
> Some have tried to frame distillation as harmful, but I think it is important to protect the principle that you can learn from anything you can observe.
- Mark Zuckerberg
https://www.meta.com/thefutureisforeveryone
You think the company buying up all the books, cutting off the bindings, and feeding them through a scanner isn't also distilling other models?
They do distill, their own bigger Muse model.
Let’s give thanks to all those meta engineers who have been ripped for my heir teams (while sitting right by them) working on manually tagging data. I guess the morale dip paid off in some way? I wish you all well and hope you find some happiness … IYKYK
Looking forward to giving this a try with llama.cpp. I’m watching the open-weights competition with high expectations.
Wow, Meta is back (at least for now)!
I like this class of model. Multi-token prediction makes it viable to run dense models at not-too-far-off speeds as MoE models with much better intelligence.
The submission’s title (open weights 30B local coding model) is luckily wrong: This is meant to be a general agentic model.
It even comes pre-quantized and with a MTP/drafter model. Looking good!
Let’s hope they aren’t dishonest with the benchmarks this time …
> The submission’s title (open weights 30B local coding model) is luckily wrong: This is meant to be a general agentic model.
https://xcancel.com/alexandr_wang/status/2086756152034066792
It's correct. See the OpenCode demo. Generic models are good enough for coding without necessarily being designed specifically for coding.
Right, so it's as correct as me claiming it to be an E-Mail sorting model. It may be good at that, but that's not its primary purpose.
> It even comes pre-quantized and with a MTP/drafter model
Glad to see the extra engineering effort that went into creating this local model and making it run well on a consumer device. I use qwen3.5-coder, and am waiting to kick the tires on this one. I hate to say this, but kudos to Meta ! I hope apple and others follow suit and create similar local models for other use cases like audio, images and video that can run on a laptop.
I'd really like to see a 45B-ish dense model ready for a dual GPU setup. Something with a little more intelligence while still within the range of some higher end local setups.
There is definitely an under-served target memory size of 48GB - almost everything aims for: 12, 16, 24, 32, 64, ...) But most dual-gpu setups, 3090/4090 (and some mac configs afaik) have 48GB, and most 64GB systems would do well with the extra 16gb of overhead saved. 48GB is also moderately common in PC memory configurations since 24gb DIMMs are a thing.
Meta released their own 4-bit quant of this model for devices with 24GB VRAM.
That's a modern gaming laptop; cheapest I see in the US with 24GB is $3.5k.
Should be quite a bit faster than the new M5 MacBook Pro, and you can run Linux on it!
Another candidate for the 7900XT (20GB VRAM) I got sitting around. I pulled latest llama.cpp (targeting vulkan during build) after seeing a muse PR merged a few hours ago, and unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL runs on my 7900XT barely (and with no MTP). Sits at 19GB VRAM w/ 4 parallel 113k context slots, all layers on GPU, and at 700 tok/s prompt, and ~36 tok/s generation.
Waiting on Q3 to download to check speed + do my usual anecdotes. I generate beefy code snippets and poems, and also ingest my HOA declaration and answer nuanced questions.
edit: i should've prefaced this somewhere with: This card ballparks at 800GB/s IO, which I can't seem to find easily on the market anymore. Kinda the ideal card for this model, if I just had a _little_ more VRAM (XTX is 24GB).
edit2: not mtp, this is dflash model (param in child comment). I'm up to ~60 tok/s generation and sitting at 19GB VRAM (i added --no-mmproj (makes it text-only i believe) because I'm used to speculative decoding wanting more VRAM and I'm already close to the limit :sweat_smile:)
Q3 results: unsloth/Muse-Glimmer-30B-GGUF:UD-Q3_K_XL gets down to 15.6GB VRAM and full context (131k) on the 4 parallel slots. Prompt/generation speeds about the same. Overall feeling like a nicer-fitting Qwen 3.6 27B, but want to test out MTP generation speeds once I can.
edit: My favorite bit of reasoning I saw go by in my "generate me a beautiful code snippet" anecdote: 'Could give a snippet of beautiful code: the "hello world" in brainfuck? No.'
edit2: my first dflash speculative model! no mtp. I'm up to ~60 tok/s on empty context with `--spec-type draft-dflash`
Many companies are stressed about token cost, as we are moving to a consumption based charge. In the meantime - new open source models, such as DeepSeek V4 Flash and GLM5.2 reduced the price to about 13x chepaer. Also OpenAI had reduced its price for considerably. Now Meta is back in this game. The upcoming months are going to be interesting (GoT)...
Meta did not abandon opensource. I would love to see a smaller distill, or a moe of this size but the benchmarks seems competetive as long as it isnt benchmaxed witch i would not be suprosed if it is.
> Meta did not abandon opensource
Open weights*
I don't think outside of the Big 3 (Ant, OAI, GDM), given the strong competition from China, any other Lab has a chance at capturing the coding market if they aren't open weights (save for xAI whose latest Grok looks every bit good & will probably rely on Cursor for distribution instead of going open weights). There's literally no other selling point, as the capabilities have mostly converged by now among the chasing pack.
Don’t sleep on NVIDIA and Nemotron.
It’s not completely open source, but they actually release their pretraining and post-training datasets with some redactions for (cough) pirated content.
They also have very good code and playbooks for actually doing a fine-tune, CPT, etc.
Even if you’re not tuning a Nemotron model, its mixes are very excellent for your replay data slice; or general experiments. Way better curation and quality than Dolma, etc; or other large huggingface data mixes I tested.
Yes, I second Nemotron. I'm using Ultra remotely and Super locally, and I find them very useful for RAG-like problems. I wouldn't really use them for coding.
There was a good discussion yesterday on the DeepSeek Flash release thread about this.
There's a large market, very large, who want the best regardless of what it costs. Probably a large enough market to keep that domain of research afloat (as opposed to shifting research manpower to cost cutting).
The reasoning is just that the marginal cost of AI is very secondary to fixed costs of the businesses themselves; it's not an excuse to sacrifice performance.
GDM -- Ok, I'll bite. Why are you including them?
Some labs go through bad patches, GDM is definitely in one right now and the recent departures are not reassuring, but I think it's too early and dismissive to count them out of the race so far. They just need one good frontier release for everyone to go "GDM is back!"
Claude models weren't really good or noteworthy until the 3 series anyway.
I think the record is quite clear, GDM was never in the race.
All of the Gemini models have been considerably behind the capabilities frontier. The only exception was 3.0 which seemed quite good, but had latent issues and we were all measuring with the incorrect metric, agentic where it's latent issues were very pronounced.
GDM+Google may have created an exceptionally efficient LLM for serving search. This is likely a great accomplishment (or maybe Google is burning money at a rate unheard of before). But Frontier capability: they have never been in the race.
This is sad, since they had everything necessary to be on or beyond the frontier.
The favourable comparisons to Gemma 4 and qwen3.6 look promising!
Those two offer MoE variants, this doesn't seem to.
Dense model makes it dog slow on anything without HBM. Max 15tok/sec on decode on DDR5 systems like a Spark or a Strix Halo -- and that's at 4 bit quant.
Dense models run at a very usable speed (Qwen 3.6 was running at ~50t/s last I looked) on my dual 7900 XTX desktop. (And before anyone brings it up, I did not buy them for this purpose, so the up-front cost is irrelevant in my case.)
3090/4090 probably would do 40 t/s, for 5090 75 t/s is shown in the blog.
The benchmark comparison is against the dense variants not MoE
Sorry I dont know if this is the right place but... 2000AD The Glimmer Rats , was the best drawn comic strip story by far in that publication.
That's a bit amusing - not that I have the hardware to run it, but officially it's not available in Hong Kong. Not that getting it would be much of a problem with a help of a VPN either, but I'll assume mainland China is also restricted. Certainly not a competition for Chinese open weight models... in China.
The combo that makes agents reliable: progressive tool routing, persistent memory, and multi-model failover.
Where is the pelican??
it's here! https://news.ycombinator.com/item?id=49245575
I'm listening to pelican sounds on youtube while I wait for Simon.
In my experience it's faster (10tk/s vs 35tk/s) and better than qwen3.6 series.
kind of a nonspecific complaint, but i haven’t yet had much luck with anything under ~120b, feels like models released on that order is coming to a trickle. the last few qwen models didn’t seem to go that high, and i got worse results than qwen3.5-122b
Next step: Burn the weights of these local models into an asic that ships cheap on a laptop (AMD/taalas looking at you), and I will be a happy camper. Make it pluggable so I can select a model I want. I use qwen3.5-coder currently on my laptop, and while it works well enough for me, it is somewhat slow processing tokens.
I would hazard a guess that fast small models with a smart agent harness can do quite well compared to large models which cant be run locally.
Just asking, what is the recommended models for M3 MacBook with 18G memory? Seems modern local models are not available.
You can try this site, toggle your computer specs at the top for a refined list of models and tokens/sec.
https://www.canirun.ai
Personally I would never trust a coding agent or agent harness from Meta.
I agree with their open-source model approach, but actually trusting Meta… to protect my privacy and my data… when it’s running on my personal hardware…
Not . In . A . Million . Years - that ship has sailed
I wish they would release the quantized versions in a safetensor format. Many frameworks can't load PTE and GGUF.
Try quantizing yourself, just a suggestion.
https://github.com/pytorch/executorch/tree/main/examples/mod...
Having just bought a 5070 Ti (16GB) instead of a 5090 (24GB), I am sad.
Why did you decide for the 5070 Ti? You will always suffer compared to the 5090?
I assume due to price. The 5070 Ti costs ~$1k, the 5090 costs ~$3.5k.
5090 has 32GB of RAM.
Throw in 5060ti. By the way 5090 is 32 GiB.
Meta knows how to win back developer's hearts .... let's see if they have the goods
If there is anything meta can do to regain hearts other than owning up their evil deeds, radically change their business model and paying up for taxes and damages, then the world is truly fucked and corporations will continue to win.
even 30B model is too large to large on local device (low end). meta should provide free hosted model api to use it.
Meanwhile those of us with 128GB RAM plus some VRAM don't have any good modern (last 8 months) open weights models to make use of all that. I don't care if it would run 5 tok/s, I want a smarter model than Qwen3.6 which avoids loops and can handle more context than 80k before crashing.
why don't you use the quantized version of kimi-k3
The SWE bench verified score is similar to Opus from not so long ago.
Sure, you can get better performance from cloud models.
But most software, not just AI, will be faster and more reliable in the cloud. The question is do we need that additional power and cost.
If the answer is no, then just like other software, people will run AI locally.
Refreshingly practical
How are you handling the tradeoff between quantization for device fit and accuracy loss on tool calling? That's where local agents typically break down in production.
Meta's clearly changing strategies back towards their original "frontier open source", but this time around they have a lot more competition from leading Chinese labs.
I'm all for it though, and I think Glimmer is a fantastic bet on locally-hostable models. I for one would love to self-host as much as I can.
Clear, useful, done
Can I run this on my RTX 5090?
Yes they have quants for 32GB and 20GB use-cases (including mmproj and kv cache + context)
How many data using in Polish Language?
Any MLX results?
https://pytorch.org/blog/fast-ondevice-agentic-ai-with-execu...
will it run on 2x 5060Ti with 16GB each?
It should - the kquant-dynamic variant is targeted towards 32GB. Downloading it now to give it a try.
It does, beautifully. Now let's wait for an NVFP4 GGUF!
yes. you can even parallelize two cards and get 1.7 times the speed.
Qwen 3.6 27B is still such a beast!
Random question: Would you be able to run this model on a Macbook Air M5 (latest)?
It it has less than 64gb then probably not
Thanks. Ah yes, I skipped over their own figures in the article.
K-Quant-17GB seems possible, though as they state 24GB.
LOL the mogging of GDM is hilarious.
I don't know why MSL released this, but it is very nice that they did.
The more open weight models get released the greater the market for personal and small business oriented hardware to run these models. This will drive lower cost hardware, which has stagnated in recent years due to most software not needing the performance and capacity.
Higher demand for 5090's did not make them cheaper, because Nvidia got much higher margin products to focus on.
The opposite happening because foundries are full to capacity making higher margin stuff.
You have to look a little past the current hysteria.
what's the catch?
From twitter Alexandr Wang
> 3/ muse glimmer was developed with its own architecture and recipe, optimized for its size and agentic performance requirements.
This means we're in the endgame does it not? If the architecture was NOT optimized for intelligence ...
and now the recent Meta model 'security issue' begins to make sense
thank you zuck.
what I can tell is that Meta is just starting and it is so underrated.
"Meta Muse" immediately made me think of Metamucil.
Product teams really need to hire at least one or two people with a 12-year-old's sense is humor. They need to winnow all the potential stupid jokes out of their product namings.
Quietly the best thing in the thread
PSA: Fast RAM isn't going to be getting cheaper anytime soon. Acquiring inference hardware is a really good way to own an appreciating hard asset. Learning how to use it and cool it is a hacker's journey worth taking. My 4090 I bought in late 2022 for $1600 is selling for a cool $3,489.95 right now, and going strong under nominal use. My DRR5 has tripled in value, my nvmes almost doubled. I grabbed a 128GB M5 Max MacBook Pro when they were still available and told all my friends to buy at least one. With that and a base M4 Studio 36GB, HuggingFace rates that hardware as:
> Amazing! You have a total of 128.94 TFLOPS of computing power. 71.3% percentile on scale of "GPU Poor" to "GPU Rich"
The way I see it, these are amazing machines that the richest folks are hovering up. I think they should be in the hands of regular people as much as possible. They depend on an incredibly global, increasingly fragile supply chain. If the become impossible to produce, their value would increase tremendously. I think they will become really valuable to you to use the tokens directly, but if that isn't the case, they can be rented out or resold. Please don't just buy any hold. Let's try to get as many people that can use them for decent things that help humans. For example:
https://spectrum.ieee.org/small-language-models-ai-pharmaceu...
As an industry, I wish we would stop calling these things "open weight" because it is too easy to confuse with actual "open source", which they are not.
Photoshop source code+ OSI license = open source
Photoshop binary you can run on your own computer = open weight
Photoshop SaaS web app = closed, proprietary (Opus, GPT, etc.)
"Open weight" models are still just binary blobs that are completely inscrutable. It's like bringing home a dog from the rescue and just hoping that it doesn't have a tendency to bite kids in the face. You just can't know. The only thing that you can do is try to add more training (fine tuning) telling it not to bite kids.
I don't think the FOSS community has ever accepted this, but somehow we're feeling like it is okay now.
Photoshop source code+ OSI license = open source
Photoshop binary you can run on your own computer = open weight
I don't think this is a correct analogy. You are not allowed to distribute modified versions of the Photoshop binary. Most open weight model licenses allow you to make and distribute your own finetunes, etc.
I believe that comparing LLMs with traditional deterministic software is fundamentally misleading. It is extremely difficult to truly interpret what LLMs do internally, and as of now, nobody fully understands it. Even if you trained the LLM yourself, there is no source code you can simply read and learn from.
Sure, having information about how these models were trained is helpful for reproducibility, but it is basically impossible for anyone without substantial capital and access to the same (likely copyrighted) data to reproduce the model. For normal users, owning the model weights essentially means owning 100% of the model, you can inspect and study the weights in much the same way as the lab that produced the model can, you can modify the weights, and you can use and distribute them if the license allows you to
Given an open weights model trained to sometimes bite kids, we can’t train it to not bite kids, even though billions of dollars of research have been thrown at this open problem.
Given an open weights model trained to never bite kids, you can get it to bite kids with 10 prompts and a linear projection, the known simple algorithm doesn’t even need a backwards pass.
yay asymmetry!
Any pointers to more info about that? Sounds interesting.
It is useful to indicate you can run the weights on your own hardware. That’s categorically different from most other commercial offerings. It’s as if your adobe example ignores the reality that would exist had photoshop been invented in 2019: cloud only.
This analogy is terrible and seems to be extremely misinformed about how rescues evaluate dogs before they are put up for adoption
I am extremely well aware of how rescues evaluate dogs. And I'm also fully aware that they do not know the full history of the dog. They go through a limited set of testing and interrogation to evaluate the safety of the dog. That's it.