This isn't a good analysis, and it's because it keeps rounding everything up. He rounds up the cost of electricity by 10%. He has a range of power use, takes the high end (which is 2x the low end) and multiplies it by the inflated electricity cost.
But then they talk about using a newly purchased Mac to do the inference, running at full capacity, 24/7. Why would you do that? Apple silicon is fast but the author points out: you're only getting 10-40 tokens per second. It's not bad, but it's not meant for this!
It's comparing apples to oranges. Yeah, data centers don't pay residential electricity rates. Data centers use chips that are power efficient. Data centers use chips that aren't designed to be a Mac.
Apple silicon works out pretty good if you're not burning tokens 24/7/365 and you're not buying hardware specifically to do it. I use my Mac Studio a few times a week for things that I need it for, but I can run ollama on it over the tailnet "for free". The economics work when I'm not trying to make my Mac Studio behave like a H100 cluster with liquid cooling. Which should come as no surprise to anyone: more tokens per watt on hardware that's multi tenant with cheap electricity will pretty much always win.
Even at just the electricity cost openrouter will be both
1) Roughly break-even to a little bit cheaper per token cost
2) Much, much, faster
So the cost of the mac barely even matters, it's just an extra cost beyond.
Sure, data center providers can pay lower rates.
The point of this article is that LLMs at home really don't make a ton of sense, unless you are willing to pay through the nose for privacy. There is absolutely no cost saving to be had.
If you're looking at your own datacenter as a larger corporate client, that could change.
There are also some providers that will contractually keep your data private, like AWS Bedrock or parts of Google/Azure (I don't know their stack names).
AWS even has AWS Secret Region and AWS Top Secret Region if you want to use LLMs on classified data.
You have to value privacy at a roughly absurd level to not want to use LLMs run efficiently at scale by someone else. For the home user, just the extra efficiency produced by batching requests from a large number of users in a datacenter in a real win.
Some of these companies are even selling tokens below cost to get marketshare. If someone will sell you a service for a dollar bill or three quarters, why wouldn't you take the three quarters?
It can get quite complex, for me the feeling of "I built this", and then watch as it makes stuff from the power generated on my solar panels is also worth a lot to me. The sheer coolness of it.
Not always. The calculations take its useful life expectancy as an input. If they estimate it correctly you have highly likelihood of it breaking, burning out or being woefully out of date by the end. At the 10 year window you are looking at losing support for security updates.
So if you are lucky you might end up with something that still runs but most folks won't find it particularly useful
Real estate is generally a "good" investment as it's considered a relatively safe way to get significant leverage. 5x leverage in the case of a 20% deposit, or even up to 20x leverage with countries that allow for 5% deposits (New Zealand).
In addition, the interest payments almost always end up being near the rent the owner would have paid, so mortgage payments are higher, but that increase is generally (and quickly becomes) principal while being able to counteract inflation of rent.
> Real estate is generally a "good" investment as it's considered a relatively safe way to get significant leverage
Leverage? People don't normally invest in property (normally involves taking out a loan) for the purpose of taking out another loan. That so called "leverage" is being used to buy the house...ie you don't have any leverage
The leverage is the loan taken for the mortgage. If you have a $1M property, $900k loan. If the property's value increases by 5%, that's $1.05M, so you've made 50% returns on your $100k capital invested. That's leverage, the leveraging of $100k to get the returns of $1M asset.
We can estimate this. US median home price right before the crash in 2007 was $240,000. Today, it is about $400,000. Median rent in 2007 was $810. Today, it is $1,698. There's some simplifying assumptions we have no choice but to make. Let's say renter's insurance is negligible enough to ignore. Meanwhile, we'll just let an online mortgage calculator assume a median $50,000 home insurance coverage payment and bake it in. We'll assume 1.1% of assessed value for property insurance, which is currently the US national average (it varies a lot state to state in reality). We'll assume an FHA loan with 4% down.
This gives us a $1,995 a month payment when we purchased and a $2,142 a month payment today, due to higher assessed value for the tax.
We can see upsides and downsides in both cases. Rent would have been quite a bit cheaper in 2007, but it has very nearly caught up by now. Meanwhile, you're probably talking about renting maybe a 2 bed/1 bath apartment, whereas the median single-family house is more like 4 beds/2 baths, with a yard. Whether or not that extra space and privacy matters to you likely depends a lot on whether you're single or have or ever plan to have a family. You could have invested into something like the S&P 500, which has historically returned about 10.5% since 1957 annually in nominal returns. Let's just kind of naively split the difference here and assume you can invest $1,000 saved on rent versus mortage a month for the first 10 years and $200 a month for the next 9. That would have gotten you somewhere around $240,000 by now. Meanwhile, you're looking at about $248,000 in home equity by now for the purchase case.
Choose different parameters if you please, but I'm not really seeing the case for renting here over the long term, and that's in spite of choosing the single worst time in the last century you could have made the purchase.
Oh I don't disagree, I hate real estate as an investment, it's a terrible asset only made "viable" by tax benefits, rent-replacement and excessive amounts of risk via leverage.
It's my shortcut for describing the idea that you'd spend $X a month on either interest lost to a bank or rent lost to a landlord, and therefore you can mostly consider that a constant expense when considering rent versus mortgage.
I’m covering my bases because Chinese real estate has been volatile recently and I’m not sure where the market is at now. It could be that renting is still way cheaper than buying, I just don’t have any direct experience to back that up. If I bought while I was living in Beijing I would probably be underwater with my investment right now, renting for 9 years was the right call and my rent was pretty affordable anyways.
Such cities still exist and have been in such a state for decades. They can change but that's meaningless as they can also change the other way around.
Articles like that still miss a bit of the nuance. Imagine having your house paid for, and you grow old and you have no rent to pay. Yes, you could have invested but likely you would have spent some of that money on something else, or your investments might have not worked out so well, or any other reason. Human reasons, to be specific. Owning property is like a lock.
Imagine having your house paid for, and you grow old and you have no rent to pay.
My home is "paid for". Except for the HOA and property taxes that are not that far off from what I was previously paying in rent, the ongoing maintenance costs with random large spikes, and the opportunity cost of having a large chunk of money in the house and not in the market. It was still probably the right decision, but it's not at all a free lunch.
Surely though, the HOA and all that would likely be baked into a renter's price.
Sure, the same way that the benefits of a fixed mortgage payment are baked into sale prices. The efficient market hypothesis would say that neither renting nor buying should be obviously superior in the long term, because if either was then people would bid up rents/prices until it wasn't.
And you didn't need to go live in a HOA
I pretty much did, unless I wanted to significantly compromise on other factors.
> The efficient market hypothesis would say that neither renting nor buying should be obviously superior in the long term, because if either was then people would bid up rents/prices until it wasn't.
Buying have much higher entry point, need a bunch of cash at start then a ton of paperwork.
It is absolutely possible that local buying market is inflated precisely because the area is so desirable buying to rent is (or was) good investment, but that's rarely is true for a bigger market
And it's gonna be interesting wherever this narrative will shift over the next 5 yrs
I keep hearing that properties are in the biggest bubble yet in the USA - with the affordable housing shortage being a red herring, because real estate managers and boomers are unwilling/unable to reduce their prices - despite not getting renters/buyers because it would kick off a death spiral as their interests would consequently go up (because of lower security). Along with the ai layoffs etc
I'm not American so I only hear the occasional interview so don't have any idea if it's really as pressing as these industry professionals keep saying but I'm definitely at the edge of my seat watching...
It is very close to universal truth, aside some small areas with very warped market.
Even if you move out after 5 years, you still own the place and can rent it out and then it pays for itself, to skip the cost of selling it back to market
It never fails, there's always someone who trots this thing out. We had bought our house, and then had to move and decided to rent. I was APPALLED that they wanted me to fill out an APPLICATION form, where they would decide my worth, and let me know if we would be allowed to live there. When buying a house, my cash was as good as anyone elses'. And then the management company would come inside my house to inspect that I wasn't running a meth lab or something. Thankfully that only lasted two years. I will never rent again. Majority owner-occupied neighborhoods have different characteristics as well.
> I was APPALLED that they wanted me to fill out an APPLICATION form, where they would decide my worth, and let me know if we would be allowed to live there. When buying a house, my cash was as good as anyone elses'.
House sellers receive offers from buyers, sometimes including letters, and can choose to sell to any of them (or none of them), whether or not those offers are higher than the listed price. It's not so different.
> And then the management company would come inside my house to inspect that I wasn't running a meth lab or something.
Yeah that part is different. I also prefer owning.
A seller might prefer a cash offer to an offer contingent on the buyer securing credit (credit might fall through). Or, like landlords, a seller might prefer a buyer with a higher credit score (same reason -- buyer is more likely to be able to secure credit and close the deal). A seller might prefer a slightly lower total offer with a serious amount of "earnest money" over a slightly higher total offer without significant earnest money (buyer might try to back out). A seller might prefer to sell to a family with a nice story in their buyer's letter than someone buying a 2nd or 3rd house. Or the seller may think all offers are too low and they can hold out and get a better offer later.
Real estate transactions are not very straightforward. A seller might accept a lower offer if they feel the buyer is more likely to be able to go through with the transaction or just go through it faster. And sometimes they do just plain like one buyer better.
Except one day the hype will catch up to reality that was always true, people will realize their $20,000 Mac is has less utility as a "way to learn AI" than some kids 3090 fortnite machine, and it'll be back to below MSRP.
It’s more than just data locality. OpenRouter is faster, no? I have an M4 pro, and anything but the smallest dumbest models are unusably slow for interactive use. I personally haven’t yet found a good use case for offline/non-interactive LLM work locally.
Yeah. The speed is the biggest issue. The intelligence of open models is good enough for serious work (though still worse than the frontier models), but the cloud models are often 3-7 times faster, and you can get more parallelization and so get speeds on the order of hundreds of tokens per second, which makes things fast!
Even extremely slow LLMs can generate Part B faster than I can audit Part A. So the LLM can generate Part A while I look over my email. Then it can worry over Part B while I look over Part A.
It can worry over Part C while I have my 10:30 group meet. And it can worry over Part D while I do whatever other silly, time-wasting thing all humans do in almost all organizations. Then I still haven't reviewed Part B, yet, so the extremely slow AI is waiting on me.
Maybe someday I'll be good enough to need faster AI so I can rewrite something like Bun in a few days. Right now, slow and local fits my use case very well.
I don’t think it matters if you’re “good enough” or not. Much of AI development is iterative. If you context switch between A from project 1 to B from project 2 back to check A, then maybe C while B finishes up, you will lose the flow state that AI assistance can enable with speed for those who are not fluent coders.
Sure, I can wait hours for my local model to finish, or I can spend basically as much and get the answer right away
There’s a lot of exciting stuff with local LLMs despite the speed, but for me I don’t have the discipline and working memory to jump from project to project.
And continuing the argument of "more than just...", if you stopped inferencing on your Mac you still have a generally nice computer. The difference between rent vs buy.
I played with classifying and summarizing my entire email history (per email) with small models, but that only took about 12h of GPU time at most. Using a coding agent cli wrapper in that case is far slower because of all the spin up cost and the system prompt they inject even if you want to turn it all off.
If I used an actual direct API it probably would've been much faster, but I'm doing it for hobby / fun reasons. You also get to fiddle with a lot more params.
I’m running a local Whisper + Gemma 4 pipeline with a cheap USB mic to extract health related data and potential todos from ambient speech. It doesn’t have to be fast doesn’t have to be 100% correct because if it captures at least a few bits of interesting information that would otherwise go unnoticed it’s still a win.
It's how marketing works. If something is a problem they have to loudly claim to have fixed it. Look around the economy and you'll see lots of it. "Healthy" (high sugar) muesli bars, clean-diesel, surveillance wrapped up as keeping us safe. The modus operandi of marketing is to change minds about self evident things otherwise what is the point?
The article makes no sense. I can't use OpenRouter as a general purpose computing device. Why are we comparing a whole computer to a single purpose SaaS?
They're responding to the people doing things like buying the most expensive Mac they can find specifically to do local inference for their AI agents.
Some do it to have control over their ability to use AI. Some do it because they think it will be cheaper to not have to pay a SaaS to generate tokens for them.
But for those interested in the latter case, it seems like it's not actually cheaper after all, at least at current prices. But then I don't expect prices to drastically jump because of how much competition there is in model development.
It's worth paying a premium for the privacy (assuming that llama.cpp and ollama aren't sending my sessions back to the cloud regardless...), and for the concerns about not getting a surprise bill.
Correct me if I'm wrong, but I believe this is a feature that only Google has figured out how to implement. All of the other pay-as-you-go token services have a cap you can set, some by monthly spending, some with API key resolution, others by how much you put into the account. I use many, and if configured with auto-purchase disabled, it's not possible to have a "surprise" bill (except for Google!)
You also have control over your costs. It is reasonable to assume that tokens will cost significantly more in the near to medium future as the market consolidates and subsidies decline.
Google, Microsoft, Meta, Anthropic, OpenAI, Oracle and others are going to be looking to recoup all the money that they’ve spent to date. Why would the price go down in the future?
Because price is driven mainly by competition, not by a desire to recoup prior spending.
Investors aren't doing things out of the sheer goodness of their hearts, so if they could just bump the price up they'd have already forced it up. The very existence of workable local models puts a cap on how high the price can realistically go, but the high level of competition still extant makes the price floor ever closer to the actual cost to generate tokens.
The AI numbers are huge, but I remember similar arguments about residential high-speed internet. According to Gemini, the "price for internet" is down 12% in real terms (ugh, capitalism!), while speeds are staggeringly faster.
The providers have spent a fortune on wireless, pulled a lot of fiber/cable, and it's cheaper than it was when it started.
No, that’s not the point. I think this is to help people who are thinking about getting a beefier Mac so they can run their LLMs on it too. Some in particular want a dedicated Mac Mini or Studio for this purpose. The breakdown, even if slightly flawed, offers a good insight into the economics of it.
For most people, they might be better off with OpenRouter models and providers supporting Zero Data Retention. On the cloud, that’s as good as it gets for privacy - your data is never retained beyond the life of the request.
> your data is never retained beyond the life of the request.
Like with OpenAI for a year?
” In June 2025, the court ordered OpenAI to retain its consumer and API customer chat logs indefinitely, including any that had been deleted, so they could be investigated […]”
I think it's because there are a lot of people writing articles about the benefits of running local models. I think it's fair to say that there are daily threads on HN singing the praises or local inference. I also see people buying new hardware where the main trigger is ability to run local models.
But the people who want to do local inference are putting some amount of value on privacy that’s not captured by the raw monetary value so just comparing the price is somewhat beside the point, it’s also true that, if you have eg a Mac and you use that as your main computing device then you would have spent money on it anyway, so you can’t even really compare its value to spend on something that’s not general purpose.
That's a lot of assumptions. I think there are also people buying new hardware specifically for this purpose, and their motivation to do it is thinking it will be cheaper in the long run. Privacy is not necessarily the motivation.
My overall opinion is that the smart thing is not to upgrade to the maximum memory for AI purposes. It's worth quantifying how much extra we pay for privacy.
That's the point: why would you buy a device that's specifically not optimized to be used for 24/7 inference? It's expensive hardware that's not designed to be used in that situation! The power use for inference isn't especially good and you're not getting even a fraction of the benefit from the hardware that you're paying for.
Nobody is suggesting you buy a pair of A100s, which is what 15k gets you these days. Get a used 5090. And the author specifically priced the hardware at over 4k, which is double the 1-2k you're noting
Good question but people are doing it anyway. It's a fact that right now tons of people are buying Mac Minis specifically for this use case, to treat them as their personal data center for agents. The concept of "power use for inference" is foreign. Those people are the ones that motivated this blog post I think.
The author isn't pricing in the multiple uses. You either compare it apples to apples or you don't. If you're using the machine for general purpose computing on top of inference then the amortized hardware costs are pointless to measure. This is exactly what I said.
Not sure where 40 tokens per second is coming from. I’ve seen 95-100 tokens per second on M5 Max 128GB running Gemma 4 31B. I’ve done experiments where it is faster than Claude Opus 4.5 for the same prompts.
It's actually a bit faster than that now it seems, about 112 tok/sec.
Configuration:
Gemma 4 31B Instruct Q6K
Context size 40960
LM Studio 0.4.13+1
Metal llama.cpp v2.14.0
LM Studio MLX (Apple M5) v1.6.0
Here are my results:
prompt eval time = 32545.36 ms / 5625 tokens ( 5.79 ms per token, 172.84 tokens per second)
eval time = 20227.99 ms / 310 tokens ( 65.25 ms per token, 15.33 tokens per second)
total time = 52773.35 ms / 5935 tokens
This was for interacting with a local MCP service, running a tool that returns a ~20KB text file to the agent to add to the chat context.
I'm seeing about the same number of tokens/second on an M2 Ultra that I have access to (also with 128GB of memory).
This is surely apples-to-oranges to the OP results (and I don't spend a great deal of time benchmarking these things, so my methodology might be lacking), but it's interesting seeing okay performance for a top open model. For most use, however, I find Gemma 4 26B A4B (Q6K) to be good enough (esp. for MCP calling) and much much faster (~1,200 tokens/second).
Actually, figuring it on generating tokens 24/7 is the best case scenario. if you figure it at 8 hours a day of actual use, you still have the fixed cost of the hardware being the highest portion of the budget, but now you generate 1/3 the tokens so you triple that cost per token.
Rounded up, yes, and oddly inefficient for someone obsessed with inefficiency. One could buy a brand new 64gb M5 macbook for well over 4k. Another could buy a scratched up but functioning M1 Max 64gb off of ebay for a little over 1k—and somehow get the same 10-20 t/s with 31b that the author does with an M5. Or better yet, have a frontier model do the planning and judging, and have a local MOE model execute at 50 t/s. All of this achievable by a former English major with too much free time.
I have an M1 Pro, and a M4 & M5 max to play with at work and the speed difference is very significant between all 3 machines, the M1 Pro is far slower, and the M5 is significantly faster than the M4. And a windows 3090 beats all of them but eats twice the amount of power per token. This is all running the same 24GB memory friendly model with LM studio.
The real reason this comparison makes no sense is that only a vanishingly small fraction of people seriously using ai to code would seriously use a model so far from the top models (including open source ones).
He should compare his MacBook to Open Router on Kimi 2.6 1.1T or GLM 5.1 (754B), at bfloat16 precision, which he can't ofc.
But it furthers his point that things like open router are a better idea, which is not surprising.
> Yeah, data centers don't pay residential electricity rates.
There are 2 caveats here:
Some places have higher prices for industrial than residential power as residential one might be subsidied by govt.
And DC also pay for cooling, which residential will only effectively pay if they have AC and is hot outside. So power rates are some multiply of industrial pricing.
Boss, I make 16.50 per hour, say 15, I work 36 hours, say 35, say 500 per week, say 4 weeks per month, that's only about 2000! Don't you agree I need a raise?
Your post makes sense if you bought the hardware for other reasons, and maybe run models occasionally as a novelty.
That isn't the case for many, though, and there is a whole social media space where people are hyping up the latest homebrew options for running models, believing it frees them from the yoke of big AI.
Millions of people are buying big $ maxed-out hardware like the Mac Studios or DGX specifically to run LLMs. Someone rationally running the numbers is a good thing.
This is HN; it has probably never occurred to half the people here that the average person even in first world countries doesn't even have the financial capacity to make an impulse five-figure USD purchase, even if on credit.
No one said anything about this being an "impulse" purchase -- it would usually be perceived as a "career investment" purchase, given that many feel they need to race to be with "it" or be left behind -- nor does "five figure" have anything to do with this -- DGX options are available at $3000 -- there's a certain irony that you are posting a comment that is basically "people in my country/circle can't, therefore no one can", while dismissing my comment that many people can.
Millions of people are paying thousands of dollars a year to buy a slightly upgraded entertainment package in their car. There are 60 million or so millionaires alone, including 6 million+ in China.
There are a lot of people with a lot of wealth on the planet. A lot. Millions...it isn't that unfounded, friend.
So doing this "this is HN" snide jerk act, and then basically projecting your lot on the planet is...I don't know if you intended it, but it's rather amazing.
Not just one, but two replies completely hung up on that. Like, why even reply when you already saw the other guy doing the same tired pedantry? Just wanted to feel like you contributed?
Ignoring that it was just tosser hyperbole (that absolutely zero reasonable people need to question), yes, enormous numbers of people are buying GPUs or hardware with the explicit goal of running local LLMs, and social media is full of people hyping various setups and models. Mac Minis are almost impossible to find, and that alone is selling at a clip of about 300,000 every four months. Large memory GPUs are basically a myth at this point. All so people can pay more to get a worse result than commercial options, which is precisely the point of the submission.
These local setups only ever make sense if you have something that confidential, or you're doing something that ToS of the majors would ban you for.
Now given this pedantry horseshit, you'll probably demand that I specifically show a citation on DGX or Studio sales, which...rofl.
Honestly, I don't even see my Macbook Pro costing me anywhere near as much as using any of these AI services, but maybe I'm just not seeing a significant increase in my power bill to notice? I am the power user who uses Claude Max pretty much all the time to prototype ideas, and build things I actually use, and has given me a lot of value, I work full time and have a family to raise and care for, my free coding time is mostly limited to ideas. Now I can draft a plan with detail, review the code, run the code, test it, and use software custom tailored to my needs.
Whether you think building data centers or not is a good idea it's inarguable that the per-token efficiency (power, hardware, etc) is FAR higher in a data center. That's literally what it's designed for.
I run two 49B parameter models on a pair of used A100s full time and it sucks down 250 watts at peak utilization. That's not gigawatts, and it's completely within the realm of comparison to what the author is describing.
Unless I'm misunderstanding, this is counting the entire laptop in the cost of generating tokens. The calculation seems to omit that, in addition to receiving LLM output, you have also received a laptop in exchange for your money. If you intend to put this machine in a dark corner and run it solely as a token-munching server, a laptop would be an exceptionally poor choice of technology for this purpose. But if you intend to use the laptop as a laptop, having a laptop is a pretty big benefit over not having a laptop.
You also get the benefit of privacy, freedom from censorship, and control over the model used (i.e. it will not be rugpulled on you in three months after you've built a workflow around a specific model's idiosyncrasies).
The base 14" m5 MacBook pro is $1700 with 16gb/1tb. The author's spec is $4300 - $2600 more.
It depends on how often you use it (and your tolerance for slow inference) and whether you would have otherwise bought a higher spec. For my needs, this costs a LOT more.
Sure. I have unlimited credits for AI with my job, but I’m learning by running things locally. Start by running OpenRouter, then downloading open source tools to run locally, then modifying some python, now creating pipelines in rust. Yes, the irony is that I’m using Opus to crack these things open.
I think you'd need to tinker quickly, realize anything with CUDA (other than the awful DGX Spark) is better for learning, the prefill is killing your ability to actually run models large enough to justify that RAM, and then cure yourself before the rest of the crowd.
Given those models aren't sold at MSRP anymore and are primarily being sought for inference, it's apropos of an article on the costs of inference on Apple Silicon.
Not sure what you mean by "can't order", both of those links show purchasable, orderable computers with RAM upgrades. You can get the Mac Studio in 12 weeks or the MBP in just a few weeks, with RAM upgrades, by ordering right now on Apple's website.
OP is giving you the absolute best case compared to most of the people who've been overcome with psychosis hoarding Macs.
An unreasonable number of these people spent $10,000+ for Mac Studios that are still compute bottlenecked and don't have anything more efficient than Gemma 4 to run.
Excusing everything else that u/bastawhiz said[0]; the obvious fact here is that Claude, OpenAI, Gemini et al. are quite literally burning through 100's of billions of dollars and selling it back to you for pennies on the dollar in the hopes that they get to be the only one left.
If I spend $10 growing Oranges and sell them to you for $1; then of course it's more expensive for you to do the growing.
I feel like I'm taking crazy pills. These models will become more expensive over time, it's functionally impossible for them not to, they just want to capture the market before they have to stop selling at a huge loss.
That seems unlikely. There are many providers for open models on openrouter. It seems unlikely that they are throwing money away for each token they sell.
Also, there a good technical reasons for inference being much more efficient at scale.
Sure. And there’s Lyft and Uber and plenty of others. And Grubhub and DoorDash and uber and how many others. And I don’t even fucking remember how many electric fucking scooter companies, I’m practically falling over scooters! I’m sure they aren’t earning market share by selling at a loss either.
Which, by analogy, would mean you could use subsided tokens for over 10 years and then, and only then, actually purchase your own hardware to do inference, with over a decade of technological advancements to compound.
I'll invest in inference hardware whenever the economics make sense, not because of my prediction that prices will get higher (also, because smaller models keep getting better and they might just suffice for most of my use cases in a few years time).
My biggest worries in terms of cost is in regards to training. Whenever that gets too expensive for Beijing to pay, we won't be getting new SOTA open small models to run on local hardware, which, again, reinforces the decision to use providers for AI inference for now.
> Whenever that gets too expensive for Beijing to pay, we won't be getting new SOTA open small models to run on local hardware, which, again, reinforces the decision to use providers for AI inference for now.
Valid takeaway.
"Invest" in the companies that release their models, because the alternative is to reward the people who will capture the market and jack up the prices to unreasonable level.
Unfortunately I think this is a bit like game theory, people will choose the local optimum despite the long term problems with doing so.
The providers on OpenRouter serving open models aren't "throwing money away", agreed.
But that's not the point I'm making. (or, it kind of is, but it's more high level than that).
They're running spot and preemptible GPU instances (60-80% cheaper than on-demand), paying wholesale industrial electricity rates, and running at multi-tenant utilisation densities that make your MacBook look like a bonfire. Of course they're not individually loss-making on inference, they're aggregating cheap commodity compute and skimming a margin, and on paper that's what makes it seem like a good idea, certainly not a loss leader right?
But zoom out a bit; the entire stack is swimming in VC money. OpenRouter itself just raised at a $1.3B valuation backed by a16z. The Chinese models that now account for 36% of all tokens routed through the platform (DeepSeek, Qwen) are priced the way they are because Beijing-adjacent capital has decided market share matters more than margin right now.
So yes, technically no single party is "throwing money away" on each token; they're just all simultaneously subsidising different parts of the stack for strategic reasons. The floor price you're seeing isn't a stable equilibrium, it's a pile of investor money that hasn't entirely finished burning yet.
> The floor price you're seeing isn't a stable equilibrium, it's a pile of investor money that hasn't entirely finished burning yet.
All that says is that it gets more expensive in the future as competitors exit the market and sustainability becomes important. That’s why Uber and Lyft were so cheap until they killed taxis. One major difference of course is that some models will remain largely good enough and the incremental cost of running will keep dropping to 0 over time since the hardware needed doesn’t get more expensive and is already purchased.
Most of these folks aren't running on spot/preemptible instances, they're doing 1-2 year reserved rentals. There isn't that much cheap compute floating around and you can be quite profitable on reservations (if you can get them — the compute shortage is real).
I think the question in terms of throwing money away isn't the inference layer: it's whether the companies training open models will be able to financially keep doing so. How long will Moonshot keep releasing future Kimi models? I think there's an interesting wedge they're exploring with being basically a base-model-trainer-as-a-service, i.e. selling rights to Fireworks to sell finetuning services to the Cursors of the world, but it's entirely possible it doesn't pan out.
That being said, Nvidia seems willing to step up to being the base model trainer of last resort via the Nemotron family of open models, since it helps sell more of their hardware — similar to their investments in the CUDA stack to sell hardware (unsurprisingly, Nemotron is designed to run most efficiently on Nvidia hardware, e.g. native NVFP4). So I suspect there will continue to be a pretty good market here.
The blog compares the cost of running Gemma4 31b, which on OpenRouter is offered by small no-name inference providers, not by frontier AI companies. It seems like a fair comparison to me.
LLM generation is bottlenecked by RAM bandwidth and latency. You can get almost linear scaling by evaluating more prompts in parallel, because the GPU has nothing to for the relative eternity it takes to read all of the weights from DRAM for every layer for every token.
On Apple Silicon you can get 4x-8x more tokens per second if you run more queries in parallel (as long as your inference server supports it, and has enough spare RAM for more KV caches).
When inference is done at datacenter scales, when you distribute generation across multiple GPUs and have kernels carefully tuned to specific hardware, the compute vs DRAM bandwidth speed ratio gets absurd like 200:1. That's why everyone gives you batch inference at a steep discount.
Interestingly if you look at the cost of Gemma3 (this is 12 months old, but demonstrates the authors point) on Vertex AI versus Gemini 2.5 pro, the cost per million tokens WAS very similar.
So many more efficiencies possible at scale though. I cannot keep a local model 98% utilized 24/7, at least not with my current workload. A big cloud can. I can’t power my servers with DC, I have this AC to DV conversion nonsense. The list goes on.
This is not true. API tokens are not sold at a loss, and hardware gets more efficient over time, so serving inference on the same model gets cheaper. LLAMA 3.1 405B parameters was $6/$12/M tokens in 2024, but in 2026 that same model is $3/$3/M tokens.
The most intelligent model at a given time is much larger than the previous, which is why token costs for GPT5.5 are higher than 5.4. But you should expect that 2 years from now, serving a GPT5.5 sized model will be cheaper than GPT5.5 today. You should expect it to be even cheaper to get an equally intelligent model 2 years from now, because distillation techniques are effective at reducing the necessary parameter count for the same benchmark scores.
So are they going to stop at GPT 5.5? This analysis only seems to be counting inference cost when the majority of the cost, and why they are burning through money, is the training.
Well, I'd be surprised if non-R&D inference providers were selling at a loss. There are a plethora to choose from, competition is quite healthy. Will they keep providing cheap tokens while the labs raise their prices? Probably, but then I don't see how they could be raised in the first place. And what timescale are you talking about? A couple of years? It is appropriate to assume inference will become more efficient over time. If you raise your prices, you are going to be out competed before it's profitable (if you assume it is unprofitable) which would be negligent. I don't see how this makes sense.
Which is where your analogy breaks down and why you think you’re taking crazy pills. Inference is growing and selling the oranges in your analogy. Model building is growing the farm to sell larger, juicier more addicting oranges.
The same mistake was made with Amazon, and a million other tech companies in the early 2010s.
Amazon were losing money, they were losing money because were growing and spent all of their cash flow on growth. It wasn't merely regarded as a hopelessly unprofitable business, if was regarded as potentially fraudulent. The share price collapsed in 2014 because, some thought, the profit would never come, investing in growth was pointless, etc.
Last year Amazon made nearly $100bn in profit. Stock is up 20x from then...this is after AWS was known (everyone also that was a massive fraud, could never be profitable...we know it was printing from day one), after it was the world's biggest retailer, etc.
It is difficult to understate how consistently people make this mistake, not just individually but in aggregate. You see the same thing with restaurants, consumer products, office leasing, so many businesses. This is not to say that the future will happen any particular way but that what Anthropic and co are doing is obviously rational and based upon very real cash flow. Anthropic's growth in revenue is, I believe, unparalleled in modern corporate history. A slight difference in this case is also that the economics of training these models is improving exponentially over time.
The restaurant next to the mines were profitable up until the moment the mines themselves shut down: one doesn't exist without the other.
You can't ringfence inference as "the profitable bit" and then hand-wave away the training. Without continuous training there is no inference product.
Claude 3 Opus isn't sitting there making revenue in 2026 - the thing is just deprecated. The moment you stop spending billions on the next model, your "profitable" inference business is on borrowed time until someone else makes it obsolete.
Maybe I made a mistake in my analogy... They're not growing a farm and then selling oranges. They're on a treadmill where stopping is death, and the treadmill costs $10bn a year to keep running.
> They're on a treadmill where stopping is death, and the treadmill costs $10bn a year to keep running.
You’re literally describing all companies. Google takes about $270bn/year to run. If they stopped spending that they’d die pretty darn quick. It’s also a description of working - unless you’d built up significant savings, if you stopped working you’re also going to die.
No, not quite. It really comes down to opex vs capex and the depreciation schedule for your investment.
Software development is typically categorized as capex, on a 3-5 year depreciation schedule. You assume the software you write today will be generating value for you that long.
If a big, expensive model training project only gives you value for a year or less, that is not like most companies.
No, the IRS made that change a while back as part of the TCJA but that’s been reverted in the OBBBA. If you build something and never touch it, sure that should probably be capex you have to depreciate. But if you’re investing continuously in it over time, I don’t see how it’s anything other than opex - there’s nothing being depreciated because you’re constantly improving it. Automobile manufacturers don’t have to count their labor force as capex. Indeed I can’t think of any other industry where labor is capex.
But believing that the financials of a project are governed solely by how IRS rules force you to account for headcount is kind of silly.
> If a big, expensive model training project only gives you value for a year or less, that is not like most companies.
The model itself that gets built? Sure (although clearly the timelines are getting longer). However the important bit here is the research that got done along the way and the infrastructure built to make that model building process cheaper, better etc. all of that stuff sticks around but because it’s hard to appreciate externally you discount it to 0 when it’s literally what they actually spent the money on.
But none of that even matters. Google had 270B in opex and their capex has grown from 50B in 2024 to 90B in 2025 and is projected to grow to ~175B for 2026. But even if you discount the “AI” treadmill, you’re still looking at many tens of billions in capex that if they stopped they’d die.
You're saying no one capitalizes software development costs anymore? The rest of the internet disagrees strongly. The finance team at my employer would disagree strongly. Accountants talk to dev teams all the time about how many hours went into new development vs maintenance. It's not just a temporary IRS rule. It's GAAP.
It's the same in other industries. In your example, if labor went into building a whole new assembly line that pays itself off over several years, capex. If it just goes into building products to sell, opex.
Software that is sold as a service and requires ongoing maintenance like running in the cloud (and people to keep it running in the cloud) is opex not capex. Google Search is most definitely opex.
There was some accounting changes passed during the Biden administration’s funding bill that made it capex for a few years. It’s been rolled back in the Trump administration’s bill and I suspect no one will ever let it happen again.
The problem is I don’t think computing is going back to the mainframe era you know where all the computing is done remotely and the only thing you have in front of you is a terminal that is the AI slop maker’s dream, the computing power on the desktop/laptop/tablet/phone is getting better and the models are getting smaller and quicker.
There is no moat. In the end, what we are calling AI today will just be something that is incorporated into an existing programs that people will use to help them accomplish a task. The public will not be paying more for it. It will just be a commodity added to the existing ecosystems we have today. They
> Without continuous training there is no inference product.
This claim deserves teasing apart.
Clearly, training is a Red Queen's race today. If a model provider were to unilaterally decide to stop training, they would very quickly lose market share to competitors with better models.
On the other hand, what if market and investment conditions change such that everybody has to stop training?
In that case, the models are still there and still as useful as they were the day before. So why wouldn't there still be an inference product?
Unless they are changing the architecture in huge ways. The pre-training done for 3 goes into later models. I am sure the frontier labs are figuring out how to pretrain generic feedstocks that can be fed into downstream training pipelines. DeepSeeks incremental training run cost was what, 5M? Alibaba and DeepSeek have the best most efficient training pipelines, look at the rate at which custom Qwen models are being pumped out.
What's the point of these words and analogies when the only thing that matters is numbers. Gross margins of 20% versus 70% makes a world of difference (literally the difference between a company that's about to collapse and a multi-trillion dollar self-sustaining juggernaut) but in your world of words these two companies are the same thing.
Dario has stated they made more from selling inference to Opus 3 than they did training it. Same with 3.5, 4.0 and I'm assuming 4.5 as well.
The reason they're losing money on paper is because the models keep growing 10x in size every generation but they're not getting 10x returns on model inference (closer to 2x)
I know what you’re quoting, but you've badly misread it.
Dario's point was the opposite of yours; he used per-model accounting to explain why the company P&L gets worse every year, not better. His own numbers (10x training costs each generation and ~2x revenue return.)…
"It looks like it's getting worse and worse" are his words, not mine.
In 2025, Anthropic's inference costs came in 23% over their own projections. They cut their gross margin forecast from 50% to 40%.
> Inference is growing and selling the oranges in your analogy. Model building is growing the farm to sell larger, juicier more addicting oranges.
In this analogy, model training would be akin to developing better oranges, but your competitors are also developing better oranges so if you stop spending heavily to improve your oranges, consumers are going to buy ~zero oranges from you within a couple years. (Expanding the farm might be analogous to expanding data centers.)
In this particular case, inference and training are intertwined. It might be one thing if Anthropic could get away with training a new model every five years and control costs that way. But they can't. Put another way, their inference has no value without continuous, very expensive training. Because consumers aren't purchasing based on price but capability, otherwise the Chinese models on OpenRouter would have buried OpenAI and Anthropic already.
Last month Anthropic tried to control the narrative by drumming up the “super scary AI” trope.
The news they successfully buried was that companies like AirBnB are now running Qwen and open source models. The free oranges are now good enough. There is no future unless the goal is to get to super intelligence and utterly take over the world before anyone else gets one. Anything else and free models are six months behind. The money now is the opposite of what everyone thought a year ago: datacenters. Everyone thought AWS was fucked. Turns out AWS is really good at running Qwen.
My reading on anthropic is that he strongly infers that they're profitable, realizes what he has said and immediately walks it back as explicitly not the case today and reframes it as a guess about some indeterminate point in the future.
> Those are the economics of the industry today, or not today but where we're projecting forward in a year or two.
The models have been dropping 10x in price for completing the same tasks, year over year. Even if you think Anthropic is losing money charging 10x more than everyone else for their 400B model, the prices will continue to go down based on model improvement alone
These models will become more expensive over time, it's functionally impossible for them not to, they just want to capture the market before they have to stop selling at a huge loss.
They could have said the same about transistors. People keep inventing new ways to keep the costs down. Just look at the latest Qwen, DeepSeek, BitNet. Interesting tidbit: they’re all open, and as Google said in 2022: they have no moat.
But isn’t training models, a forever task like iterating in tech you can never take a day off, adding humans to the equation don’t humans train/teach themselves new skills over a lifetime, and isn’t one of the selling points in the future when selling this AI slop your AI never goes to sleep and can always be trained forever? The AI price for entry as we go on into the future will only increase.
I agree that training is a forever task, and the current rate of training is probably not sustainable. But all that means is that once the current investment mania ends, the market will most likely find a new equilibrium where continuous training still happens, but at a slower rate that can be sustained by inference revenue.
> but at a slower rate that can be sustained by inference revenue.
reply
also it's possible that the scale of inference needed (e.g. Jevons paradox) keeps growing to the point that training costs can fully be absorbed (since training cost is one off vs. inference that can scale).
(I suspect that might be the thinking, I don't know if it will be true, it's also possible that no model will create a moat big enough to attract enough of the inference traffic to make it true).
Depending on the chips/architecture used, the off-peak traffic from inference can also subsidize the training costs.
Just keeping it up to date with competitors is much cheaper, by copying better ones like Qwen did with Claude. Also a bunch of research is trickling into open source / arxiv so catching up should continue becoming cheaper at least as a fraction of training from scratch
The inference is absolutely not sold at a loss, at least not when paying API prices (the subscriptions are less clear). The reason frontier model companies aren’t profitable is because training the models is so costly, not inference.
Seems to be on its way! I know of at least one person whose company is looking at a 20x increase, and afaict (from related looking around, nothing concrete tho) business accounts are missing some costs in the calculator so it'll likely be higher.
You should probably take some stay-on-topic pills, as this article is clearly and unambiguously talking about open weight models (e.g. gemma 4), not the ones allegedly being sold at a loss (Opus, ChatGPT, etc)[0].
[0]: these API are not sold at a loss either, by the way. But it's a nice meme so let's just pretend they are.
Except that’s not what the analysis is. They’re spending < $1 to get $1 from you and the other $9 to figure out how to improve the model further and build up products on top of that to turn that $1 spend into $5 in the future.
In other words, inference is fairly profitable for them and the rest of the money is spent growing revenue as quickly as possible. Building models is still an expensive line item but the costs for that are going down with time.
There is also maybe a “capture the market” mentality but I don’t think that’s necessarily it - the tools and processes are largely fungible and that’s a huge problem. They need to figure out how to make it sticky for “capture the market”, but there’s also a very real “grow as big as possible as quickly as possible to take on Google”; Google has an existential threat here.
I feel like I read this every day for years that Uber did this same "idiotic, losing" strategy (how it was pitched/discussed) and then one day we woke up and... without much fuss, boom, they were profitable seemingly overnight.
Well and uber cut the driver pay in half and doubled the price. They didn’t really find any efficiencies, robo drivers don’t exist yet. Also why I hardly touch them anymore.
All that tells me is they did find an efficiency. If they didn’t, their driver supply would have dropped. Unlike the taxi business, Uber/Lyft can tap into otherwise dormant supply of drivers who already own a car but aren’t willing to spend all 40-60 hours a week driving a taxi. With Uber/Lyft, they can become part-time drivers (they have flexibility and they can use an asset they already own anyway). Is it worse for the full time taxi drivers who used to have the supply artificially constrained in the old medallion system? Yes, but does it also benefit others who want to do this as a flexible job, zero skills required other than driving, no boss to deal with, no job interviews, etc. Yes!
As long as you have slaves/sharecroppers, driving the people at the top of the pyramid at Uber they’re profitable and Uber makes money as long as you don’t care about the workers and as long as you can get around all of the regulations that are put on traditional cab companies if there are any left on the road.
For me nothing says low class like the Porsche dealer saying we can call Uber for you to take you home ridiculous… and it was a low class experience dirty car small never again ha ha ha…
Why? It's no less crazy than when Uber and Lyft were doing the same thing. Or when the entire tech industry was doing it in the dot com boom.
Investment-driven market growth at a loss is like the least surprising thing in all of this. The tech is new and fascinating. The bubble is just another trip through the funhouse.
There are huge economies to be had by batching requests and using lots of RAM for MoE (sparse models). You can't achieve that efficiency at batch size 1 on a single node.
If you want a good dense model, use qwen3.6 27B instead, speed will be up, and if you don't take my word for it being smarter, take openrouter's prices of it against the bigger, slower and less memory-efficient gemma do the talking.
If you want a faster model, go for qwen3.6 35B (or gemma 4 26B if gemma models perform better for your tasks). There is a reason why people (myself included) haven't shut up about those two (especially the 27B). Its small enough to run at a decent speed (especially with the built in MTP that finally has official llama.cpp support) and for many workloads (every benchmark I have ever thrown at it) it is matching or surpassing models it has no right to.
A couple of days ago I woke up with my internet being down, started 27B in pi, told it to diagnose whats wrong by giving it my router's password, went to grab a coffee and by the time I got back, i had a full report with suggestion on how to proceed. I love openrouter and I use it for many things, but it is not cheaper.
Subjectivity and opinions based on personal experience with all those models implied naturally, I assume the 31B gemma has cases in which it edges out, I've just failed finding any and I have been running all 4 models mentioned since hours after each of them dropped nonstop for different tasks. Hell, for my hermes, I've started getting better results once I switched from gemma 4 26B to qwen3.5 9B, not even the massively improved 3.6 series. It just feels outdated/ cherrypicked to not use what by many accounts is the current consumer hardware SOTA if doing such an analysis.
Right. Qwen 3.6 45b (6 parameter) runs on a commodity 5090, which, if you're into video games, you probably already have one of. It is entirely usable for most code generation tasks. (Not all, but most.)
Likewise, DeepSeek V4 Flash is quite accessible on local models, with DwarfStar 4 making it easy to run on a 96GB MacBook.
There's nothing wrong with paying for inference, but local models bring up some pretty amazing possibilities, such as entirely offline usage or being able to work on private PII, legally privileged, etc. sort of data, or performing tasks with no concern given whatsoever towards billing overruns.
The other possibility is being able to build a service which you can be 100% assured you can keep running without worrying about a service going down or being end-of-lifed, which is currently a problem with frontier models. My local Qwen setup is entirely predictable. It can run as long as I can keep finding hardware to run it.
A sensible strategy uses both: have local inference tools available, and use both low-cost and high-cost cloud based models. You can use GPT-5.5 and Opus-4.7 for things they excel at (including laundering the latter via a Claude subscription to make it cheaper) for demanding reasoning tasks, DeepSeek V4 Pro for slightly less demanding tasks, V4 Flash for most (not all) code generation, and then local models for things where you want a local model.
> If you want a good dense model, use qwen3.6 27B instead, speed will be up, and if you don't take my word for it being smarter, take openrouter's prices of it against the bigger, slower and less memory-efficient gemma do the talking.
Don't know if this is the correct read. I think those providers are simply taking cue from Alibaba's first-party pricing for the 27B Dense. It's kinda overpriced imo. Perhaps it can be explained by how 'reasoning-inefficient' (relative to frontier models or even Gemma) the Qwen models are and longer sequence lengths are expensive to serve.
I feel like if I had the infrastructure and saw that there is a huge interest in the model, i'd just undercut alibaba's prices a little harder to grab all the consumers. I am sure that the providers have done the math and found that there is a reason not to do this (compute-bound if too many users?), but the delta is very stark, especially for output. Last I checked the cheapest 27b on openrouter was 2$ out vs 0.38$ for the 31b.
But I do agree that the openrouter prices aren't a strong signal and probably should have worded it a little better. It's just a really stark and 'in your eyes' gap.
I'm interested in how you evaluate quantized models against each other; haven't found a benchmark I love for that. I love this example about 27B debugging. I've seen similar success after I got a Mac with 4x memory; and Qwen 35B A3B all of a sudden is doing a great job (the 9B on my laptop wasn't great to say the least).
A lot of comments here are about the issues with the analysis in OP’s post but much of them are “a distinction without a difference” with respect to the broader conclusion. When we look at purely cost and performance (setting aside privacy) then it’s better for individual devs to pay for hosted then for self hosting. Employers are paying for tokens on the job and most devs are finding the $PREFERRED_PROVIDER’s $20/$100/$200/month subscription sufficient outside of work. Most devs don’t fall in the conditions under which running local models make sense purely on the basis of cost vs performance.
More critically, in practice, setting up local models seems more like a hobby, an educational exercise, or an act of privacy control than it is for cost cutting or productivity.
The model makers, mainframe dream of computer’s isn’t coming back no matter what OpenAI, Google, Anthropic or Microsoft want, there are too many smart tech barbarians at the gate that want in and they’re not going to be satisfied to go back to the computer terminal era.
Personal computers eliminated an earlier terminal era, and most if not all of those companies are gone except for IBM and a few stragglers and they are a shell of their former selves.
The author only compared output token costs -- but for typical agentic workloads, input tokens dominate the costs by a large margin. Running inference locally, input tokens are, to first order, free. (They only generate implicit costs through higher time-to-first-token, higher power use, and lower token output speed).
Even ignoring superior caching on a local setup, Mac hardware can often process input token around 10x as quickly as they produce output tokens. Openrouter seems to have only a 2x difference on the same
models.
For larger contexts (eg. 20,000+ token agent workflows), being 10x faster still isn't enough. You have to be close to ~100x faster at crunching contexts for it to feel like realtime.
I've dug into this previously for one simple reason: NVidia segments the market by capping VRAM and Apple silicon uses a shared memory model that could challenge that but it currently doesn't. And I really wonder if Apple realizes the potential of what they have or if they even care.
So, for comparison, a 5090 has 32GB of VRAM and you can get one for ~$3000 maybe. To go beyond that memory with current generation (ie Blackwell) GPUs, you have to go to the RTX 6000 Pro w/ 96GB of VRAM and that's almost $10,000 for the GPU by itself. Beyond that you're in the H100/H200 GPUs and you're talking much bigger money.
Part of the problem here is the author is looking at laptops. That's the only place you'll find the M5 Max currently. The real problem here is that the Mac Studios haven't been updated in almost 2 years. There were configs of those with 256/512GB of RAM but they've been discontinued, possibly because of the RAM shortage and possibly because of they're reaching EOL. Apple hasn't said why. They never do.
Many expect M5 Ultra Mac Studios in Q3 and the M5 Ultra may well have >1TB/s of memory bandwidth (for comparison, the 5090 is 1.8TB/s). Memory bandwidth isn't the only issue. A 5090 will still have more compute power (most likely) but being able to run large models without going to a $10k+ GPU could be huge.
But yes, it's hard to compete with the scales and discounted electricity of a data center. Even H200 compute hours are kinda cheap if you consider the capital cost of what you're using.
I've looked into getting a 128GB M5 Max 16" MBP. That retails for $6k. You might be able to get it for $5400. But I don't think the value proposition is quite there yet. It's close though.
Mmmm, nope if you do the smart thing. MacBook M5 max 128gb is a premium laptop at 6k, but with it you can do many things and is your good main driver for the day. Then, it can also run DeepSeek V4 flash and perform non trivial tasks locally, without censorship or limitations, even without an internet connection and on very privacy sensitive data. That's a good deal. If you buy 25k for a dual Mac Studio 512gb to abandon OpenAI and company you are going to be disappointed by both performance and cost.
The smart thing is to get a ~48gb MacBook and use it as your daily driver, and then budget ~$800/year for AI subscriptions or tokens and you'll end up at the same price.
I say, as the author of the blog post, writing this on a MacBook M5 max 128gb..
I agree with you, practically. But there is another angle of the story: for instance models are starting to be useless to do security stuff, since they are every day more censored. Also prices skyrocketed in the latest months, what will happen later? A few months ago I was shocked people resisted to spend 20$/month to get basically free frontier models, and I warned we were headed to house monthly rent figures in the future as AI becomes more and more required to do work. So indeed what you say is absolutely true now (but $800/year is not accurate: you need 20x accounts to do real work in my experience, so $200 * 12 = 2400$/year). But if you have a 128GB MacBook, that no longer looks so costly compared to 2400$/year of frontier models, you can experience uncensored LLMs, a quick thing that always works to do low-value work like TLDR this blog post for me, or what's wrong in this function? Or could you explain me this API? And for this kind of work, DeepSeek v4 Flash looks basically frontier. So if you look at things in perspective, they could have a different shape.
"starting to be useless to do security stuff, since they are every day more censored"
I have not observed this at all, If anything the big 3 are getting both better and more helpful in this area. And if you pass the security checks Antropic will give you Limited access to Mythos.
Yea my m4 max with 128gb has ended up making a lot of sense for me. I do video editing, I train ml models, I run large open AI models, I do 3d modeling, rendering and cad work. I never do all of this 100% of the time, I’ll setup a ml training to run over night and check results in the morning, during work I’ll set it up as a server and run local models, on my own time I’ll edit video and work on 3d modeling. It’s an incredibly versatile machine - and all of this is done while keeping your data on your device and giving you full control over your workflows.
Don't tell the HN crowd, but you can run some of these models on a $200 rpi5 or a $500 AMD mini PCs.
Another open secret is that that certain companies give you tens of thousands of tokens freely, with pretty respectable models such as Gemini 3.1 and GLM 4.6.
OP is comparing against Gemma everywhere but concludes paying Anthropic make more sense. Anthropic is $15 per million output token which is 30-35x more expensive even in openrouter .
This is like comparing e-bike at home with e-bike rental and concluding therefore we need to rent Toyota since it can go at similar speeds. Getting tired of bad posts getting much attention .
Article is seriously wrong, because it makes a huge mistake in the last part. You can't simply look at the produced tokens and that is your cost. In agentic coding there are lots of turns meaning you not only pay for the output tokens you also pay for all the input tokens sent each time (even if a lot cheaper, like 10x when cached). So this calculation does not accurately represent the api cost at all.
Second thing is you can starkly upgrade the token generation locally if you use agent teams. Single conversations are memory bandwidth bound and don't fully make use of your compute. If you can batch tokens from multiple agents you can easily 5x token generation.
Also predictability, resilience, sovereignty. I'm not worried about other people's outages, that unexpected demand will impact me at an inconvenient time, that someone's watering down my model, that my costs will change unpredictably or that some unforseen error will lead to a huge bill.
It's in the same category as rooftop solar for me. It doesn't have to make strict economic sense if you're the particular type of person who gets peace of mind from control of infrastructure / reduced dependency.
The amount of FUD and notion that hardware depreciates in this manner is widely held. I blame Michael Burry of the Big Short who is perpetuating these lies to the investor community today.
There's a bunch of retro hardware which should make people pause and realize they're stupid to assume hardware slows down on average even 5% 20 years later (it's probably closer to 2% and I'm being generous).
HVAC/power delivery and generation are the major factors, and if you didn't skimp/get defective parts for this and replace failed moving parts (usually fans), your hardware is basically the same 20 years down the line as it was today.
Also using LLMs locally doesn't even induce sustained 100% GPU usage over significant periods of time for most real (agentic coding in OpenCode) use-cases.
There are tons of things that can start failing on hardware. I don't realistically expect some LLM usage to materially reduce the lifespan of the laptop, but running it 24/7 for AI usage makes me think that I'm more likely to get 3 years out of the device instead of 10.
As stated in the analysis, thousands of dollars. That said, the smart thing to do is target smaller models (few billion parameters) and then use larger models for non-privacy tasks.
The true advantage of locally self-hostable, open weight models isn't about monetary cost at all, it's about the CIA triad.
Running locally, you get confidentiality of knowing your tokens are only ever being processed by your own hardware. You get the integrity of knowing your model isn't being secretly or silently quantized differently behind the scenes, or having it's weights updated in ways you don't want. And you get the availability of never having to worry about an API outage, or even an internet outage, for local inference capacity.
And this isn't even starting to address the whole added world of features and tunability you get when you control the inference stack. Sampling parameters, caching mechanisms, interpretability etc.
OpenRouter may be cheaper than frontier labs, but you still lose all of these benefits from open weight models the moment you decide to rely on someone else's hardware for your processing.
Will this cost structure always be this way and are there other benefits to not running your LLM on the cloud?
E.g.
Privacy
Uptime
Future cost structure controls
This is a field that has moved very quickly. And it has moved in a direction to try to trap users into certain habits. But these habits might not best align with what best benefits end users today or some time in the future.
In my testing, qwen-3.6-27b in full precision is well below sonnet, but above claude haiku in coding tasks. Gemma is not even close to qwen, it’s much, much worse.
How do you test? I made this comment elsewhere... but I don't see a good benchmark that covers "how good is this thing at actually driving coding with tool use locally"?
Everything is currently heavily subsidized. If the AI companies don't improve efficiency, they'll eventually have to start charging what it actually costs to offer the service, which is a multiple of what the currently charge.
I expect self-hosted to be quite competitive pretty soon. Github Copilot is already wildly more expensive than it was last month. People are going from spending a few bucks to a few thousand for that same usage. So, if it doesn't get a lot more efficient (like 3x the tokens, or more, from the same infrastructure), the prices will have to go up quite a lot to keep the lights on. Everything in AI is running partly on investors money, everyone is trying to buy a monopoly and insurmountable lead and some way to lock people into a specific model and ecosystem, but so far that hasn't happened (except for people who voluntarily lock themselves into a specific ecosystem, but even in those cases, it's usually easy to get the AI to help move to another, there are no truly unique features in AI that at least one, and probably three or four, other players don't also offer).
I suppose folks here already know this but it deserves a mention: subscription pricing is 10-20x cheaper than API pricing at Anthropic for example and it will be a far better experience (better models, faster responses, as much parallelism as you want, etc) so if it works for you there's no economic argument to buy a machine for inference at the moment.
I don't hear people debating which is cheaper, local or cloud run models. The conversation, at least what I hear, is a lot of the time users are not utilizing an awful lot of tickets all the time, those providers will be paid if you never use them. If 80% - 90% of the work I and my team are doing with Ai is grunt work, write tests for this, implement a FFT here, write the dB query for X. Nothing exhausting. Those who are using AI for whole cloth "vibe coded" applications and services are definitely better suited to cloud.
If a work laptop can run my local models and get my works needed performance for development, why wouldn't I as a company prefer that?
Add to that the privacy improvements and data protection and potentially further specific inferance if needed it's a no brainer.
Again, Ai is a tool, and the right tool for the job, I would wager with no evidence looked up, is that the majority of Devs would be happy with 10-30 per second locally.
This is not surprising at all. The biggest benefit of cloud model in terms of energy efficiency is that when running more than 1 requests, the power consumption of said GPU roughly stayed the same. The more concurrency requests the server can handle, the less power each request consume. The server GPU is already likely more energy efficient than local GPU, concurrency make the cost structure unbeatable by local hardware. It is generally assumed the local hardware only run 1 request, but if the local engine is meant to serve a small business with meaningful concurrency, the economy might still work out.
It should not at all be surprising that running models at home is more expensive than commodity providers. That's just generally true of running your own stuff. Even if the cost in money isn't higher, the cost in time is often _significantly_ higher.
This is why the idea that the AI labs are in trouble because inference will be a commodity is _completely backwards_. Some of the largest and most powerful companies in the world sell commodities. They compete on scale and efficiency, and you are never going to be able to compete with the big labs on either.
> "run a model like Gemma 4 31b, which is almost anthropic sonnet levels of performance"
I wish people stopped deluding themselves — I regularly try (and benchmark for my purposes) local models and they are NOWHERE near the huge models like Sonnet or Opus. Nowhere. Yes, you can sometimes get plausibly-looking output for simple tasks, but for anything even remotely requiring thinking there is simply no comparison.
Local models are useful. I use them for spam filtering, and soon intend to use them for image tagging and OCR. But let's stop saying they can get us "anthropic sonnet levels of performance", because that's just not true.
It all depends on use case. A local fine tuned model on a very specific use case can definitely out perform a much bigger cloud model that doesn’t have the training on your use case. But, that requires looking at the ai models as a means to end and not a Swiss Army knife that can do it all.
Even if a Mac mini at home was slightly cheaper per token I still use OpenRouter because I want to out source the heat generation and noise to a datacenter.
So I did the India-specific analysis for a tier-3 city. Here, electricity costs 1/3rd of the US version, and you also get solar subsidy up to a certain amount.
But, if we assume ZERO hardware deprecation (not realistic), then local inference becomes super cheap.. roughly, 90%+ cheaper.
Third case: the break-even happens only if we can get at the very very very least, 8.7 years of useful hardware life. A more realistic number, however, when working 8 hrs/day and not of 24 hrs/day, is around 25 years.
So, for now, local inference is preferable if you deeply care about privacy. From cost perspective, it's still not there.
not much of an analysis really.. just simple math... anyways, that markdown share site takes around 5-10 seconds to load the page.. so, just hang on a bit more time :)
> Throwing money at anthropic makes more sense in this context.
But you are dependent on them, which is the biggest factor IMO, there was a website posted here before of people getting banned from using it over silly reasons, not to mention price hikes, or privacy concerns. Maybe now it’s more expensive or slower to run locally, but you are in full control of everything.
I run the latest 20b-30b models on a MacBook Air... running inference with an MoE (25 tps) for like 2 hours is like 10% battery.. (look me up on huggingface to download my models)
also you gotta realize frontier models have massive "system prompts" that clog up the context window with garbage.
being able to write your own system prompts gives you a MASSIVE edge..
But I want there to be an incentive to develop smaller, private models that can run on local machines that I own. I want that business model to win.
If it costs me slightly more in the short term, but I don't depend on any providers' price gouging as soon as I fully depend on them, I will consider it a win.
Author forgot that after 3 years when hardware no longer decent for inference you can still resell it for 25-50% of price.
Obviously if RAM apocalypse passes by then high-end configurations preserve resale value worse than base models, but still it's hefty bonus of Apple hardware that might change math a lot.
What is the security of OpenRouter? I have a feeling user has no idea where their data is going and how it will be used or am I wrong?
When I see so many options, that looks like it would take months to audit whether it actually makes sense and is safe to use. But I guess some people are fine with YOLO-ing it.
One important difference is that costs are bounded on your own machine. Like with cloud providers, I'm always worried that cost may accidentally explode if I launch an agent swarm wrong.
Now, it looks like the providers I use have good limits. But I do worry about this.
And when the heavy subsidies of cloud AI come to an end? Plus the Macbook can do other tasks as well; presumably you need a computer anyway so should deduct the value of a computer for non-AI tasks from the cost.
This doesn't compare like for like, since its comparing the total cost for the local machine with the usage cost for the cloud service, despite the cloud service also needing a local machine to be useful.
For me, the appeal of local compute is first and foremost confidentiality and having the possibility to run my 200K documents through an LLM just to see what happen without having to consider the cost.
Regardless, I see lots of people using local AI as an excuse to buy over provisioned macs and AMD Ryzen machines. Then not using it at all for local AI.
What would really elevate an article like this is if we could somehow quantify human brain’s equivalent outputs and compare the costs with local LLM and cloud LLMs.
OpenRouter doesn't expose all the LLM sampling parameters/research that llamacpp, vllm, sglang, et al expose (so no high temperature/highly diverse outputs). Also OpenRouter doesn't let you use steering vectors or LoRA or other personalization techniques per-request. Also no true guarantees of ZDR/privacy/data sovereignty.
Oh, and the author didn't mention at all anything related to inference optimization, so no idea if they even know about or enabled things like speculative decoding, optimized attention backends, quantization, etc.
At least AI slop would have hit on far more of the things I listed above. This is worse-than-AI.
> OpenRouter has Gemma4 31b at ~38-50 cents per million tokens. This means that on the optimistic side (50 watts, 40 tokens per second, and 10 years) the pro max is as cheap as openrouter. On the pessimistic side (100 watts and 3 years at 10 tokens per second) the pro max is 10x the cost. I think ~3x the cost per million tokens is likely the right number for local inference on the pro max from an accounting perspective.
Apart from that, like detailed in the the article, pricing for local compute also depends on electricity prices.
By the way, I don't want to snark about it, my English is not very good, but it's "per se", not "per say". Just commenting on this petty thing because it seems to be a common misspelling, and it always trips me up a bit. Makes me wonder about another supposed meaning like "from hearsay".
> At ~50-100 watts and $0.18/kWh that's $0.009 or $0.018 per hour. $0.02 per hour. $0.48 cents per day for the electricity to be running inference at 100%.
Except I already have a local Mac to run Xcode. OpenRouter cannot help with that, at any price.
> 64 gigs should run a model like Gemma 4 31b
No, it can run anything in the 70B range. It's a notable quality upgrade from the 30B, which isn't obvious because the famous flurry of April releases didn't contain any 70Bs.
It can also run 120B in UD-Q3. Or 230B disk-streamed.
I'm even surprised people ignorantly talking about advantages of buying very expensive device , run it only sometimes and aiming to beat cloud vendors.
If small model is great it will be hosted with good electricity cost and will be utilized 24/7.
Isn't it 2+2 of economics ?
CPU is a commodity, and we are still buying cpu and ram from vendors for same reason
Put a cost on sending your intellectual property to a saas provider who knows where. Half a problem when it is just your IP, hopefully not the IP of your clients. Maybe if one is building yet another html nobody really cares about.
It's a good thing the market is adjusting to the reality that no one needs to own powerful computers, just terminals into the feed of compute available through the cloud, then! Nothing could possibly go wrong from that!
You can’t actually - due to the RAM shortage you can’t even upgrade to an M3 Ultra Mac Studio with 128GB RAM. That model has been discontinued. Even the 96GB model has a wait time of 5 months in most locations. This is the reason why the resale value is so high.
I think that the main flaw in the reasoning is assuming that cost of token will stay the same over the years.
Chances are that token prices will go down, but chances also are that the AI bubble pops and all of a sudden all these companies will either have to make a buck out of the inference or go bankrupt.
Getting your own hardware just grants you stable pricing.
if all those companies go bankrupt, you could also buy their hardware on the cheap.
its almost guaranteed imo that as the model quality evens out, inference cost will drop towards 0 at insane speeds, given how well llama works as an asic.
Now include the externalized cost in the U.S. of deploying ~100% of productive capital to build data centers instead of, for example, first-world transportation infrastructure, and tell me which one is cheaper
Can’t reply to the reply here, but yes, you do pay for it with money. Absorbing all of America’s capital and construction labor capacity to build AI data centers, rather than, for example, reaching parity with every other developed country in transportation infrastructure, is a cost you pay every day in gas and time spent in traffic. More to the point, building AI datacenters without paying for modernizing the grid is raising electricity rates. Token costs aren’t just subsidized by VC money, they are subsidized by all of us because of idiotic policy choices. Lots of people have pointed out that the current price per token is heavily subsidized, and a fair analysis would account for that.
Bingo. I, for one, am loving this phase of enjoying the LLMs at the expense of VC money. Just like how I enjoyed cheap rides and deliveries on Uber. And with the fragmentation in the field, I don't see a monopoly coming up.
When the AI bubble inevitably pops, the author will find a new way to skew results in favor of cloud LLMs. Like including the price of a desk and a chair in the local token cost.
The full-amortization framing is doing a lot of work here. I bought my laptop because I needed a laptop, not as an inference box, and running a model on it is incidental to that. Once the hardware is sunk for other reasons, the only cost left is electricity plus whatever depreciation you accelerate by hammering the SoC, which the post actually acknowledges in one parenthetical before allocating the full $4299 to tokens anyway.
Also nobody I know picks local over OpenRouter on price. They pick it for offline, for data not leaving the machine, for no rate limits, for not having a provider go down mid-task. If $/Mtok is the only axis, sure, cloud wins.
In practice the pattern I see is leaving a small model running on easy background tasks while using the laptop normally, not a dedicated inference box hammered flat out for 5 years.
This isn't a good analysis, and it's because it keeps rounding everything up. He rounds up the cost of electricity by 10%. He has a range of power use, takes the high end (which is 2x the low end) and multiplies it by the inflated electricity cost.
But then they talk about using a newly purchased Mac to do the inference, running at full capacity, 24/7. Why would you do that? Apple silicon is fast but the author points out: you're only getting 10-40 tokens per second. It's not bad, but it's not meant for this!
It's comparing apples to oranges. Yeah, data centers don't pay residential electricity rates. Data centers use chips that are power efficient. Data centers use chips that aren't designed to be a Mac.
Apple silicon works out pretty good if you're not burning tokens 24/7/365 and you're not buying hardware specifically to do it. I use my Mac Studio a few times a week for things that I need it for, but I can run ollama on it over the tailnet "for free". The economics work when I'm not trying to make my Mac Studio behave like a H100 cluster with liquid cooling. Which should come as no surprise to anyone: more tokens per watt on hardware that's multi tenant with cheap electricity will pretty much always win.
Rounding everything down in the most optimistic setting got me to $0.40 per million tokens, and openrouter has the same model at $.38/mtok.
But once all that is done you still own a Mac in one case, and you don’t in the other, correct?
Even at just the electricity cost openrouter will be both
1) Roughly break-even to a little bit cheaper per token cost 2) Much, much, faster
So the cost of the mac barely even matters, it's just an extra cost beyond.
Sure, data center providers can pay lower rates.
The point of this article is that LLMs at home really don't make a ton of sense, unless you are willing to pay through the nose for privacy. There is absolutely no cost saving to be had.
If you're looking at your own datacenter as a larger corporate client, that could change.
There are also some providers that will contractually keep your data private, like AWS Bedrock or parts of Google/Azure (I don't know their stack names).
AWS even has AWS Secret Region and AWS Top Secret Region if you want to use LLMs on classified data.
You have to value privacy at a roughly absurd level to not want to use LLMs run efficiently at scale by someone else. For the home user, just the extra efficiency produced by batching requests from a large number of users in a datacenter in a real win.
Some of these companies are even selling tokens below cost to get marketshare. If someone will sell you a service for a dollar bill or three quarters, why wouldn't you take the three quarters?
> If someone will sell you a service for a dollar bill or three quarters, why wouldn't you take the three quarters?
Because one day they'll send you an email informing you the new rate is $1.50, and if you missed the email, that's not their problem.
Plus your privacy.
Minus opportunity cost and depreciation
It can get quite complex, for me the feeling of "I built this", and then watch as it makes stuff from the power generated on my solar panels is also worth a lot to me. The sheer coolness of it.
Not always. The calculations take its useful life expectancy as an input. If they estimate it correctly you have highly likelihood of it breaking, burning out or being woefully out of date by the end. At the 10 year window you are looking at losing support for security updates.
So if you are lucky you might end up with something that still runs but most folks won't find it particularly useful
Yea this; it’s the same reason why mortgaging is cheaper than renting
This is far from a universal truth: https://www.nytimes.com/interactive/2024/upshot/buy-rent-cal...
Real estate is only a clearly good investment if you ignore opportunity cost.
Real estate is generally a "good" investment as it's considered a relatively safe way to get significant leverage. 5x leverage in the case of a 20% deposit, or even up to 20x leverage with countries that allow for 5% deposits (New Zealand).
In addition, the interest payments almost always end up being near the rent the owner would have paid, so mortgage payments are higher, but that increase is generally (and quickly becomes) principal while being able to counteract inflation of rent.
> Real estate is generally a "good" investment as it's considered a relatively safe way to get significant leverage
Leverage? People don't normally invest in property (normally involves taking out a loan) for the purpose of taking out another loan. That so called "leverage" is being used to buy the house...ie you don't have any leverage
The leverage is the loan taken for the mortgage. If you have a $1M property, $900k loan. If the property's value increases by 5%, that's $1.05M, so you've made 50% returns on your $100k capital invested. That's leverage, the leveraging of $100k to get the returns of $1M asset.
> That's leverage, the leveraging of $100k to get the returns of $1M asset.
Obviously. But that's not leveraging real estate, that's just leveraging cash.
Leveraging real estate would be using the property as collateral for a loan larger than the property itself
Isn't it incredibly obvious in my first message that the leverage is on your deposit, and therefore leveraging the cash.
Nah cos you said real estate is a good way to get leverage...
If I own a house outright, and use it as collateral for a loan... this my friend, is "using real estate for leverage"
Going to the bank and asking for money so you can do the above is NOT "using real estate for leverage"... It's using your cash deposit for leverage
We can estimate this. US median home price right before the crash in 2007 was $240,000. Today, it is about $400,000. Median rent in 2007 was $810. Today, it is $1,698. There's some simplifying assumptions we have no choice but to make. Let's say renter's insurance is negligible enough to ignore. Meanwhile, we'll just let an online mortgage calculator assume a median $50,000 home insurance coverage payment and bake it in. We'll assume 1.1% of assessed value for property insurance, which is currently the US national average (it varies a lot state to state in reality). We'll assume an FHA loan with 4% down.
This gives us a $1,995 a month payment when we purchased and a $2,142 a month payment today, due to higher assessed value for the tax.
We can see upsides and downsides in both cases. Rent would have been quite a bit cheaper in 2007, but it has very nearly caught up by now. Meanwhile, you're probably talking about renting maybe a 2 bed/1 bath apartment, whereas the median single-family house is more like 4 beds/2 baths, with a yard. Whether or not that extra space and privacy matters to you likely depends a lot on whether you're single or have or ever plan to have a family. You could have invested into something like the S&P 500, which has historically returned about 10.5% since 1957 annually in nominal returns. Let's just kind of naively split the difference here and assume you can invest $1,000 saved on rent versus mortage a month for the first 10 years and $200 a month for the next 9. That would have gotten you somewhere around $240,000 by now. Meanwhile, you're looking at about $248,000 in home equity by now for the purchase case.
Choose different parameters if you please, but I'm not really seeing the case for renting here over the long term, and that's in spite of choosing the single worst time in the last century you could have made the purchase.
Oh I don't disagree, I hate real estate as an investment, it's a terrible asset only made "viable" by tax benefits, rent-replacement and excessive amounts of risk via leverage.
What is "rent-replacement"? I never heard that term before. I cannot find anything obvious using Google search.
It's my shortcut for describing the idea that you'd spend $X a month on either interest lost to a bank or rent lost to a landlord, and therefore you can mostly consider that a constant expense when considering rent versus mortgage.
You also need to pay close attention to rent vs purchase ratios. A lot of cities are cheap to rent but expensive to buy (eg beijing 10 years ago).
Key word being „ago“.
I’m covering my bases because Chinese real estate has been volatile recently and I’m not sure where the market is at now. It could be that renting is still way cheaper than buying, I just don’t have any direct experience to back that up. If I bought while I was living in Beijing I would probably be underwater with my investment right now, renting for 9 years was the right call and my rent was pretty affordable anyways.
Such cities still exist and have been in such a state for decades. They can change but that's meaningless as they can also change the other way around.
Articles like that still miss a bit of the nuance. Imagine having your house paid for, and you grow old and you have no rent to pay. Yes, you could have invested but likely you would have spent some of that money on something else, or your investments might have not worked out so well, or any other reason. Human reasons, to be specific. Owning property is like a lock.
Imagine having your house paid for, and you grow old and you have no rent to pay.
My home is "paid for". Except for the HOA and property taxes that are not that far off from what I was previously paying in rent, the ongoing maintenance costs with random large spikes, and the opportunity cost of having a large chunk of money in the house and not in the market. It was still probably the right decision, but it's not at all a free lunch.
Surely though, the HOA and all that would likely be baked into a renter's price.
And you didn't need to go live in a HOA. I don't, and it's much cheaper.
Surely though, the HOA and all that would likely be baked into a renter's price.
Sure, the same way that the benefits of a fixed mortgage payment are baked into sale prices. The efficient market hypothesis would say that neither renting nor buying should be obviously superior in the long term, because if either was then people would bid up rents/prices until it wasn't.
And you didn't need to go live in a HOA
I pretty much did, unless I wanted to significantly compromise on other factors.
> The efficient market hypothesis would say that neither renting nor buying should be obviously superior in the long term, because if either was then people would bid up rents/prices until it wasn't.
Buying have much higher entry point, need a bunch of cash at start then a ton of paperwork.
It is absolutely possible that local buying market is inflated precisely because the area is so desirable buying to rent is (or was) good investment, but that's rarely is true for a bigger market
And it's gonna be interesting wherever this narrative will shift over the next 5 yrs
I keep hearing that properties are in the biggest bubble yet in the USA - with the affordable housing shortage being a red herring, because real estate managers and boomers are unwilling/unable to reduce their prices - despite not getting renters/buyers because it would kick off a death spiral as their interests would consequently go up (because of lower security). Along with the ai layoffs etc
I'm not American so I only hear the occasional interview so don't have any idea if it's really as pressing as these industry professionals keep saying but I'm definitely at the edge of my seat watching...
You and Matt Levine would get along: https://news.bloomberglaw.com/banking-law/matt-levines-money...
It is very close to universal truth, aside some small areas with very warped market.
Even if you move out after 5 years, you still own the place and can rent it out and then it pays for itself, to skip the cost of selling it back to market
It never fails, there's always someone who trots this thing out. We had bought our house, and then had to move and decided to rent. I was APPALLED that they wanted me to fill out an APPLICATION form, where they would decide my worth, and let me know if we would be allowed to live there. When buying a house, my cash was as good as anyone elses'. And then the management company would come inside my house to inspect that I wasn't running a meth lab or something. Thankfully that only lasted two years. I will never rent again. Majority owner-occupied neighborhoods have different characteristics as well.
> I was APPALLED that they wanted me to fill out an APPLICATION form, where they would decide my worth, and let me know if we would be allowed to live there. When buying a house, my cash was as good as anyone elses'.
House sellers receive offers from buyers, sometimes including letters, and can choose to sell to any of them (or none of them), whether or not those offers are higher than the listed price. It's not so different.
> And then the management company would come inside my house to inspect that I wasn't running a meth lab or something.
Yeah that part is different. I also prefer owning.
Why would a house seller accept any offer that not the highest total price?
A seller might prefer a cash offer to an offer contingent on the buyer securing credit (credit might fall through). Or, like landlords, a seller might prefer a buyer with a higher credit score (same reason -- buyer is more likely to be able to secure credit and close the deal). A seller might prefer a slightly lower total offer with a serious amount of "earnest money" over a slightly higher total offer without significant earnest money (buyer might try to back out). A seller might prefer to sell to a family with a nice story in their buyer's letter than someone buying a 2nd or 3rd house. Or the seller may think all offers are too low and they can hold out and get a better offer later.
Real estate transactions are not very straightforward. A seller might accept a lower offer if they feel the buyer is more likely to be able to go through with the transaction or just go through it faster. And sometimes they do just plain like one buyer better.
Except one day the hype will catch up to reality that was always true, people will realize their $20,000 Mac is has less utility as a "way to learn AI" than some kids 3090 fortnite machine, and it'll be back to below MSRP.
Where is a Mac above MSRP?
I’ll keep my data local over a $.02/mtok difference.
It’s more than just data locality. OpenRouter is faster, no? I have an M4 pro, and anything but the smallest dumbest models are unusably slow for interactive use. I personally haven’t yet found a good use case for offline/non-interactive LLM work locally.
Yeah. The speed is the biggest issue. The intelligence of open models is good enough for serious work (though still worse than the frontier models), but the cloud models are often 3-7 times faster, and you can get more parallelization and so get speeds on the order of hundreds of tokens per second, which makes things fast!
Even extremely slow LLMs can generate Part B faster than I can audit Part A. So the LLM can generate Part A while I look over my email. Then it can worry over Part B while I look over Part A.
It can worry over Part C while I have my 10:30 group meet. And it can worry over Part D while I do whatever other silly, time-wasting thing all humans do in almost all organizations. Then I still haven't reviewed Part B, yet, so the extremely slow AI is waiting on me.
Maybe someday I'll be good enough to need faster AI so I can rewrite something like Bun in a few days. Right now, slow and local fits my use case very well.
I don’t think it matters if you’re “good enough” or not. Much of AI development is iterative. If you context switch between A from project 1 to B from project 2 back to check A, then maybe C while B finishes up, you will lose the flow state that AI assistance can enable with speed for those who are not fluent coders.
Sure, I can wait hours for my local model to finish, or I can spend basically as much and get the answer right away
There’s a lot of exciting stuff with local LLMs despite the speed, but for me I don’t have the discipline and working memory to jump from project to project.
> Even extremely slow LLMs can generate Part B faster than I can audit Part A.
Depends what part a and b are...because workloads are different
And continuing the argument of "more than just...", if you stopped inferencing on your Mac you still have a generally nice computer. The difference between rent vs buy.
I played with classifying and summarizing my entire email history (per email) with small models, but that only took about 12h of GPU time at most. Using a coding agent cli wrapper in that case is far slower because of all the spin up cost and the system prompt they inject even if you want to turn it all off.
If I used an actual direct API it probably would've been much faster, but I'm doing it for hobby / fun reasons. You also get to fiddle with a lot more params.
I’m running a local Whisper + Gemma 4 pipeline with a cheap USB mic to extract health related data and potential todos from ambient speech. It doesn’t have to be fast doesn’t have to be 100% correct because if it captures at least a few bits of interesting information that would otherwise go unnoticed it’s still a win.
I run whisper through openwebui to gemma4 moe and use kokoro TTS back to me.
I use a 5060ti 16gb and a minipc.
I tunnel in via Tailscale and access it with my phone or laptop from anywhere. It’s pretty good and will only get better as I optimize.
What is it with AI SaaS naming themselves "openxyz" when there is 0% open about them?
They learnt from ooenai that naming yourself open-xyz doesn't actually require opening anything.
It's the next co-opted buzzword after "democratize".
Yeah, anytime that I see "People's" or "Democratic" in the name of a nation, I grow suspicious. It is rarely a well-functioning democracy.
It's how marketing works. If something is a problem they have to loudly claim to have fixed it. Look around the economy and you'll see lots of it. "Healthy" (high sugar) muesli bars, clean-diesel, surveillance wrapped up as keeping us safe. The modus operandi of marketing is to change minds about self evident things otherwise what is the point?
Also many have power even cheaper or even free unused surplus power with solar.
I don't do local inference other than hobby & learning reasons because electricity is so expensive where I am at.
The article makes no sense. I can't use OpenRouter as a general purpose computing device. Why are we comparing a whole computer to a single purpose SaaS?
They're responding to the people doing things like buying the most expensive Mac they can find specifically to do local inference for their AI agents.
Some do it to have control over their ability to use AI. Some do it because they think it will be cheaper to not have to pay a SaaS to generate tokens for them.
But for those interested in the latter case, it seems like it's not actually cheaper after all, at least at current prices. But then I don't expect prices to drastically jump because of how much competition there is in model development.
It's worth paying a premium for the privacy (assuming that llama.cpp and ollama aren't sending my sessions back to the cloud regardless...), and for the concerns about not getting a surprise bill.
> not getting a surprise bill.
Correct me if I'm wrong, but I believe this is a feature that only Google has figured out how to implement. All of the other pay-as-you-go token services have a cap you can set, some by monthly spending, some with API key resolution, others by how much you put into the account. I use many, and if configured with auto-purchase disabled, it's not possible to have a "surprise" bill (except for Google!)
You also have control over your costs. It is reasonable to assume that tokens will cost significantly more in the near to medium future as the market consolidates and subsidies decline.
Google, Microsoft, Meta, Anthropic, OpenAI, Oracle and others are going to be looking to recoup all the money that they’ve spent to date. Why would the price go down in the future?
> Why would the price go down in the future?
Because price is driven mainly by competition, not by a desire to recoup prior spending.
Investors aren't doing things out of the sheer goodness of their hearts, so if they could just bump the price up they'd have already forced it up. The very existence of workable local models puts a cap on how high the price can realistically go, but the high level of competition still extant makes the price floor ever closer to the actual cost to generate tokens.
The AI numbers are huge, but I remember similar arguments about residential high-speed internet. According to Gemini, the "price for internet" is down 12% in real terms (ugh, capitalism!), while speeds are staggeringly faster.
The providers have spent a fortune on wireless, pulled a lot of fiber/cable, and it's cheaper than it was when it started.
No, that’s not the point. I think this is to help people who are thinking about getting a beefier Mac so they can run their LLMs on it too. Some in particular want a dedicated Mac Mini or Studio for this purpose. The breakdown, even if slightly flawed, offers a good insight into the economics of it.
For most people, they might be better off with OpenRouter models and providers supporting Zero Data Retention. On the cloud, that’s as good as it gets for privacy - your data is never retained beyond the life of the request.
> your data is never retained beyond the life of the request.
Like with OpenAI for a year?
” In June 2025, the court ordered OpenAI to retain its consumer and API customer chat logs indefinitely, including any that had been deleted, so they could be investigated […]”
https://www.techspot.com/news/109839-openai-no-longer-requir...
[dead]
I think it's because there are a lot of people writing articles about the benefits of running local models. I think it's fair to say that there are daily threads on HN singing the praises or local inference. I also see people buying new hardware where the main trigger is ability to run local models.
But the people who want to do local inference are putting some amount of value on privacy that’s not captured by the raw monetary value so just comparing the price is somewhat beside the point, it’s also true that, if you have eg a Mac and you use that as your main computing device then you would have spent money on it anyway, so you can’t even really compare its value to spend on something that’s not general purpose.
That's a lot of assumptions. I think there are also people buying new hardware specifically for this purpose, and their motivation to do it is thinking it will be cheaper in the long run. Privacy is not necessarily the motivation.
My overall opinion is that the smart thing is not to upgrade to the maximum memory for AI purposes. It's worth quantifying how much extra we pay for privacy.
I replied to a comment asking why the article exists.
As for privacy, I'm sure there are many people that are not so interested in that aspect.
using it 24/7 brings the average cost down, not up.
the less you use local LLM, the less sense it makes since you paid a lot for hardware you don't use
That's the point: why would you buy a device that's specifically not optimized to be used for 24/7 inference? It's expensive hardware that's not designed to be used in that situation! The power use for inference isn't especially good and you're not getting even a fraction of the benefit from the hardware that you're paying for.
> why would you buy a device that's specifically not optimized to be used for 24/7 inference
because it costs $1k-$2k instead of $10k-30k+ for optimized devices
Nobody is suggesting you buy a pair of A100s, which is what 15k gets you these days. Get a used 5090. And the author specifically priced the hardware at over 4k, which is double the 1-2k you're noting
Good question but people are doing it anyway. It's a fact that right now tons of people are buying Mac Minis specifically for this use case, to treat them as their personal data center for agents. The concept of "power use for inference" is foreign. Those people are the ones that motivated this blog post I think.
The hardware has multiple uses for the same cost. The pay-per-use server does not.
The author isn't pricing in the multiple uses. You either compare it apples to apples or you don't. If you're using the machine for general purpose computing on top of inference then the amortized hardware costs are pointless to measure. This is exactly what I said.
Ok you can resell it at the end
Not sure where 40 tokens per second is coming from. I’ve seen 95-100 tokens per second on M5 Max 128GB running Gemma 4 31B. I’ve done experiments where it is faster than Claude Opus 4.5 for the same prompts.
can you provide your configurations pls ?
It's actually a bit faster than that now it seems, about 112 tok/sec.
Configuration:
Gemma 4 31B Instruct Q6K Context size 40960 LM Studio 0.4.13+1 Metal llama.cpp v2.14.0 LM Studio MLX (Apple M5) v1.6.0
Here are my results:
prompt eval time = 32545.36 ms / 5625 tokens ( 5.79 ms per token, 172.84 tokens per second) eval time = 20227.99 ms / 310 tokens ( 65.25 ms per token, 15.33 tokens per second) total time = 52773.35 ms / 5935 tokens
This was for interacting with a local MCP service, running a tool that returns a ~20KB text file to the agent to add to the chat context.
I'm seeing about the same number of tokens/second on an M2 Ultra that I have access to (also with 128GB of memory).
This is surely apples-to-oranges to the OP results (and I don't spend a great deal of time benchmarking these things, so my methodology might be lacking), but it's interesting seeing okay performance for a top open model. For most use, however, I find Gemma 4 26B A4B (Q6K) to be good enough (esp. for MCP calling) and much much faster (~1,200 tokens/second).
Actually, figuring it on generating tokens 24/7 is the best case scenario. if you figure it at 8 hours a day of actual use, you still have the fixed cost of the hardware being the highest portion of the budget, but now you generate 1/3 the tokens so you triple that cost per token.
Rounded up, yes, and oddly inefficient for someone obsessed with inefficiency. One could buy a brand new 64gb M5 macbook for well over 4k. Another could buy a scratched up but functioning M1 Max 64gb off of ebay for a little over 1k—and somehow get the same 10-20 t/s with 31b that the author does with an M5. Or better yet, have a frontier model do the planning and judging, and have a local MOE model execute at 50 t/s. All of this achievable by a former English major with too much free time.
I have an M1 Pro, and a M4 & M5 max to play with at work and the speed difference is very significant between all 3 machines, the M1 Pro is far slower, and the M5 is significantly faster than the M4. And a windows 3090 beats all of them but eats twice the amount of power per token. This is all running the same 24GB memory friendly model with LM studio.
The real reason this comparison makes no sense is that only a vanishingly small fraction of people seriously using ai to code would seriously use a model so far from the top models (including open source ones).
He should compare his MacBook to Open Router on Kimi 2.6 1.1T or GLM 5.1 (754B), at bfloat16 precision, which he can't ofc.
But it furthers his point that things like open router are a better idea, which is not surprising.
> Yeah, data centers don't pay residential electricity rates.
There are 2 caveats here:
Some places have higher prices for industrial than residential power as residential one might be subsidied by govt.
And DC also pay for cooling, which residential will only effectively pay if they have AC and is hot outside. So power rates are some multiply of industrial pricing.
Generally you don't build a data center in a place that doesn't sell you electricity for cheap
Still for purpose of calculation you still need to calculate the price of both power and removing heat created by that power.
Boss, I make 16.50 per hour, say 15, I work 36 hours, say 35, say 500 per week, say 4 weeks per month, that's only about 2000! Don't you agree I need a raise?
We also have no idea what it actually costs Anthropic. This could be wildly subsidized and actually Apple Silicon is more cost effective.
Your post makes sense if you bought the hardware for other reasons, and maybe run models occasionally as a novelty.
That isn't the case for many, though, and there is a whole social media space where people are hyping up the latest homebrew options for running models, believing it frees them from the yoke of big AI.
Millions of people are buying big $ maxed-out hardware like the Mac Studios or DGX specifically to run LLMs. Someone rationally running the numbers is a good thing.
Let's not get ahead of ourselves. Millions, really? I can believe there are a lot of enthusiasts doing this, but "millions" needs a citation.
This is HN; it has probably never occurred to half the people here that the average person even in first world countries doesn't even have the financial capacity to make an impulse five-figure USD purchase, even if on credit.
No one said anything about this being an "impulse" purchase -- it would usually be perceived as a "career investment" purchase, given that many feel they need to race to be with "it" or be left behind -- nor does "five figure" have anything to do with this -- DGX options are available at $3000 -- there's a certain irony that you are posting a comment that is basically "people in my country/circle can't, therefore no one can", while dismissing my comment that many people can.
Millions of people are paying thousands of dollars a year to buy a slightly upgraded entertainment package in their car. There are 60 million or so millionaires alone, including 6 million+ in China.
There are a lot of people with a lot of wealth on the planet. A lot. Millions...it isn't that unfounded, friend.
So doing this "this is HN" snide jerk act, and then basically projecting your lot on the planet is...I don't know if you intended it, but it's rather amazing.
> Millions of people are buying big $ maxed-out hardware like the Mac Studios or DGX specifically to run LLMs.
What's your source for this?
Not just one, but two replies completely hung up on that. Like, why even reply when you already saw the other guy doing the same tired pedantry? Just wanted to feel like you contributed?
Ignoring that it was just tosser hyperbole (that absolutely zero reasonable people need to question), yes, enormous numbers of people are buying GPUs or hardware with the explicit goal of running local LLMs, and social media is full of people hyping various setups and models. Mac Minis are almost impossible to find, and that alone is selling at a clip of about 300,000 every four months. Large memory GPUs are basically a myth at this point. All so people can pay more to get a worse result than commercial options, which is precisely the point of the submission.
These local setups only ever make sense if you have something that confidential, or you're doing something that ToS of the majors would ban you for.
Now given this pedantry horseshit, you'll probably demand that I specifically show a citation on DGX or Studio sales, which...rofl.
Honestly, I don't even see my Macbook Pro costing me anywhere near as much as using any of these AI services, but maybe I'm just not seeing a significant increase in my power bill to notice? I am the power user who uses Claude Max pretty much all the time to prototype ideas, and build things I actually use, and has given me a lot of value, I work full time and have a family to raise and care for, my free coding time is mostly limited to ideas. Now I can draft a plan with detail, review the code, run the code, test it, and use software custom tailored to my needs.
nothing about the current data center craze looks efficient.
Whether you think building data centers or not is a good idea it's inarguable that the per-token efficiency (power, hardware, etc) is FAR higher in a data center. That's literally what it's designed for.
im talking per value. look at the efgiency of chinese open source models; then look at SOTA sucking gigawatts, then the proposals.
America is basically proposing AI using the equivalent bloatware of Windows 11.
I run two 49B parameter models on a pair of used A100s full time and it sucks down 250 watts at peak utilization. That's not gigawatts, and it's completely within the realm of comparison to what the author is describing.
Probably because lots of data centres are being built (or half-built) which are sitting idle.
If there are datacenters sitting idle right now then you could probably make a lot of money selling that capacity to Anthropic at this point...
If you have racks of idle H100s, you are doing a terrible job of running a business.
[dead]
Unless I'm misunderstanding, this is counting the entire laptop in the cost of generating tokens. The calculation seems to omit that, in addition to receiving LLM output, you have also received a laptop in exchange for your money. If you intend to put this machine in a dark corner and run it solely as a token-munching server, a laptop would be an exceptionally poor choice of technology for this purpose. But if you intend to use the laptop as a laptop, having a laptop is a pretty big benefit over not having a laptop.
You also get the benefit of privacy, freedom from censorship, and control over the model used (i.e. it will not be rugpulled on you in three months after you've built a workflow around a specific model's idiosyncrasies).
Yeah, a better metric might be, the difference in cost between the laptop you need to run local models, and the laptop you would have bought anyway.
The base 14" m5 MacBook pro is $1700 with 16gb/1tb. The author's spec is $4300 - $2600 more.
It depends on how often you use it (and your tolerance for slow inference) and whether you would have otherwise bought a higher spec. For my needs, this costs a LOT more.
> control over the model used
but you lose access to the most capable models, you can run only the small ones
Sure. I have unlimited credits for AI with my job, but I’m learning by running things locally. Start by running OpenRouter, then downloading open source tools to run locally, then modifying some python, now creating pipelines in rust. Yes, the irony is that I’m using Opus to crack these things open.
And they run slower and quantized.
> in addition to receiving LLM output, you have also received a laptop in exchange for your money
And, since it's a Mac, whenever you're ready to upgrade it'll still have a fairly decent resale value.
How is it going to have decent resale when you bought it for $15,000 over MSRP because of a time-limited LLM mass-psychosis?
https://www.ebay.com/sch/i.html?_nkw=apple+mac+studio+m3+ult...
I think you'd need to tinker quickly, realize anything with CUDA (other than the awful DGX Spark) is better for learning, the prefill is killing your ability to actually run models large enough to justify that RAM, and then cure yourself before the rest of the crowd.
https://www.apple.com/shop/buy-mac/mac-studio/m3-ultra-chip-...
https://www.apple.com/shop/buy-mac/macbook-pro/16-inch-space...
Well obviously I'm not talking about buying over MSRP.
And I am.
Given those models aren't sold at MSRP anymore and are primarily being sought for inference, it's apropos of an article on the costs of inference on Apple Silicon.
https://www.apple.com/shop/buy-mac/mac-studio/m3-ultra-chip-...
https://www.apple.com/shop/buy-mac/macbook-pro/16-inch-space...
Maybe before double posting these links, you should have considered why the eBay sold listings are for thousands over MSRP.
You can't order the Mac Studio with RAM upgrades anymore, and the demand propping up prices is for AI inference
Not sure what you mean by "can't order", both of those links show purchasable, orderable computers with RAM upgrades. You can get the Mac Studio in 12 weeks or the MBP in just a few weeks, with RAM upgrades, by ordering right now on Apple's website.
I don't know why he's being so obtuse, but Apple did discontinue the model with 512 GB of RAM, which is what's being scalped on ebay.
You, uh, can't get the RAM upgrades anymore now. But you? (I) could, when I last clicked the link.
OpenRouter can’t play Cyberpunk 2077 max setting 5K HDR!
OP is giving you the absolute best case compared to most of the people who've been overcome with psychosis hoarding Macs.
An unreasonable number of these people spent $10,000+ for Mac Studios that are still compute bottlenecked and don't have anything more efficient than Gemma 4 to run.
Frontier AI companies are selling at a loss.
Excusing everything else that u/bastawhiz said[0]; the obvious fact here is that Claude, OpenAI, Gemini et al. are quite literally burning through 100's of billions of dollars and selling it back to you for pennies on the dollar in the hopes that they get to be the only one left.
If I spend $10 growing Oranges and sell them to you for $1; then of course it's more expensive for you to do the growing.
I feel like I'm taking crazy pills. These models will become more expensive over time, it's functionally impossible for them not to, they just want to capture the market before they have to stop selling at a huge loss.
[0]: https://news.ycombinator.com/item?id=48168433
That seems unlikely. There are many providers for open models on openrouter. It seems unlikely that they are throwing money away for each token they sell.
Also, there a good technical reasons for inference being much more efficient at scale.
Sure. And there’s Lyft and Uber and plenty of others. And Grubhub and DoorDash and uber and how many others. And I don’t even fucking remember how many electric fucking scooter companies, I’m practically falling over scooters! I’m sure they aren’t earning market share by selling at a loss either.
Uber has been profitable since 2023.
And it was founded in 2009, 14 years without turning a profit
Which, by analogy, would mean you could use subsided tokens for over 10 years and then, and only then, actually purchase your own hardware to do inference, with over a decade of technological advancements to compound.
I'll invest in inference hardware whenever the economics make sense, not because of my prediction that prices will get higher (also, because smaller models keep getting better and they might just suffice for most of my use cases in a few years time).
My biggest worries in terms of cost is in regards to training. Whenever that gets too expensive for Beijing to pay, we won't be getting new SOTA open small models to run on local hardware, which, again, reinforces the decision to use providers for AI inference for now.
> Whenever that gets too expensive for Beijing to pay, we won't be getting new SOTA open small models to run on local hardware, which, again, reinforces the decision to use providers for AI inference for now.
Valid takeaway.
"Invest" in the companies that release their models, because the alternative is to reward the people who will capture the market and jack up the prices to unreasonable level.
Unfortunately I think this is a bit like game theory, people will choose the local optimum despite the long term problems with doing so.
It is hard for the engineer to grasp the lengths VCs will go to to grab market and mind share.
The providers on OpenRouter serving open models aren't "throwing money away", agreed.
But that's not the point I'm making. (or, it kind of is, but it's more high level than that).
They're running spot and preemptible GPU instances (60-80% cheaper than on-demand), paying wholesale industrial electricity rates, and running at multi-tenant utilisation densities that make your MacBook look like a bonfire. Of course they're not individually loss-making on inference, they're aggregating cheap commodity compute and skimming a margin, and on paper that's what makes it seem like a good idea, certainly not a loss leader right?
But zoom out a bit; the entire stack is swimming in VC money. OpenRouter itself just raised at a $1.3B valuation backed by a16z. The Chinese models that now account for 36% of all tokens routed through the platform (DeepSeek, Qwen) are priced the way they are because Beijing-adjacent capital has decided market share matters more than margin right now.
So yes, technically no single party is "throwing money away" on each token; they're just all simultaneously subsidising different parts of the stack for strategic reasons. The floor price you're seeing isn't a stable equilibrium, it's a pile of investor money that hasn't entirely finished burning yet.
> The floor price you're seeing isn't a stable equilibrium, it's a pile of investor money that hasn't entirely finished burning yet.
All that says is that it gets more expensive in the future as competitors exit the market and sustainability becomes important. That’s why Uber and Lyft were so cheap until they killed taxis. One major difference of course is that some models will remain largely good enough and the incremental cost of running will keep dropping to 0 over time since the hardware needed doesn’t get more expensive and is already purchased.
I think we agree.
I only object to taking current prices as if they are perpetual prices.
Most of these folks aren't running on spot/preemptible instances, they're doing 1-2 year reserved rentals. There isn't that much cheap compute floating around and you can be quite profitable on reservations (if you can get them — the compute shortage is real).
I think the question in terms of throwing money away isn't the inference layer: it's whether the companies training open models will be able to financially keep doing so. How long will Moonshot keep releasing future Kimi models? I think there's an interesting wedge they're exploring with being basically a base-model-trainer-as-a-service, i.e. selling rights to Fireworks to sell finetuning services to the Cursors of the world, but it's entirely possible it doesn't pan out.
That being said, Nvidia seems willing to step up to being the base model trainer of last resort via the Nemotron family of open models, since it helps sell more of their hardware — similar to their investments in the CUDA stack to sell hardware (unsurprisingly, Nemotron is designed to run most efficiently on Nvidia hardware, e.g. native NVFP4). So I suspect there will continue to be a pretty good market here.
The blog compares the cost of running Gemma4 31b, which on OpenRouter is offered by small no-name inference providers, not by frontier AI companies. It seems like a fair comparison to me.
LLM generation is bottlenecked by RAM bandwidth and latency. You can get almost linear scaling by evaluating more prompts in parallel, because the GPU has nothing to for the relative eternity it takes to read all of the weights from DRAM for every layer for every token.
On Apple Silicon you can get 4x-8x more tokens per second if you run more queries in parallel (as long as your inference server supports it, and has enough spare RAM for more KV caches).
When inference is done at datacenter scales, when you distribute generation across multiple GPUs and have kernels carefully tuned to specific hardware, the compute vs DRAM bandwidth speed ratio gets absurd like 200:1. That's why everyone gives you batch inference at a steep discount.
Interestingly if you look at the cost of Gemma3 (this is 12 months old, but demonstrates the authors point) on Vertex AI versus Gemini 2.5 pro, the cost per million tokens WAS very similar.
So many more efficiencies possible at scale though. I cannot keep a local model 98% utilized 24/7, at least not with my current workload. A big cloud can. I can’t power my servers with DC, I have this AC to DV conversion nonsense. The list goes on.
Besides fill factor being hard to match, there is also scaling - you can't scale local inference 10x for a spike, but you can with cloud inference.
This is not true. API tokens are not sold at a loss, and hardware gets more efficient over time, so serving inference on the same model gets cheaper. LLAMA 3.1 405B parameters was $6/$12/M tokens in 2024, but in 2026 that same model is $3/$3/M tokens.
The most intelligent model at a given time is much larger than the previous, which is why token costs for GPT5.5 are higher than 5.4. But you should expect that 2 years from now, serving a GPT5.5 sized model will be cheaper than GPT5.5 today. You should expect it to be even cheaper to get an equally intelligent model 2 years from now, because distillation techniques are effective at reducing the necessary parameter count for the same benchmark scores.
So are they going to stop at GPT 5.5? This analysis only seems to be counting inference cost when the majority of the cost, and why they are burning through money, is the training.
Well, I'd be surprised if non-R&D inference providers were selling at a loss. There are a plethora to choose from, competition is quite healthy. Will they keep providing cheap tokens while the labs raise their prices? Probably, but then I don't see how they could be raised in the first place. And what timescale are you talking about? A couple of years? It is appropriate to assume inference will become more efficient over time. If you raise your prices, you are going to be out competed before it's profitable (if you assume it is unprofitable) which would be negligent. I don't see how this makes sense.
Do you have a proof? Anthropic’s CEO said they Are profitable. Same with OpenAI.
Profitable for inference if you completely ignore training costs and that you absolutely must continuously train new models.
Which is where your analogy breaks down and why you think you’re taking crazy pills. Inference is growing and selling the oranges in your analogy. Model building is growing the farm to sell larger, juicier more addicting oranges.
The same mistake was made with Amazon, and a million other tech companies in the early 2010s.
Amazon were losing money, they were losing money because were growing and spent all of their cash flow on growth. It wasn't merely regarded as a hopelessly unprofitable business, if was regarded as potentially fraudulent. The share price collapsed in 2014 because, some thought, the profit would never come, investing in growth was pointless, etc.
Last year Amazon made nearly $100bn in profit. Stock is up 20x from then...this is after AWS was known (everyone also that was a massive fraud, could never be profitable...we know it was printing from day one), after it was the world's biggest retailer, etc.
It is difficult to understate how consistently people make this mistake, not just individually but in aggregate. You see the same thing with restaurants, consumer products, office leasing, so many businesses. This is not to say that the future will happen any particular way but that what Anthropic and co are doing is obviously rational and based upon very real cash flow. Anthropic's growth in revenue is, I believe, unparalleled in modern corporate history. A slight difference in this case is also that the economics of training these models is improving exponentially over time.
Are ya fuckin' serious mate?
The restaurant next to the mines were profitable up until the moment the mines themselves shut down: one doesn't exist without the other.
You can't ringfence inference as "the profitable bit" and then hand-wave away the training. Without continuous training there is no inference product.
Claude 3 Opus isn't sitting there making revenue in 2026 - the thing is just deprecated. The moment you stop spending billions on the next model, your "profitable" inference business is on borrowed time until someone else makes it obsolete.
Maybe I made a mistake in my analogy... They're not growing a farm and then selling oranges. They're on a treadmill where stopping is death, and the treadmill costs $10bn a year to keep running.
> They're on a treadmill where stopping is death, and the treadmill costs $10bn a year to keep running.
You’re literally describing all companies. Google takes about $270bn/year to run. If they stopped spending that they’d die pretty darn quick. It’s also a description of working - unless you’d built up significant savings, if you stopped working you’re also going to die.
> You’re literally describing all companies.
No, not quite. It really comes down to opex vs capex and the depreciation schedule for your investment.
Software development is typically categorized as capex, on a 3-5 year depreciation schedule. You assume the software you write today will be generating value for you that long.
If a big, expensive model training project only gives you value for a year or less, that is not like most companies.
No, the IRS made that change a while back as part of the TCJA but that’s been reverted in the OBBBA. If you build something and never touch it, sure that should probably be capex you have to depreciate. But if you’re investing continuously in it over time, I don’t see how it’s anything other than opex - there’s nothing being depreciated because you’re constantly improving it. Automobile manufacturers don’t have to count their labor force as capex. Indeed I can’t think of any other industry where labor is capex.
But believing that the financials of a project are governed solely by how IRS rules force you to account for headcount is kind of silly.
> If a big, expensive model training project only gives you value for a year or less, that is not like most companies.
The model itself that gets built? Sure (although clearly the timelines are getting longer). However the important bit here is the research that got done along the way and the infrastructure built to make that model building process cheaper, better etc. all of that stuff sticks around but because it’s hard to appreciate externally you discount it to 0 when it’s literally what they actually spent the money on.
But none of that even matters. Google had 270B in opex and their capex has grown from 50B in 2024 to 90B in 2025 and is projected to grow to ~175B for 2026. But even if you discount the “AI” treadmill, you’re still looking at many tens of billions in capex that if they stopped they’d die.
You're saying no one capitalizes software development costs anymore? The rest of the internet disagrees strongly. The finance team at my employer would disagree strongly. Accountants talk to dev teams all the time about how many hours went into new development vs maintenance. It's not just a temporary IRS rule. It's GAAP.
It's the same in other industries. In your example, if labor went into building a whole new assembly line that pays itself off over several years, capex. If it just goes into building products to sell, opex.
https://www.hibob.com/blog/r-and-d-capitalization/
Software that is sold as a service and requires ongoing maintenance like running in the cloud (and people to keep it running in the cloud) is opex not capex. Google Search is most definitely opex.
There was some accounting changes passed during the Biden administration’s funding bill that made it capex for a few years. It’s been rolled back in the Trump administration’s bill and I suspect no one will ever let it happen again.
Those changes were passed during the first Trump administration by a Republican congress, though they didn't go into effect until Biden was in office.
https://kpmg.com/kpmg-us/content/dam/kpmg/pdf/2023/tcja-chan...
The problem is I don’t think computing is going back to the mainframe era you know where all the computing is done remotely and the only thing you have in front of you is a terminal that is the AI slop maker’s dream, the computing power on the desktop/laptop/tablet/phone is getting better and the models are getting smaller and quicker.
There is no moat. In the end, what we are calling AI today will just be something that is incorporated into an existing programs that people will use to help them accomplish a task. The public will not be paying more for it. It will just be a commodity added to the existing ecosystems we have today. They
> Without continuous training there is no inference product.
This claim deserves teasing apart.
Clearly, training is a Red Queen's race today. If a model provider were to unilaterally decide to stop training, they would very quickly lose market share to competitors with better models.
On the other hand, what if market and investment conditions change such that everybody has to stop training?
In that case, the models are still there and still as useful as they were the day before. So why wouldn't there still be an inference product?
> Claude 3 Opus
Unless they are changing the architecture in huge ways. The pre-training done for 3 goes into later models. I am sure the frontier labs are figuring out how to pretrain generic feedstocks that can be fed into downstream training pipelines. DeepSeeks incremental training run cost was what, 5M? Alibaba and DeepSeek have the best most efficient training pipelines, look at the rate at which custom Qwen models are being pumped out.
What's the point of these words and analogies when the only thing that matters is numbers. Gross margins of 20% versus 70% makes a world of difference (literally the difference between a company that's about to collapse and a multi-trillion dollar self-sustaining juggernaut) but in your world of words these two companies are the same thing.
Dario has stated they made more from selling inference to Opus 3 than they did training it. Same with 3.5, 4.0 and I'm assuming 4.5 as well.
The reason they're losing money on paper is because the models keep growing 10x in size every generation but they're not getting 10x returns on model inference (closer to 2x)
I know what you’re quoting, but you've badly misread it.
Dario's point was the opposite of yours; he used per-model accounting to explain why the company P&L gets worse every year, not better. His own numbers (10x training costs each generation and ~2x revenue return.)…
"It looks like it's getting worse and worse" are his words, not mine.
In 2025, Anthropic's inference costs came in 23% over their own projections. They cut their gross margin forecast from 50% to 40%.
> Inference is growing and selling the oranges in your analogy. Model building is growing the farm to sell larger, juicier more addicting oranges.
In this analogy, model training would be akin to developing better oranges, but your competitors are also developing better oranges so if you stop spending heavily to improve your oranges, consumers are going to buy ~zero oranges from you within a couple years. (Expanding the farm might be analogous to expanding data centers.)
In this particular case, inference and training are intertwined. It might be one thing if Anthropic could get away with training a new model every five years and control costs that way. But they can't. Put another way, their inference has no value without continuous, very expensive training. Because consumers aren't purchasing based on price but capability, otherwise the Chinese models on OpenRouter would have buried OpenAI and Anthropic already.
Last month Anthropic tried to control the narrative by drumming up the “super scary AI” trope.
The news they successfully buried was that companies like AirBnB are now running Qwen and open source models. The free oranges are now good enough. There is no future unless the goal is to get to super intelligence and utterly take over the world before anyone else gets one. Anything else and free models are six months behind. The money now is the opposite of what everyone thought a year ago: datacenters. Everyone thought AWS was fucked. Turns out AWS is really good at running Qwen.
And ignore capital costs, depreciation, user churn etc
AI CEOs are known to say many things telling the truth, probably isn’t one of them.
Do you mind sharing source links to that profitability claim?
I’m struggling to find the quotes.
Open AI: https://simonwillison.net/2025/Aug/17/sam-altman/
Anthropic: https://x.com/jaminball/status/2052112309364162874
My reading on anthropic is that he strongly infers that they're profitable, realizes what he has said and immediately walks it back as explicitly not the case today and reframes it as a guess about some indeterminate point in the future.
> Those are the economics of the industry today, or not today but where we're projecting forward in a year or two.
If only they had their books open to do more than just "say"
Anthropic says they are losing money but hope to be profitable soon, if they can double their revenue. https://techcrunch.com/2026/05/20/anthropic-says-its-about-t...
OpenAI published their operating margins today. I can't find a non-paywalled source but Judd Legum reported it to be -122%.
do you have proof? Taking these guys at face value is not wise
The models have been dropping 10x in price for completing the same tasks, year over year. Even if you think Anthropic is losing money charging 10x more than everyone else for their 400B model, the prices will continue to go down based on model improvement alone
These models will become more expensive over time, it's functionally impossible for them not to, they just want to capture the market before they have to stop selling at a huge loss.
They could have said the same about transistors. People keep inventing new ways to keep the costs down. Just look at the latest Qwen, DeepSeek, BitNet. Interesting tidbit: they’re all open, and as Google said in 2022: they have no moat.
It is the model training that is dragging them down.
If the arms race stopped tomorrow the current price pays for the inference.
But isn’t training models, a forever task like iterating in tech you can never take a day off, adding humans to the equation don’t humans train/teach themselves new skills over a lifetime, and isn’t one of the selling points in the future when selling this AI slop your AI never goes to sleep and can always be trained forever? The AI price for entry as we go on into the future will only increase.
I agree that training is a forever task, and the current rate of training is probably not sustainable. But all that means is that once the current investment mania ends, the market will most likely find a new equilibrium where continuous training still happens, but at a slower rate that can be sustained by inference revenue.
> but at a slower rate that can be sustained by inference revenue. reply
also it's possible that the scale of inference needed (e.g. Jevons paradox) keeps growing to the point that training costs can fully be absorbed (since training cost is one off vs. inference that can scale).
(I suspect that might be the thinking, I don't know if it will be true, it's also possible that no model will create a moat big enough to attract enough of the inference traffic to make it true).
Depending on the chips/architecture used, the off-peak traffic from inference can also subsidize the training costs.
Just keeping it up to date with competitors is much cheaper, by copying better ones like Qwen did with Claude. Also a bunch of research is trickling into open source / arxiv so catching up should continue becoming cheaper at least as a fraction of training from scratch
The inference is absolutely not sold at a loss, at least not when paying API prices (the subscriptions are less clear). The reason frontier model companies aren’t profitable is because training the models is so costly, not inference.
https://old.reddit.com/r/GithubCopilot/comments/1tbb5bj/gith...
Seems to be on its way! I know of at least one person whose company is looking at a 20x increase, and afaict (from related looking around, nothing concrete tho) business accounts are missing some costs in the calculator so it'll likely be higher.
You should probably take some stay-on-topic pills, as this article is clearly and unambiguously talking about open weight models (e.g. gemma 4), not the ones allegedly being sold at a loss (Opus, ChatGPT, etc)[0].
[0]: these API are not sold at a loss either, by the way. But it's a nice meme so let's just pretend they are.
Except that’s not what the analysis is. They’re spending < $1 to get $1 from you and the other $9 to figure out how to improve the model further and build up products on top of that to turn that $1 spend into $5 in the future.
In other words, inference is fairly profitable for them and the rest of the money is spent growing revenue as quickly as possible. Building models is still an expensive line item but the costs for that are going down with time.
There is also maybe a “capture the market” mentality but I don’t think that’s necessarily it - the tools and processes are largely fungible and that’s a huge problem. They need to figure out how to make it sticky for “capture the market”, but there’s also a very real “grow as big as possible as quickly as possible to take on Google”; Google has an existential threat here.
> Frontier AI companies are selling at a loss.
How big/deep of a loss?
I feel like I read this every day for years that Uber did this same "idiotic, losing" strategy (how it was pitched/discussed) and then one day we woke up and... without much fuss, boom, they were profitable seemingly overnight.
Well and uber cut the driver pay in half and doubled the price. They didn’t really find any efficiencies, robo drivers don’t exist yet. Also why I hardly touch them anymore.
All that tells me is they did find an efficiency. If they didn’t, their driver supply would have dropped. Unlike the taxi business, Uber/Lyft can tap into otherwise dormant supply of drivers who already own a car but aren’t willing to spend all 40-60 hours a week driving a taxi. With Uber/Lyft, they can become part-time drivers (they have flexibility and they can use an asset they already own anyway). Is it worse for the full time taxi drivers who used to have the supply artificially constrained in the old medallion system? Yes, but does it also benefit others who want to do this as a flexible job, zero skills required other than driving, no boss to deal with, no job interviews, etc. Yes!
> Well and uber cut the driver pay in half and doubled the price
Devil's advocate:
* inflation caused everything to go up to some degree since then
* if it was "that bad" as you say, they wouldn't be extremely profitable and have so many users
both things can be true? "they cut the driver pay in half and doubled the price" did not lead to the collapse of the business/people to stop using it.
Ed Zitron discusses this as part of his post on AI economics : https://www.wheresyoured.at/ais-economics-dont-make-sense/
As long as you have slaves/sharecroppers, driving the people at the top of the pyramid at Uber they’re profitable and Uber makes money as long as you don’t care about the workers and as long as you can get around all of the regulations that are put on traditional cab companies if there are any left on the road.
For me nothing says low class like the Porsche dealer saying we can call Uber for you to take you home ridiculous… and it was a low class experience dirty car small never again ha ha ha…
The Michael Scott AI Companies.
> I feel like I'm taking crazy pills.
Why? It's no less crazy than when Uber and Lyft were doing the same thing. Or when the entire tech industry was doing it in the dot com boom.
Investment-driven market growth at a loss is like the least surprising thing in all of this. The tech is new and fascinating. The bubble is just another trip through the funhouse.
> Frontier AI companies are selling at a loss.
There are huge economies to be had by batching requests and using lots of RAM for MoE (sparse models). You can't achieve that efficiency at batch size 1 on a single node.
Exactly, they put a lot of money into engineering and it does give results
Inference isn't the problem.
If you want a good dense model, use qwen3.6 27B instead, speed will be up, and if you don't take my word for it being smarter, take openrouter's prices of it against the bigger, slower and less memory-efficient gemma do the talking.
If you want a faster model, go for qwen3.6 35B (or gemma 4 26B if gemma models perform better for your tasks). There is a reason why people (myself included) haven't shut up about those two (especially the 27B). Its small enough to run at a decent speed (especially with the built in MTP that finally has official llama.cpp support) and for many workloads (every benchmark I have ever thrown at it) it is matching or surpassing models it has no right to.
A couple of days ago I woke up with my internet being down, started 27B in pi, told it to diagnose whats wrong by giving it my router's password, went to grab a coffee and by the time I got back, i had a full report with suggestion on how to proceed. I love openrouter and I use it for many things, but it is not cheaper.
Subjectivity and opinions based on personal experience with all those models implied naturally, I assume the 31B gemma has cases in which it edges out, I've just failed finding any and I have been running all 4 models mentioned since hours after each of them dropped nonstop for different tasks. Hell, for my hermes, I've started getting better results once I switched from gemma 4 26B to qwen3.5 9B, not even the massively improved 3.6 series. It just feels outdated/ cherrypicked to not use what by many accounts is the current consumer hardware SOTA if doing such an analysis.
Right. Qwen 3.6 45b (6 parameter) runs on a commodity 5090, which, if you're into video games, you probably already have one of. It is entirely usable for most code generation tasks. (Not all, but most.)
Likewise, DeepSeek V4 Flash is quite accessible on local models, with DwarfStar 4 making it easy to run on a 96GB MacBook.
There's nothing wrong with paying for inference, but local models bring up some pretty amazing possibilities, such as entirely offline usage or being able to work on private PII, legally privileged, etc. sort of data, or performing tasks with no concern given whatsoever towards billing overruns.
The other possibility is being able to build a service which you can be 100% assured you can keep running without worrying about a service going down or being end-of-lifed, which is currently a problem with frontier models. My local Qwen setup is entirely predictable. It can run as long as I can keep finding hardware to run it.
A sensible strategy uses both: have local inference tools available, and use both low-cost and high-cost cloud based models. You can use GPT-5.5 and Opus-4.7 for things they excel at (including laundering the latter via a Claude subscription to make it cheaper) for demanding reasoning tasks, DeepSeek V4 Pro for slightly less demanding tasks, V4 Flash for most (not all) code generation, and then local models for things where you want a local model.
Ah yes, most people who play any sort of video game own a $2000 graphics card :p
Doesn't everyone have at least one of them on their yacht?
[dead]
Not disagreeing with your argument, but:
> If you want a good dense model, use qwen3.6 27B instead, speed will be up, and if you don't take my word for it being smarter, take openrouter's prices of it against the bigger, slower and less memory-efficient gemma do the talking.
Don't know if this is the correct read. I think those providers are simply taking cue from Alibaba's first-party pricing for the 27B Dense. It's kinda overpriced imo. Perhaps it can be explained by how 'reasoning-inefficient' (relative to frontier models or even Gemma) the Qwen models are and longer sequence lengths are expensive to serve.
I feel like if I had the infrastructure and saw that there is a huge interest in the model, i'd just undercut alibaba's prices a little harder to grab all the consumers. I am sure that the providers have done the math and found that there is a reason not to do this (compute-bound if too many users?), but the delta is very stark, especially for output. Last I checked the cheapest 27b on openrouter was 2$ out vs 0.38$ for the 31b.
But I do agree that the openrouter prices aren't a strong signal and probably should have worded it a little better. It's just a really stark and 'in your eyes' gap.
I'm interested in how you evaluate quantized models against each other; haven't found a benchmark I love for that. I love this example about 27B debugging. I've seen similar success after I got a Mac with 4x memory; and Qwen 35B A3B all of a sudden is doing a great job (the 9B on my laptop wasn't great to say the least).
A lot of comments here are about the issues with the analysis in OP’s post but much of them are “a distinction without a difference” with respect to the broader conclusion. When we look at purely cost and performance (setting aside privacy) then it’s better for individual devs to pay for hosted then for self hosting. Employers are paying for tokens on the job and most devs are finding the $PREFERRED_PROVIDER’s $20/$100/$200/month subscription sufficient outside of work. Most devs don’t fall in the conditions under which running local models make sense purely on the basis of cost vs performance.
More critically, in practice, setting up local models seems more like a hobby, an educational exercise, or an act of privacy control than it is for cost cutting or productivity.
The model makers, mainframe dream of computer’s isn’t coming back no matter what OpenAI, Google, Anthropic or Microsoft want, there are too many smart tech barbarians at the gate that want in and they’re not going to be satisfied to go back to the computer terminal era.
Personal computers eliminated an earlier terminal era, and most if not all of those companies are gone except for IBM and a few stragglers and they are a shell of their former selves.
The author only compared output token costs -- but for typical agentic workloads, input tokens dominate the costs by a large margin. Running inference locally, input tokens are, to first order, free. (They only generate implicit costs through higher time-to-first-token, higher power use, and lower token output speed).
Yeah, that completely invalidates his point.
I looked at a couple random agentic sessions in my openrouter activity, and the input cost is 10x the output cost.
Prompt caching on openrouter is complicated and unreliable. On local hardware with llama-cpp, it's mostly free.
Even ignoring superior caching on a local setup, Mac hardware can often process input token around 10x as quickly as they produce output tokens. Openrouter seems to have only a 2x difference on the same models.
For larger contexts (eg. 20,000+ token agent workflows), being 10x faster still isn't enough. You have to be close to ~100x faster at crunching contexts for it to feel like realtime.
I've dug into this previously for one simple reason: NVidia segments the market by capping VRAM and Apple silicon uses a shared memory model that could challenge that but it currently doesn't. And I really wonder if Apple realizes the potential of what they have or if they even care.
So, for comparison, a 5090 has 32GB of VRAM and you can get one for ~$3000 maybe. To go beyond that memory with current generation (ie Blackwell) GPUs, you have to go to the RTX 6000 Pro w/ 96GB of VRAM and that's almost $10,000 for the GPU by itself. Beyond that you're in the H100/H200 GPUs and you're talking much bigger money.
Part of the problem here is the author is looking at laptops. That's the only place you'll find the M5 Max currently. The real problem here is that the Mac Studios haven't been updated in almost 2 years. There were configs of those with 256/512GB of RAM but they've been discontinued, possibly because of the RAM shortage and possibly because of they're reaching EOL. Apple hasn't said why. They never do.
Many expect M5 Ultra Mac Studios in Q3 and the M5 Ultra may well have >1TB/s of memory bandwidth (for comparison, the 5090 is 1.8TB/s). Memory bandwidth isn't the only issue. A 5090 will still have more compute power (most likely) but being able to run large models without going to a $10k+ GPU could be huge.
But yes, it's hard to compete with the scales and discounted electricity of a data center. Even H200 compute hours are kinda cheap if you consider the capital cost of what you're using.
I've looked into getting a 128GB M5 Max 16" MBP. That retails for $6k. You might be able to get it for $5400. But I don't think the value proposition is quite there yet. It's close though.
I think Apple really do care and know that Moore’s law is likely to position them as major winners in this race in 3-7 years time.
This. The M5’s massive speed up in refill is a good sign.
Apple isn’t expecting wholesale adoption of on-device models this year or next. But all of their design and iteration suggests they see it coming.
Apple has been doing tons of on-device ML for years now, but it's primarily background magic like photo tagging.
Mmmm, nope if you do the smart thing. MacBook M5 max 128gb is a premium laptop at 6k, but with it you can do many things and is your good main driver for the day. Then, it can also run DeepSeek V4 flash and perform non trivial tasks locally, without censorship or limitations, even without an internet connection and on very privacy sensitive data. That's a good deal. If you buy 25k for a dual Mac Studio 512gb to abandon OpenAI and company you are going to be disappointed by both performance and cost.
The smart thing is to get a ~48gb MacBook and use it as your daily driver, and then budget ~$800/year for AI subscriptions or tokens and you'll end up at the same price.
I say, as the author of the blog post, writing this on a MacBook M5 max 128gb..
I agree with you, practically. But there is another angle of the story: for instance models are starting to be useless to do security stuff, since they are every day more censored. Also prices skyrocketed in the latest months, what will happen later? A few months ago I was shocked people resisted to spend 20$/month to get basically free frontier models, and I warned we were headed to house monthly rent figures in the future as AI becomes more and more required to do work. So indeed what you say is absolutely true now (but $800/year is not accurate: you need 20x accounts to do real work in my experience, so $200 * 12 = 2400$/year). But if you have a 128GB MacBook, that no longer looks so costly compared to 2400$/year of frontier models, you can experience uncensored LLMs, a quick thing that always works to do low-value work like TLDR this blog post for me, or what's wrong in this function? Or could you explain me this API? And for this kind of work, DeepSeek v4 Flash looks basically frontier. So if you look at things in perspective, they could have a different shape.
"starting to be useless to do security stuff, since they are every day more censored"
I have not observed this at all, If anything the big 3 are getting both better and more helpful in this area. And if you pass the security checks Antropic will give you Limited access to Mythos.
Yea my m4 max with 128gb has ended up making a lot of sense for me. I do video editing, I train ml models, I run large open AI models, I do 3d modeling, rendering and cad work. I never do all of this 100% of the time, I’ll setup a ml training to run over night and check results in the morning, during work I’ll set it up as a server and run local models, on my own time I’ll edit video and work on 3d modeling. It’s an incredibly versatile machine - and all of this is done while keeping your data on your device and giving you full control over your workflows.
Don't tell the HN crowd, but you can run some of these models on a $200 rpi5 or a $500 AMD mini PCs.
Another open secret is that that certain companies give you tens of thousands of tokens freely, with pretty respectable models such as Gemini 3.1 and GLM 4.6.
OP is comparing against Gemma everywhere but concludes paying Anthropic make more sense. Anthropic is $15 per million output token which is 30-35x more expensive even in openrouter .
This is like comparing e-bike at home with e-bike rental and concluding therefore we need to rent Toyota since it can go at similar speeds. Getting tired of bad posts getting much attention .
Article is seriously wrong, because it makes a huge mistake in the last part. You can't simply look at the produced tokens and that is your cost. In agentic coding there are lots of turns meaning you not only pay for the output tokens you also pay for all the input tokens sent each time (even if a lot cheaper, like 10x when cached). So this calculation does not accurately represent the api cost at all.
Second thing is you can starkly upgrade the token generation locally if you use agent teams. Single conversations are memory bandwidth bound and don't fully make use of your compute. If you can batch tokens from multiple agents you can easily 5x token generation.
I simply can't go back to cloud AI. Privacy and full control are more important to me than speed and SOTA models.
Also predictability, resilience, sovereignty. I'm not worried about other people's outages, that unexpected demand will impact me at an inconvenient time, that someone's watering down my model, that my costs will change unpredictably or that some unforseen error will lead to a huge bill.
It's in the same category as rooftop solar for me. It doesn't have to make strict economic sense if you're the particular type of person who gets peace of mind from control of infrastructure / reduced dependency.
"Accelerated depreciation (if any) from shortening the lifespan of the device will be more expensive than the electricity"
Shortening the lifespan?
The amount of FUD and notion that hardware depreciates in this manner is widely held. I blame Michael Burry of the Big Short who is perpetuating these lies to the investor community today.
There's a bunch of retro hardware which should make people pause and realize they're stupid to assume hardware slows down on average even 5% 20 years later (it's probably closer to 2% and I'm being generous).
HVAC/power delivery and generation are the major factors, and if you didn't skimp/get defective parts for this and replace failed moving parts (usually fans), your hardware is basically the same 20 years down the line as it was today.
Also using LLMs locally doesn't even induce sustained 100% GPU usage over significant periods of time for most real (agentic coding in OpenCode) use-cases.
There are tons of things that can start failing on hardware. I don't realistically expect some LLM usage to materially reduce the lifespan of the laptop, but running it 24/7 for AI usage makes me think that I'm more likely to get 3 years out of the device instead of 10.
This is the FUD I’m talking about. B200 GPUs will easily have economically useful lifespans assuming proper cooling and power delivery.
How much does your data privacy cost?
As stated in the analysis, thousands of dollars. That said, the smart thing to do is target smaller models (few billion parameters) and then use larger models for non-privacy tasks.
The true advantage of locally self-hostable, open weight models isn't about monetary cost at all, it's about the CIA triad.
Running locally, you get confidentiality of knowing your tokens are only ever being processed by your own hardware. You get the integrity of knowing your model isn't being secretly or silently quantized differently behind the scenes, or having it's weights updated in ways you don't want. And you get the availability of never having to worry about an API outage, or even an internet outage, for local inference capacity.
And this isn't even starting to address the whole added world of features and tunability you get when you control the inference stack. Sampling parameters, caching mechanisms, interpretability etc.
OpenRouter may be cheaper than frontier labs, but you still lose all of these benefits from open weight models the moment you decide to rely on someone else's hardware for your processing.
Will this cost structure always be this way and are there other benefits to not running your LLM on the cloud?
E.g.
Privacy
Uptime
Future cost structure controls
This is a field that has moved very quickly. And it has moved in a direction to try to trap users into certain habits. But these habits might not best align with what best benefits end users today or some time in the future.
I like that the numbers were crunched, but the answer to these is always a bit of a foregone conclusion.
* Industrial power pricing
* Wholesale hardware pricing
* Utilization density of shared API
means API always wins a cost shootout.
Privacy & tinkering is cool too though
In my testing, qwen-3.6-27b in full precision is well below sonnet, but above claude haiku in coding tasks. Gemma is not even close to qwen, it’s much, much worse.
How do you test? I made this comment elsewhere... but I don't see a good benchmark that covers "how good is this thing at actually driving coding with tool use locally"?
Everything is currently heavily subsidized. If the AI companies don't improve efficiency, they'll eventually have to start charging what it actually costs to offer the service, which is a multiple of what the currently charge.
I expect self-hosted to be quite competitive pretty soon. Github Copilot is already wildly more expensive than it was last month. People are going from spending a few bucks to a few thousand for that same usage. So, if it doesn't get a lot more efficient (like 3x the tokens, or more, from the same infrastructure), the prices will have to go up quite a lot to keep the lights on. Everything in AI is running partly on investors money, everyone is trying to buy a monopoly and insurmountable lead and some way to lock people into a specific model and ecosystem, but so far that hasn't happened (except for people who voluntarily lock themselves into a specific ecosystem, but even in those cases, it's usually easy to get the AI to help move to another, there are no truly unique features in AI that at least one, and probably three or four, other players don't also offer).
Slightly different slice into this a very similar situation (local vs OpenRouter AI inference).
But in _every_ metric other than privacy it was better to run via OpenRouter than a local model, and not by a small amount.
Direct link to the comparison charts:
https://sendcheckit.com/blog/ai-powered-subject-line-alterna...
I suppose folks here already know this but it deserves a mention: subscription pricing is 10-20x cheaper than API pricing at Anthropic for example and it will be a far better experience (better models, faster responses, as much parallelism as you want, etc) so if it works for you there's no economic argument to buy a machine for inference at the moment.
I don't hear people debating which is cheaper, local or cloud run models. The conversation, at least what I hear, is a lot of the time users are not utilizing an awful lot of tickets all the time, those providers will be paid if you never use them. If 80% - 90% of the work I and my team are doing with Ai is grunt work, write tests for this, implement a FFT here, write the dB query for X. Nothing exhausting. Those who are using AI for whole cloth "vibe coded" applications and services are definitely better suited to cloud. If a work laptop can run my local models and get my works needed performance for development, why wouldn't I as a company prefer that?
Add to that the privacy improvements and data protection and potentially further specific inferance if needed it's a no brainer.
Again, Ai is a tool, and the right tool for the job, I would wager with no evidence looked up, is that the majority of Devs would be happy with 10-30 per second locally.
Local LLMs aren’t about cost, but control.
This is not surprising at all. The biggest benefit of cloud model in terms of energy efficiency is that when running more than 1 requests, the power consumption of said GPU roughly stayed the same. The more concurrency requests the server can handle, the less power each request consume. The server GPU is already likely more energy efficient than local GPU, concurrency make the cost structure unbeatable by local hardware. It is generally assumed the local hardware only run 1 request, but if the local engine is meant to serve a small business with meaningful concurrency, the economy might still work out.
Right now, local inference only make sense for privacy reasons.
This is common when processing PII. Lawyers, doctors our similar should not be using cloud solutions.
Also it's harder to setup and always more expensive than any cloud solution.
It should not at all be surprising that running models at home is more expensive than commodity providers. That's just generally true of running your own stuff. Even if the cost in money isn't higher, the cost in time is often _significantly_ higher.
This is why the idea that the AI labs are in trouble because inference will be a commodity is _completely backwards_. Some of the largest and most powerful companies in the world sell commodities. They compete on scale and efficiency, and you are never going to be able to compete with the big labs on either.
Consider deepseek as well. About 50 cents per 1M tokens, for >1T model
> "run a model like Gemma 4 31b, which is almost anthropic sonnet levels of performance"
I wish people stopped deluding themselves — I regularly try (and benchmark for my purposes) local models and they are NOWHERE near the huge models like Sonnet or Opus. Nowhere. Yes, you can sometimes get plausibly-looking output for simple tasks, but for anything even remotely requiring thinking there is simply no comparison.
Local models are useful. I use them for spam filtering, and soon intend to use them for image tagging and OCR. But let's stop saying they can get us "anthropic sonnet levels of performance", because that's just not true.
It all depends on use case. A local fine tuned model on a very specific use case can definitely out perform a much bigger cloud model that doesn’t have the training on your use case. But, that requires looking at the ai models as a means to end and not a Swiss Army knife that can do it all.
Even if a Mac mini at home was slightly cheaper per token I still use OpenRouter because I want to out source the heat generation and noise to a datacenter.
So I did the India-specific analysis for a tier-3 city. Here, electricity costs 1/3rd of the US version, and you also get solar subsidy up to a certain amount.
https://shorturl.at/q6gRE
tldr;
Hardware deprecation costs are the major factor.
But, if we assume ZERO hardware deprecation (not realistic), then local inference becomes super cheap.. roughly, 90%+ cheaper.
Third case: the break-even happens only if we can get at the very very very least, 8.7 years of useful hardware life. A more realistic number, however, when working 8 hrs/day and not of 24 hrs/day, is around 25 years.
So, for now, local inference is preferable if you deeply care about privacy. From cost perspective, it's still not there.
I think your link is broken? Would love to see the analysis as well.
not much of an analysis really.. just simple math... anyways, that markdown share site takes around 5-10 seconds to load the page.. so, just hang on a bit more time :)
> Throwing money at anthropic makes more sense in this context.
But you are dependent on them, which is the biggest factor IMO, there was a website posted here before of people getting banned from using it over silly reasons, not to mention price hikes, or privacy concerns. Maybe now it’s more expensive or slower to run locally, but you are in full control of everything.
I run the latest 20b-30b models on a MacBook Air... running inference with an MoE (25 tps) for like 2 hours is like 10% battery.. (look me up on huggingface to download my models)
also you gotta realize frontier models have massive "system prompts" that clog up the context window with garbage.
being able to write your own system prompts gives you a MASSIVE edge..
But I want there to be an incentive to develop smaller, private models that can run on local machines that I own. I want that business model to win.
If it costs me slightly more in the short term, but I don't depend on any providers' price gouging as soon as I fully depend on them, I will consider it a win.
Author forgot that after 3 years when hardware no longer decent for inference you can still resell it for 25-50% of price.
Obviously if RAM apocalypse passes by then high-end configurations preserve resale value worse than base models, but still it's hefty bonus of Apple hardware that might change math a lot.
What is the security of OpenRouter? I have a feeling user has no idea where their data is going and how it will be used or am I wrong?
When I see so many options, that looks like it would take months to audit whether it actually makes sense and is safe to use. But I guess some people are fine with YOLO-ing it.
One important difference is that costs are bounded on your own machine. Like with cloud providers, I'm always worried that cost may accidentally explode if I launch an agent swarm wrong.
Now, it looks like the providers I use have good limits. But I do worry about this.
For me, the value in local inference is getting your hands dirty and goofing around. That is to say, learning.
So we shouldn’t be comparing it to the cost of open router api access at all, we should be comparing it to the cost of a 4 credit university course.
And when the heavy subsidies of cloud AI come to an end? Plus the Macbook can do other tasks as well; presumably you need a computer anyway so should deduct the value of a computer for non-AI tasks from the cost.
This doesn't compare like for like, since its comparing the total cost for the local machine with the usage cost for the cloud service, despite the cloud service also needing a local machine to be useful.
So typical AI booster slop
For me, the appeal of local compute is first and foremost confidentiality and having the possibility to run my 200K documents through an LLM just to see what happen without having to consider the cost.
Regardless, I see lots of people using local AI as an excuse to buy over provisioned macs and AMD Ryzen machines. Then not using it at all for local AI.
We all like our expensive toys.
Your laptop AI costs too much? Speculative investors can help!
What would really elevate an article like this is if we could somehow quantify human brain’s equivalent outputs and compare the costs with local LLM and cloud LLMs.
The computer does most specific tasks better, faster, and cheaper than I do.
I have free electricity from solar and an old Macbook Pro M1 Max that has depreciated to zero and has no other use. Now how do the economics work out?
OpenRouter doesn't expose all the LLM sampling parameters/research that llamacpp, vllm, sglang, et al expose (so no high temperature/highly diverse outputs). Also OpenRouter doesn't let you use steering vectors or LoRA or other personalization techniques per-request. Also no true guarantees of ZDR/privacy/data sovereignty.
Oh, and the author didn't mention at all anything related to inference optimization, so no idea if they even know about or enabled things like speculative decoding, optimized attention backends, quantization, etc.
At least AI slop would have hit on far more of the things I listed above. This is worse-than-AI.
Bizarre running local models have nothing to do with cost. It's about privacy first and foremost
Open router doesn't cost money per say, it depends on the providers pricing
> OpenRouter has Gemma4 31b at ~38-50 cents per million tokens. This means that on the optimistic side (50 watts, 40 tokens per second, and 10 years) the pro max is as cheap as openrouter. On the pessimistic side (100 watts and 3 years at 10 tokens per second) the pro max is 10x the cost. I think ~3x the cost per million tokens is likely the right number for local inference on the pro max from an accounting perspective.
Apart from that, like detailed in the the article, pricing for local compute also depends on electricity prices.
By the way, I don't want to snark about it, my English is not very good, but it's "per se", not "per say". Just commenting on this petty thing because it seems to be a common misspelling, and it always trips me up a bit. Makes me wonder about another supposed meaning like "from hearsay".
They do take a cut of 5.5%, (as they should)
> Let's round up to $0.20 per kWh.
Next paragraph
> At ~50-100 watts and $0.18/kWh that's $0.009 or $0.018 per hour. $0.02 per hour. $0.48 cents per day for the electricity to be running inference at 100%.
lol
And this all assumes OpenRouter costs and availability will persist.
I think local LLMs are good for experimentation for now.
Except I already have a local Mac to run Xcode. OpenRouter cannot help with that, at any price.
> 64 gigs should run a model like Gemma 4 31b
No, it can run anything in the 70B range. It's a notable quality upgrade from the 30B, which isn't obvious because the famous flurry of April releases didn't contain any 70Bs.
It can also run 120B in UD-Q3. Or 230B disk-streamed.
Local isn’t (just) about cost, it’s control and trust.
I'm even surprised people ignorantly talking about advantages of buying very expensive device , run it only sometimes and aiming to beat cloud vendors.
If small model is great it will be hosted with good electricity cost and will be utilized 24/7.
Isn't it 2+2 of economics ?
CPU is a commodity, and we are still buying cpu and ram from vendors for same reason
Put a cost on sending your intellectual property to a saas provider who knows where. Half a problem when it is just your IP, hopefully not the IP of your clients. Maybe if one is building yet another html nobody really cares about.
It's a good thing the market is adjusting to the reality that no one needs to own powerful computers, just terminals into the feed of compute available through the cloud, then! Nothing could possibly go wrong from that!
Wouldn’t a Mac Mini be a better comparison?
Yes, or Mac Studio. Laptops with screens aren't made to run 24/7 heavy workloads.
Also after a few years you can sell and upgrade.
A 2022 Mac Studio w/ M1 Ultra and 128gb was ~$5200 new and I see them selling for over $4k on eBay.
Can’t sell your used tokens…
You can’t actually - due to the RAM shortage you can’t even upgrade to an M3 Ultra Mac Studio with 128GB RAM. That model has been discontinued. Even the 96GB model has a wait time of 5 months in most locations. This is the reason why the resale value is so high.
Interesting! I just spec'd one out maybe ~2 weeks ago and I guess I should have pulled the trigger!
Apple services are ~27% of revenue and growing double-digits. The chip is a moat for that flywheel, not a standalone compute bet.
I think that the main flaw in the reasoning is assuming that cost of token will stay the same over the years.
Chances are that token prices will go down, but chances also are that the AI bubble pops and all of a sudden all these companies will either have to make a buck out of the inference or go bankrupt.
Getting your own hardware just grants you stable pricing.
if all those companies go bankrupt, you could also buy their hardware on the cheap.
its almost guaranteed imo that as the model quality evens out, inference cost will drop towards 0 at insane speeds, given how well llama works as an asic.
Isn't this just saying cloud AI providers are heavily subsidizing the true cost of the service.
Not necessarily since they are definitely not running them on Apple Silicon and you don't know their costs
... but comes with privacy guarantees *
* but Apple will collect all your keystrokes anyway
Now include the externalized cost in the U.S. of deploying ~100% of productive capital to build data centers instead of, for example, first-world transportation infrastructure, and tell me which one is cheaper
Why would I want to include that when determining the cost per token?
It’s part of the cost per token
Can’t reply to the reply here, but yes, you do pay for it with money. Absorbing all of America’s capital and construction labor capacity to build AI data centers, rather than, for example, reaching parity with every other developed country in transportation infrastructure, is a cost you pay every day in gas and time spent in traffic. More to the point, building AI datacenters without paying for modernizing the grid is raising electricity rates. Token costs aren’t just subsidized by VC money, they are subsidized by all of us because of idiotic policy choices. Lots of people have pointed out that the current price per token is heavily subsidized, and a fair analysis would account for that.
I live in one of the developed countries, not in the US.
In that case, I’m subsidizing your cheap token costs. You’re welcome.
I don't pay for it with my money.
OpenRouter and other LLM platforms are being subsidized by VC investment to less than it costs them to run inference, the MacBook Pro is not
Bingo. I, for one, am loving this phase of enjoying the LLMs at the expense of VC money. Just like how I enjoyed cheap rides and deliveries on Uber. And with the fragmentation in the field, I don't see a monopoly coming up.
When the AI bubble inevitably pops, the author will find a new way to skew results in favor of cloud LLMs. Like including the price of a desk and a chair in the local token cost.
I really wanted the laptop to look better cost-wise, but it doesn't.
I mean if you’re buying it just as an LLM inference server it’s not, but most people already have laptops, in which case it’s practically free
[flagged]
[flagged]
The full-amortization framing is doing a lot of work here. I bought my laptop because I needed a laptop, not as an inference box, and running a model on it is incidental to that. Once the hardware is sunk for other reasons, the only cost left is electricity plus whatever depreciation you accelerate by hammering the SoC, which the post actually acknowledges in one parenthetical before allocating the full $4299 to tokens anyway.
Also nobody I know picks local over OpenRouter on price. They pick it for offline, for data not leaving the machine, for no rate limits, for not having a provider go down mid-task. If $/Mtok is the only axis, sure, cloud wins.
In practice the pattern I see is leaving a small model running on easy background tasks while using the laptop normally, not a dedicated inference box hammered flat out for 5 years.
[dead]
[dead]
[dead]
[dead]