1) Something happened during 2025 that made the models (or crucially, the wrapping terminal-based apps like Claude Code or Codex) much better. I only type in the terminal anymore.
2) The quality of the code is still quite often terrible. Quadruple-nested control flow abounds. Software architecture in rather small scopes is unsound. People say AI is “good at front end” but I see the worst kind of atrocities there (a few days ago Codex 5.3 tried to inject a massive HTML element with a CSS before hack, rather than proprerly refactoring markup)
Two forces feel true simultaneously but in permanent tension. I still cannot make out my mind and see the synthesis in the dialectic, where this is truly going, if we’re meaningfully moving forward or mostly moving in circles.
> People say AI is “good at front end” but I see the worst kind of atrocities there
It's commonly universal to say "AI is great in X", where one is not professional in X. It's because that's how AI is designed: to output tokens according to stats, not logic, not semantic, and not meaning: stats.
Reading discussions online and comparing them to my own experience makes me feel crazy, because I've found today's LLMs and agents to be seemingly good at everything except writing code. Including everything else in software engineering around code (debugging, reviewing, reading code, brainstorming architecture, etc.) as well as discussing various questions in the humanities and sciences where I'm a dilettante. But whenever I've asked them to generate any substantial amount of code, beyond a few lines to demonstrate usage of some API I'm unfamiliar with, the results have always been terrible and I end up either throwing it out or rewriting almost all of it myself and spending more time than if I'd just written it myself from the start.
It's occurred to me that maybe this just shows that I'm better at writing code and/or worse at everything else than I'd realized.
This matches my experience too. The models write code that would never pass a review normally. Mega functions, "copy and pasted" code with small changes, deep nested conditionals and loops. All the stuff we've spent a lot of time trying to minimise!
You could argue it's OK because a model can always fix it later. But the problem comes when there's subtle logic bugs and its basically impossible to understand. Or fixing the bug in one place doesn't fix it in the 10 other places almost the same code exists.
I strongly suspect that LLMs, like all technologies, are going to follow an S curve of capability. The question is where in that S curve we are right now.
I only say that because I'm a shit frontend dev. Honestly, I'm not that bad anymore, but I'm still shit, and the AI will probably generate better code than I will.
As long as humans are needed to review code, it sounds your role evolves toward prompting and reviewing.
Which is akin to driving a car - the motor vehicle itself doesn’t know where to go. It requires you to prompt via steering and braking etc, and then to review what is happening in response.
That’s not necessarily a bad thing - reviewing code ultimately matters most. As long as what is produced is more often than not correct and legible.. now this is a different issue for which there isn’t a consensus across software engineer’s.
I don't think that reviewing code is so important as reviewing results. Nobody is reviewing the IL or assembly code when they write in higher level languages. It's the end result that matters in most cases.
But we don't evolve IL or assembly code as the system evolves. We regenerate it from scratch every time.
It is therefore not important whether some intermediate version of that low-level code was completely impossible to understand.
It is not so with LLM-written high-level code. More often than not, it does need to be understood and maintained by someone or something.
These days, I mainly focus on two things in LLM code reviews:
1. Making sure unit tests have good coverage of expected behaviours.
2. Making sure the model is making sound architectural decisions, to avoid accumulating tech debt that'll need to be paid back later. It's very hard to check this with unit tests.
We get stuck reviewing the output assembly when it's broken, and that does happen from time to time. The reason that doesn't happen often is that generation of assembly follows strict rules, which people have tried their best to test. That's not the behavior we're going to get out of a LLM.
Yes, prompts aren't analogous to higher-level code, they're analogous to wizards or something like that which were always rightly viewed with suspicion.
> 1) Something happened during 2025 that made the models (or crucially, the wrapping terminal-based apps like Claude Code or Codex) much better. I only type in the terminal anymore.
I have heard say that the change was better context management and compression.
A lot of enhancements came on the model side which in many ways enabled context engineering.
200k and now 1M contexts. Better context management was enabled by improvements in structured outputs/tool calling at the model level. Also reasoning models really upped the game “plan” mode wouldn’t work well without them.
I don't find this very compelling. If you look at the actual graph they are referencing but never showing [1] there is a clear improvement from Sonnet 3.7 -> Opus 4.0 -> Sonnet 4.5. This is just hidden in their graph because they are only looking at the number of PRs that are mergable with no human feedback whatsoever (a high standard even for humans).
And even if we were to agree that that's a reasonable standard, GPT 5 shouldn't be included. There is only one datapoint for all OpenAI models. That data point more indicative of the performance of OpenAI models (and the harness used) than of any progression. Once you exclude it it matches what you would expect from a logistic model. Improvements have slowed down, but not stopped
If you measure completion rate on a task where a single mistake can cause a failure, you won't see noticeable improvements on that metric until all potential sources of error are close to being eliminated, and then if they do get eliminated it causes a sudden large jump in performance.
That's fine if you just want to know whether the current state is good enough on your task of choice, but if you also want to predict future performance, you need to break it down into smaller components and track each of them individually.
> until all potential sources of error are close to being eliminated
This is what PSP/TSP did - one has to (continually) review its' own work to identify most frequent sources of (user facing) defects.
> if you also want to predict future performance, you need to break it down into smaller components and track each of them individually.
This is also one of tenets of PSP/TSP. If you have a task with estimate longer that a day (8 hours), break it down.
This is fascinating. LLM community discovers PSP/TSP rules that were laid over more than twenty years ago.
What LLM community miss is that in PSP/TSP it is an individual software developer who is responsible to figure out what they need to look after.
What I see is that it is LLM users who try to harness LLMs with what they perceive as errors. It's not that LLMs are learning, it is that users of LLMs are trying to stronghold these LLMs with prompts.
I don’t know it’s fair to characterize the LLM community as being ignorant and rediscovering PSP/TCP. I in fact see that as programmers rediscovering survival analysis, and most LLM folks I know have learned these perspectives from that lens. Could be wrong about PSP, maybe things are more nuanced? But what is there that isn’t already covered by foundational statistics?
Robot vacuum is allowed to crash into things and is still quite useful. You add bumpers, maybe some sort of proximity sensors to make the crash less damaging. It is safe by construction - cant harm humans because it is too small.
Things have improved a bit? Now robot shelves becomes a possibility. Map everything, use more sensors, designate humans to a particular area only. Still quite useful. It is safe by design of areas, where humans rarely walk among robots.
Improved further? Now we can do food delivery service robot. Slow down a bit, use much more sensors, think extra hard how to make it safer. Add a flag on a flagpole. Rounded body. Collisions are probably going to happen. Make the robot lighter than humans so that robot gets more damage than the human in a collision. Humans are vulnerable to falling over - make the robot hight just right to grab onto to regain balance, somewhere near waist hight.
Something like that... Now I wish this would be an actual progress requirement for a robo taxy company to do before they start releasing robo taxies onto our streets. But at least we do it as mankind, algorithm improvements, safety solutuon still benefit the whole chain. And benefit to humanity grows despite it being not quite good enough for one particular task.
I don't know that graph to me shows Sonnet 4.5 as worse than 3.7. Maybe the automated grader is finding code breakages in 3.7 and not breaking that out? But I'd much prefer to add code that is a different style to my codebase than code that breaks other code. But even ignoring that the pass rate is almost identical between the two models.
There is a decent case for this thesis to hold true especially if we look at the shift in training regimes and benchmarking over the last 1-2 years. Frontier labs don't seem to really push pure size/capability anymore, it's an all in focus on agentic AI which is mainly complex post-training regimes.
There are good reasons why they don't or can't do simple param upscaling anymore, but still, it makes me bearish on AGI since it's a slow, but massive shift in goal setting.
In practice this still doesn't mean 50 % of white collar can't be automated though.
> In practice this still doesn't mean 50 % of white collar can't be automated though.
Let me ask you this, though: if we wanted to, what percentage of white collar jobs could have been automated or eliminated prior to LLMs?
Meta has nearly 80k employees to basically run two websites and three mobile apps. There were 18k people working at LinkedIn! Many big tech companies are massive job programs with some product on the side. Administrative business partners, program managers, tech writers, "stewards", "champions", "advocates", 10-layer-deep reporting chains... engineers writing cafe menu apps and pet programming languages... a team working on in-house typefaces... the list goes on.
I can see AI producing shifts in the industry by reducing demand for meaningful work, but I doubt the outcome here is mass unemployment. There's an endless supply of bs jobs as long as the money is flowing.
Meta has 80k employees to run the world's most massive engine of commerce through advertising and matching consumers to products.
They build generative AI tools so people can make ads more easily.
They have some of the most sophisticated tracking out there. They have shadow profiles on nearly everyone. Have you visited a website? You have a shadow profile even if you don't have a Facebook account. They know who your friends are based on who you are near. They know what stores you visit when.
Large fractions of their staff are making imperceptible changes to ads tracking and feed ranking that are making billions of dollars of marginal revenue.
What draws you in as a consumer is a tiny tip of the iceberg of what they actually do.
There are many reasons why we are seeing cuts economically, but the fact that it is possible to make such large cuts is because there were way too many people working at these companies. They had so much cheap money that they over-hired, now money isn't so cheap and they need to reduce headcount. AI need not enter the conversation to get to that point.
This is unfair and dismissive of many roles. Coordination in a massive, technically complex company that has to adhere to laws and regulations is a critical role. I don't get why people shit on certain roles (I'm a SWE). Our PgMs reduce friction and help us be more productive and focused. Technical writers produce customer-facing content and code, and have nothing to do with supporting internal bureaucracy. There are arguments against this in Bullshit Jobs but do you think companies pay PgMs or HR employees hundreds of thousands of dollars a year out of the goodness of their own hearts? Or maybe they actually help the business?
It's also because as you increase organisational complexity, you need to manage it somehow, which generally means hiring more people to do that. And then you need to hire people to manage those new managers. Ad infinitum. The increased complexity begets more complexity.
It sort of reminds me of The Collapse of Complex Societies by Joseph Tainter. These companies are their own microcosms of a complex society and I bet we will see mass layoffs in the future, not from AI but from those companies collapsing into a more sustainable state.
You realize that the reason you need to manage this organizational complexity is largely because the organization is so huge?...
The reality is that you could run LinkedIn with far, far fewer people. You probably need fewer than 100 for core engineering, and likely less than 1,000 overall if you include compliance, sales, and so on - especially since a lot of overseas compliance stuff is outsourced to consulting firms, it's not like you have a team of lawyers in every country in the world.
Before there was so much money in the system, we used to run companies that way. Two decades ago, I worked for a company that had tens of millions of users, maintained its own complex nationwide infra (no AWS back then), and had 400 full-time employees. That made coordination problems a lot easier too. We didn't need ten layers of people and project management because there just wasn't that many of us.
When doubling the number of employees can triple your revenue, you do it.
Keeping a website running with high uptime is not the goal. Maximizing revenue and profit is. The extra people aren't waste, they're what drive the incremental imperceptible changes that make these companies profitable.
Did reducing waste affect the user experience or uptime of Twitter? Not really.
But advertising revenues plummeted, because those extra employees were mostly not about the user experience or keeping the website up, they were about servicing the advertisers that brought the company revenue.
Content moderation was an ad-buyer-facing feature. I really see no evidence Musk actually understood that. When he took it away he was all like surprised Pikachu face that advertisers left.
Their last para seems to acknowledge the correlation, but flips your assumed causal direction. I.e. they seem to be implying that the that excess money causes the complexity.
I am pretty convinced that for most types of day to day work, any perceived improvements from the latest Claude models for example were total placebo. In blind tests and with normal tasks, people would probably have no idea if they're using Opus 4.5 or 4.6.
This has basically been my experience since Sonnet 3.5. I've been working on a personal project on and off with various models and things since then and the biggest difference between then and now is that it will do larger chunks of work than it did before, but the quality of the code is not particularly better, I still have to do a lot of cleanup and it still goes off the rails pretty frequently. I have to do fewer individual prompts, but the time spent reviewing the code takes longer because I also have to mentally process and fix larger chunks of code too
Is it a better user experience now? Yes. Has it boosted my productivity on this project? Absolutely.
But it still needs a ton of hand holding for anything complicated and I still deal with tons of "OK, this bug is fixed now!" followed by manually confirming a bug still exists.
4.6 has been a very, very slight regression for me, but the tradeoff is they've added better compaction - and now larger context windows. That's a reasonable tradeoff for me.
Very possible. Some things work like a charm on first try for me, others you can spell it out again and again. And then yet again. Something to do with training data, obviously.
Interesting article, although with so few data points and such a specific time slice it is difficult to draw serious conclusions about the "improvement" of LLM models.
It's notably lacking newer models (4.5 Opus, 4.6 Sonnet) and models from Gemini.
LLMs appear to naturally progress in short leaps followed by longer plateaus, as breakthroughs are developed such as chain-of-thought, mixture-of-experts, sub-agents, etc.
Focusing on flashy breakthroughs hides the issue that bigger models and merge benchmarks rarely translate to reliability in real codebases. For routine merges, subtle regressions and context quirks matter more than headline progress. Unless evals stress nasty scenarios like multi-file renames with tricky conflicts, the numbers are mostly for show. Progress will plateau until someone tunes for the boring, messy cases that waste dev time.
I feel even if the models are stagnating, the tooling around them, and the integrations and harnesses they have are getting significantly more capable (if not always 'better' - the recent vscode update really handicapped them for some reason). Things like the new agent from booking.com or whatever, if it could integrate with all hotels, activities, mapping tools, flight system, etc could be hugely powerful.
Assuming we get no better than opus 4.6, they're very capable. Even if they make up nonsense 5% of the time!
Controversial opinion from a casual user, but state-of-art LLMs now feel to me more intelligent then the average person on the steet. Also explains why training on more average-quality data (if there's any left) is not making improvements.
But LLMs are hamstrung by their harnesses. They are doing the equivalent of providing technical support via phone call: little to no context, and limited to a bidirectional stream of words (tokens). The best agent harnesses have the equivalent of vision-impairment accessibility interfaces, and even those are still subpar.
Heck, giving LLMs time to think was once a groundbreaking idea. Yesterday I saw Claude Code editing a file using shell redirects! It's barbaric.
I expect future improvements to come from harness improvements, especially around sub agents/context rollbacks (to work around the non-linear cost of context) and LLM-aligned "accessibility tools". That, or more synthetic training data.
And search engines are narrow tools that can only output copies of its dataset. An LLM is capable of surprisingly novel output, even if the exact level of creativity is heavily debated.
entirely so. i think anthropic updated something about the compact algorithm recently, and its gone from working well over long times to basically garbage whenever a compact happens
It's so disrespectful to say an LLM is more intelligent than a person on the street. The LLM has nothing at stake, cares not a sausage about the consequences of what it spits out. People have all kinds of pressures, dependants, and personal issues like health. Our thoughts and actions have real consequences. It's so easy to be intelligent when you're the pretend human that gets switched on for five minutes then switched off again.
It's not a value judgement, I'm no misanthrope. But it's a fact or life that we humans must specialize, while LLMs can afford to have "studied" a staggering variety of topics. It's no different than being slower than a car, or weaker than a hydraulic press.
On a different note, LLMs are still not very wise, as displayed by all the prompt attacks and occasional inane responses like walking to the car wash.
On the contrary, it's the machines that have the luxury of specialisation.
I'm faster than a car when scrambling up a mountain. It's easy to be fast when all you do is drive on smooth roads.
I'm stronger than a hydraulic press when carrying a load over a distance. It's easy to be strong when you're limited to a few centimetres of movement.
Intelligence of LLMs is a trick; they are literally trained to sound intelligent. But it's easy to sound intelligent when all you have to do is sound intelligent. A person has to live in the real world, deal with the weather, with feelings, stress, health and, above all, consequences. We don't have the luxury of just being able to sound intelligent, or have wheels that only work on smooth roads. We have to actually be intelligent in our actions. We have to traverse difficult ground, deal with obstacles we've never encountered before. When we make mistakes, people get hurt.
We're probably thinking about this at very different levels. Here's what I meant: I can ask Claude for "a bilingual German-Russian poem about the side effects of the most common drugs used in anesthesia". I would bet my left shoe that if I asked people on the street, no one will do a better job than Claude. And to me, answering questions correctly is a very good metric for intelligence.
We can debate whether that's real intelligence, and whether the question is fair, but this is still a real, measurable capability, that just eight years ago was a pipe dream. This capability is what OP is tracking, and what I believe is impressive but hamstrung by harnesses.
>I would bet my left shoe that if I asked people on the street, no one will do a better job than Claude.
Because it's unrealistic to expect such a niche population to be out and wandering around. What it sounds like you want is something that masquerades at intelligence, but is really alien.
My experience has been that raw “one-shot intelligence” hasn’t improved as dramatically in the last year, but the workflow around the models has improved massively.
I don't think it's true, but am I alone in wishing it was? My world is disrupted somewhat but so far I don't think we have a thing that upends our way of life completely yet. If it stayed exactly this good I'd be pretty content.
I agree with your sentiment, but I think we've yet to see the full application of the current technology. (Even if LLMs themselves don't improve, there's significant opportunity for people to use it in ways not currently being done)
That's a big problem with very specific manifestations. My startup helps customers handle regulatory compliance, also by forwarding complex questions to a pool of consultants.
We've compared now more than a hundred replies to that of GPT Pro, and the quality is roughly the same. Sometimes a little worse, sometimes a little better. Always more detailed. Never unacceptable.
But how to convince our customers that we have the right technology and know how to use it appropriately? We're trying, but it's not easy.
Part of that's accountability. In the event of the LLM producing rubbish, as rare as it may be, who is accountable? There is not a person and her reputation attached to it.
LLM plus human should be better than either standalone. You won’t be able to make as much money scaling out, though.
You won’t be able to scale out and make as much money though. But surely you’re not only concerned about profit, right? What’s the point of life if you’re just trying to get rich.
When the dust settles, for example if LLM's were to stop improving today, we would come to learn their exact capabilities, what they can do reliably and what they can't.
Once we know what they can do well and how to get them to do it well, and what they can't, you could say we "trust" them to do the first category well and just stop trying to get it to do the second category.
This feeds the adoption problem, though: a lot of companies are thinking "why settle for the current models when even the vendors are saying the models in six months will be exponentially better? Let's let the early adopters work out the bugs and move when these things are more stable"
I think we're getting to a point where LLM randomness is relevant to someone writing a white paper on LLMs, but not as relevant to consumers of them. Yes the technology uses randomness, but the quality of response somehow still seems very consistent and predictable in 2026.
I think what happened with static image generation is happening with LLMs. Basically the tools around are becoming better, but all the AI improvements stall, the error rate stay the same (but external tools curate the results so it won't be noticeable if you don't run your own model), the accuracy is still slightly improving, but slower and slower, and never reach the 'perfect' point. Basically stablediffusion early 2025
Image quality has improved a lot in recent months thanks to better models. The ability of people to notice these improvements is plateauing because they are not trained to spot artifacts, which are becoming more obscure.
Yes, slight increase in that kind of accuracy. And newer models still generate absurd stuff. Ask for an historical picture, like 'a London market in the 18th century', and it is still as historically wrong as it was 2 years ago. It is useful for fantasy/sci-fi though, I use them a lot. But I don't see the point of newer models since late 2024.
I have no idea how a London market in the 18th century should look like so I cannot challenge that but recent models like nano banana ones, Z-image (on a lesser extend) can generate images that are essentially indistinguishable from actual stock photos, this wasn't true for late 2024 models, with wonky backgrounds, too smooth skins and general lack of details (the classic AI-look that AI images had).
Mmm, I mostly generate scenes, not people, and when I do, it's always in a not photorealistic way. It is true that newer models have more details, I think that I put that into 'slight accuracy improvements', is it really major? Or is it mostly for close up people/animals and it is likely that I just didn't notice?
It will be wrong on a lot of details. Basically you would get a market scene that feels 18th century gb, but will use 18th century russian/french/Austrian details, or 19th century/20th century British artefacts, or a mix of both. And the further you go from western places, the higher the error rate is. Basically generating fiction scenes. That's pretty much my usecase, so that's fine, but I won't ever use it to illustrate a historical TTrpg.
That's an interesting claim, but I don't see it in my own work. They have got better but it's very hard to quantify. I just find myself editing their work much less these days (currently using GPT 5.4).
Without meaning to sound dismissive, because I'm really not intending to, there's also the possibility that you've gotten worse after enough time using them. You're treating yourself as a constant in this, but man cannot walk in the same river twice.
This is such a silly response when "You've gotten better at using them and know how to work around their flaws now." is right there and seems a lot more plausible.
That's a possibility, but I doubt it. I've been programming for 35 years and know what I like in code. I've also previously maintained a long review prompt in which I tell the models all the ways in which they get things wrong and to go look for/fix those problems. But those review passes now don't take as long because there are fewer such problems to begin with.
In particular GPT 5.4 is much better at not duplicating code unnecessarily. It'll take the time to refactor, to search for pre-existing utility functions, etc.
The problem with evals is the underlying rubric will always be either subjective, or a quantitative score based on something that is likely now baked into the training set directly.
You kind of have to go on "feels" for a lot of this.
Truth is I'm probably wrong. I should keep on testing ... but at the same time I precisely gave up because I didn't think the trend was fast enough to keep on investing on checking it so frequently. Now I just read this kind of post, ask around (mainly arguing with comments asking for genuine examples that should be "surprising" and kept on being disappointed) and that seems to be enough for a proxy.
I should though, as I mentioned in another comment, keep track of failed attempts.
PS: I check solely on self-hosted models (even if not on my machine but least on machines I could setup) because I do NOT trust the scaffolding around proprietary closed sources models. I can't verify that nobody is in the loop.
I reckon LLM merge rates will go up, but not necessarily due to quality improvements. Instead I think maintainers will just become fatigued. The amount of code I'm expected to review now is way higher than before. And while I'm reviewing you know more is being generated. I'm sure I've let through more crap due to this fatigue attack on me.
This has been the general consensus for about three years now. "Drastic increases in capability have happened the last 3-6 months" have been a constant refrain.
Without any data from the study past September I think its not unreasonable, if you want to make an argument based on evidence.
For me personally, I agree with you, I'm really seeing it as well.
> "Drastic increases in capability have happened the last 3-6 months" have been a constant refrain.
well, yeah. because that's been the experience for many people.
3 years ago, trying to use ChatGPT 3.5 for coding tasks was more of a gimmick than anything else, and was basically useless for helping me with my job.
today, agentic Opus 4.6 provides more value to me than probably 2 more human engineers on my team would
I tried GPT3.5 for translating code from typescript to rust. It made many mistakes in rust. It couldn't fix borrow checker issues. The context was so small that I could only feed it small amounts of my program at a time. It also introduced new bugs into the algorithm.
Yesterday I had an idea for a simple macos app I wanted. I prompted claude code. It programmed the whole thing start to finish in 10 minutes, no problem. I asked it to optimize the program using a technique I came up with, and it did. I asked it to make a web version and it did. (Though for some reason, the web version needed several rounds of "it doesn't work, here's the console output").
I'm slowly coming to terms with the idea that my job is fundamentally changing. I can get way more done by prompting claude than I can by writing the code myself.
There's a consensus that SOMETHING changed with Opus 4.5. It might have been the "merge rates" metric, it might have not.
I'm certainly getting faster and cleaner-looking solutions for certain issues on Opus 4.6 than I was 5 months ago, but I'm not sure about the ability to solve (or even weigh in) the actual hard stuff, i.e. the stuff I'm paid for.
And I'm definitely not sure about the supposed big step between 4.5 and 4.6. I'm literally not seeing any.
> This means llms have not improved in their programming abilities for over a year. Isn’t that wild? Why is nobody talking about this?
Because it's not true. They have improved tremendously in the last year, but it looks like they've hit a wall in the last 3 months. Still seeing some improvements but mostly in skills and token use optimization.
> but mostly in skills and token use optimization.
I have heard rumors that token use optimization has been a recent focus to try to tidy up the financials of these companies before they IPO. take that with a grain of salt though
Benchmaxxing aside, if you are using those tools for programming on a regular basis it should be self-evident that they are improving. I find it very hard to believe that someone using LLMs today vs what was available one year ago (Claude Code released Feb 2025) would have any difficulty answering this question.
I think it is important to try to find more rigorous things to test than the general sentiment of the people using the tools. If only because the more benchmarks we have the more we can improve models without regressions. METR is asking a really interesting question here, "are models improving at making one shot PRs?". The answer seems to be, yes, but slower than benchmarks suggest, if you look at the pass rate of different versions of Claude Sonnet. A reasonable answer is "you're not supposed to use them by making one shot PRs", but then ideally we would need to have some kind of standarized test for the ability of models to incorporate feedback and evolve PRs.
You really can't model these 5 data points with a linear regression or a step function. The models are of different sizes / use cases, and from two different labs. I feel like what we've observed generally is that different labs releasing similarly sized models at similar times are generally pretty similar.
I think the only reasonable thing to read into is Sonnet 3.5 -> 3.7 -> 4.5. But yeah, you just can't draw a line through this thing.
I will die on the hill that LLMs are getting better, particularly Anthropic's releases since December. But I can't point at a graph to prove that, I'm just drawing on my personal experience. I do use Claude Code though, so I think a large part of the improvement comes from the harness.
These studies are always really hard to judge the efficacy of. I would say though the most surprising thing to me about LLMs in the past year is how many people got hyped about the Opus 4.5 release. Having used Claude Code at work since it was released I haven't really noticed any step changes in improvement. Maybe that's because I've never tried to use it to one shot things?
Regardless I'm more inclined to believe that 4.5 was the point that people started using it after having given up on copy/pasting output in 2024. If you're going from chat to agentic level of interaction it's going to feel like a leap.
I used it with Sonnet 4.0 a lot, and there was vastly more back-and-forth and correction of "dumb" things, such as forgetting to add "using" statements in C# files.
I don't know if it's model, or harness improvements, or inbuilt-memory or all of the above, but it often has a step where it'll check itself that is done now before trying to build and getting an inevitable failure.
Those small things add up to a much smoother and richer experience today compared to 6 months ago.
I've been able to supercharge a hobby project of mine over the last couple months using Opus 4.6 in claude code. I had to collaborate and write code still, but claude did like 75% of the work to add meaningful new features to an iOS/Android native mobile app, including Live Activities which is so overly complicated i would not have been able to figure that out. I have it running in a folder that contains both my back end api (express) and my mobile app (nativescript), so it does back end and front end work simultaneously to support new features. this wasnt possible 8 months ago.
I have a similar experience. My hobby project was put on hold after a burnout and lack of motivation. I got a big burst of energy back when I started implementing some long desired features quickly with these new models. I was able to get the project to the point of what I consider is maturity. I did in a month during free time the kind of work that would have burned me up in a good six months fulltime.
I feel like anyone used AI coding tools before 11/25 and after 1/26 (with frontier models) will say there has been a massive jump in, there is a difference between whether LLM can do a specific task or pass some arguably arbitrary checks by maintainers vs. what the are capable of.
We still have tons of gaps about how to build and maintain code with AI, but LLM themselves getting better at an unbelievable pace, even with this kind of data analysis I’m surprised anyone can even question it.
> I feel like anyone used AI coding tools before 11/25 and after 1/26 (with frontier models) will say there has been a massive jump in, there is a difference between whether LLM can do a specific task or pass some arguably arbitrary checks by maintainers vs. what the are capable of.
How much of that is the model and how much of that is the tooling built around it? Also why is the tooling, specifically Claude Code, so buggy?
From my personal experience, they have gotten better, but they haven’t unlocked any new capabilities. They’ve just improved at what I was already using them for.
At the end of the day they still produce code that I need to manually review and fully understand before merging. Usually with a session of back-and-forth prompting or manual edits by me.
That was true 2 years ago, and it’s true now (except 2 years ago I was copy/pasting from the browser chat window and we have some nicer IDE integration now).
Well, on one hand they lack new data. Lot's of new code came out of an LLM, so it feeds back.
On the other hand, LLMs tend to go for an average by their nature (if you squint enough). What's more common in their training data, it's more common in the output, so getting them better without fundamental changes, requires one to improve the training data on average too which is hard.
What did improve a lot is the tooling around them. That's gotten way better.
> Well, on one hand they lack new data. Lot's of new code came out of an LLM, so it feeds back.
Supposedly model curation is a Big Deal at Big AI, and they're especially concerned about Ouroboros effects and poisoned data. Also people are still contributing to open source and open sourcing new projects, something that should have slowed to trickle by 2023, once it became clear that from now on, you're just providing the fuel for the machines that will ultimately render you unemployable (or less employable), and that these machines will completely disregard your license terms, including those of the most permissive licenses that seek only attribution, and that you're doing all of this for free.
As someone who builds with LLMs daily without being a developer, I notice quality differences more in practical output than benchmarks. Claude handles complex multi-step tasks better in my experience, but consistency is still the biggest challenge – same prompt can give very different results day to day.
It's my experience that opus 4, and then, particularly, 4.5, in Claude code, are head and shoulders above the competition.
I wrote an agentic coder years ago and it yielded trash. (Tried to make it do then what kiro does today).
The models are better. Now, caveat - I don't use anything but opus for coding - Sonnet doesn't do the trick. My experience with Codex and Gemini is that their top models are as good as Sonnet for coding...
I was trying to do something yestesrday and Claude was keep messing it up, after like an hour i realized the model somehow switched to sonet, opus 4.6 is crazy good. It’s very obvious in practice.
Although I feel like for chasing bugs and big systems codex is even better
Tangential: I've found that having an LLM recreate the full file, with changes appllied, is less mistake-prone, than producing a patch. I wonder if anyone else came to this conclusion too.
Yesterday I asked a frontier model to help generate a report. It said great, it can do that, and output a table. I asked it to evaluate its prompt compliance in the result. It concluded that it had failed on every requirement. I asked why it had expressed such confidence, was it analagous to narcissism or psycopathy? It said no, and then said that if I just had to anthropomorphize it, I should think of it as a brilliant friend with severe frontal lobe brain damage.
I agree completely. I haven't noticed much improvement in coding ability in the last year. I'm using frontier models.
What's been the game changer are tools like Claude Code. Automatic agentic tool loops purpose built for coding. This is what I have seen as the impetus for mainstream adoption rather than noticeable improvements in ability.
I write a lot of C++ and QML code. Codex 5.3, only released in Feb, is the the first model I've used that would regularly generate code that passes my 25 years expert smell test and has turned generative coding from a timesap/nuisance into a tool I can somewhat rely on not to set me back.
Claude still wasn't quite there at the time, but I haven't tried 4.6 yet.
QML is a declarative-first markup language that is a superset of the JavaScript syntax. It's niche and doesn't have a giant amount of training data in the corpus. Codex 5.3 is the first model that doesn't super botch it or prefers to write reams of procedural JS embeds (yes, after steering). Much reduced is also the tendency to go overboard on spamming everything with clouds of helper functions/methods in both C++ and QML. It knows when to stop, so to speak, and is either trained or able to reason toward a more idiomatic ideal, with far less explicit instruction / AGENTS.md wrangling.
It's a huge difference. It might be the result of very specific optimization, or perhaps simultaneous advancements in the harness play a bigger role, but in my books my kneck of the woods (or place on the long tail) only really came online in 2026 as far as LLMs are concerned.
Maybe n=1, but I disagree? I notice that Sonnet 4.6 follows instructions much better than 4.5 and it generates code much closer to our already in-place production code.
It's just a point release and it isn't a significant upgrade in terms of features or capabilities, but it works... better for me.
Are you using a tool like Claude Code or Codex or windsurf? I ask because I've found their ability to pull in relevant context improves tasks in exactly the way you're describing.
My own experience is that some things get better and some things get worse in perceived quality at the micro-level on each point release. i.e. 4.5->4.6
fwiw the merge rate metric itself might be misleading. most real codebases have implicit conventions and architectural patterns that aren't captured in the issue description, so even if the model writes correct code it might not match what the maintainer actually wanted. imo the bigger signal is how much back-and-forth it takes before merging, not whether the first attempt lands cleanly.
Yeah I'm not buying the last bit about lower MSE with one term in the model vs two (Brier with one outcome category is MSE of the probabilities). That's the sort of thing that would make me go dig to find where I fucked up the calculation.
With one term it gets more robust in the face of excluding endpoints when constructing the jackknife train/test split, I think. But you're right, it does sound fishy.
What the post is describing is just ANOVA. If removing a category improves the overall fit then fitting the two terms independently has the same optimal solution (with the two independent terms found to be identical). MSE never increases when adding a category.
This is why you have to reach to things that penalize adding parameters to models when running model comparisons.
Even if one-shot LLM performance has plateaued (which I'm not convinced this data shows given omission of recent models that are widely claimed to be better) that missing the point that I see in my own work. The improved tooling and agent-based approaches that I'm using now make the LLM one-shot performance only a small part of the puzzle in terms of how AI tools have accelerated the time from idea to decent code. For instance the planning dialogs I now have with Claude are an important part of what's speeding things up for me. Also, the iterative use of AI to identify, track, and take care of small coding tasks (none of which are particularly challenging in terms of benchmarks) is simply more effective. Could this all have been done with the LLM engines of late 2024. Perhaps, but I think the fine-tuning (and conceivably the system prompts) that make the current LLM's more effective at agent-centered workflows (including tool-use) are a big part of it. One-shot task performance at challenging tasks is an interesting, certainly foundational, metric. But I don't think it captures the important advances I see in how LLM's have gotten better over the last year in ways that actually matter to me. I rarely have a well-defined programming challenge and the obligation to solve it in a single-shot.
>This means the step function has more predictive power (“fits better”) than the linear slope. For fun, we can also fit a function that is completely constant across the entire timespan. That happens to get the best Brier score.
I mean, sure. but it's obvious in that graph that the single openai model is dragging down the right side. Wouldn't it be better to just stick to analyzing models from only one lab so that this was showing change over time rather than differences between models?
>To study how agent success on benchmark tasks relates to real-world usefulness, we had 4 active maintainers from 3 SWE-bench Verified repositories review 296 AI-generated pull requests (PRs). We had maintainers (hypothetically) accept or request changes for patches as well as provide the core reason they were requesting changes: core functionality failure, patch breaks other code or code quality issues.
I would also advise taking a look at the rejection reasons for the PRs. For example, Figure 5 shows two rejections for "code quality" because of (and I quote) "looks like a useless AI slop comment." This is something models still do, but that is also very easily fixable. I think in that case the issue is that the level of comment wanted hasn't been properly formalized in the repo and the model hasn't been able to deduce it from the context it had.
As for the article, I think mixing all models together doesn't make sense. For example, maybe a slope describe the increasing Claude Sonnet better than a step function.
LLM's have 100% gotten better, but it's hard to say if it's "intrinsically better", if that makes sense.
> OpenAI’s leading researchers have not completed a successful full-scale pre-training run that was broadly deployed for a new frontier model since GPT-4o in May 2024 [1]
That's evidence against "intrinsically better". They've also trained on the entire internet - we only have 1 internet, so.
However, late 2024 was the introduction of o1 and early 2025 was Deepseek R1 and o3. These were definitely significant reasoning models - the introduction of test time compute and significant RL pipelines were here.
Mid 2025 was when they really started getting integrated with tool calling.
Late 2025 is when they really started to become agentic and integrate with the CLI pretty well (at least for me). For example, codex would at least try and run some smoke tests for itself to test its code.
In early 2026, the trend now appears to be harness engineering - as opposed to "context engineering" in 2025, where we had to preciously babysit 1 model's context, we make it both easier to rebuild context (classic CS trick btw: rebooting is easier than restoring stale state [2]) and really lean into raw cli tool calling, subagents, etc.
FWIW, AI programming has still been as frustrating as it was when it was just TTC in 2025. Maybe because I don't have the "full harness" but it still has programming styles embedded such as silent fallback values, overly defensive programming, etc. which are obvoiusly gleaned from the desire to just pass all tests, rather than truly good programming design. I've been able to do more, but I have to review more slop... also the agents are really unpleasant to work with, if you're trying to have any reasonable conversation with them and not just delegate to them. It's as if they think the entire world revolves around them, and all information from the operator is BS, if you try and open a proper 2-way channel.
It seems like 2026 will go full zoom with AI tooling because the goal is to replace devs, but hopefully AI agents become actually nice to work with. Not sycophantic, but not passively aggressively arrogant either.
Anecdotally, I haven't seen any real improvement from the AI tools I leverage. They're all good-ish at what they do, but all still lie occasionally, and all need babysitting.
I also wonder how much of the jump in early 2025 comes from cultural acceptance by devs, rather than an improvement in the tools themselves.
I think I'm coming to the same conclusion Gpt-3 to 5.3 have had real tangible but incremental improvements with quite diminishing returns.
Perhaps we won't see a phase change like improvement as we did from gpt-2 through to 3 until there is several more orders of magnitude parameters and/or training. Perhaps we will never see it again!
What is getting rapidly better is scaffolding but this seems to be more about understanding and building tools around LLMs than the LLMs themselves improving.
I'm still excited about AI but not constantly hyped to the rafters as some.
I think it depends on what you're using it for. If it is a simple kubernetes config then the model doesn't matter too much. Contract that with writing the scenario for a backtest for an algo that trades on a venue: it is not the same complexity and the basic models are terrible. I've had it tell me that it has added tests to find that they're just stubs!
Opus seems to be getting there, but on more complex tasks the others are a complete waste of time.
> If it is a simple kubernetes config then the model doesn't matter too much
I guess at least this person https://www.tomshardware.com/tech-industry/artificial-intell... might disagree. I think already to know what Kubernetes even is requires quite a bit of knowledge. Using a tool that manipulate its configuration files IN PRODUCTION without risking data loss is another ball game entirely.
I feel that two things are true at the same time:
1) Something happened during 2025 that made the models (or crucially, the wrapping terminal-based apps like Claude Code or Codex) much better. I only type in the terminal anymore.
2) The quality of the code is still quite often terrible. Quadruple-nested control flow abounds. Software architecture in rather small scopes is unsound. People say AI is “good at front end” but I see the worst kind of atrocities there (a few days ago Codex 5.3 tried to inject a massive HTML element with a CSS before hack, rather than proprerly refactoring markup)
Two forces feel true simultaneously but in permanent tension. I still cannot make out my mind and see the synthesis in the dialectic, where this is truly going, if we’re meaningfully moving forward or mostly moving in circles.
> People say AI is “good at front end” but I see the worst kind of atrocities there
It's commonly universal to say "AI is great in X", where one is not professional in X. It's because that's how AI is designed: to output tokens according to stats, not logic, not semantic, and not meaning: stats.
Reading discussions online and comparing them to my own experience makes me feel crazy, because I've found today's LLMs and agents to be seemingly good at everything except writing code. Including everything else in software engineering around code (debugging, reviewing, reading code, brainstorming architecture, etc.) as well as discussing various questions in the humanities and sciences where I'm a dilettante. But whenever I've asked them to generate any substantial amount of code, beyond a few lines to demonstrate usage of some API I'm unfamiliar with, the results have always been terrible and I end up either throwing it out or rewriting almost all of it myself and spending more time than if I'd just written it myself from the start.
It's occurred to me that maybe this just shows that I'm better at writing code and/or worse at everything else than I'd realized.
Gell-Mann Amnesia for code quality.
This matches my experience too. The models write code that would never pass a review normally. Mega functions, "copy and pasted" code with small changes, deep nested conditionals and loops. All the stuff we've spent a lot of time trying to minimise!
You could argue it's OK because a model can always fix it later. But the problem comes when there's subtle logic bugs and its basically impossible to understand. Or fixing the bug in one place doesn't fix it in the 10 other places almost the same code exists.
I strongly suspect that LLMs, like all technologies, are going to follow an S curve of capability. The question is where in that S curve we are right now.
The models lose the ability to inject subtle and nuance stuff as they scale up, is what I’ve observed.
> People say AI is “good at front end”
I only say that because I'm a shit frontend dev. Honestly, I'm not that bad anymore, but I'm still shit, and the AI will probably generate better code than I will.
As long as humans are needed to review code, it sounds your role evolves toward prompting and reviewing.
Which is akin to driving a car - the motor vehicle itself doesn’t know where to go. It requires you to prompt via steering and braking etc, and then to review what is happening in response.
That’s not necessarily a bad thing - reviewing code ultimately matters most. As long as what is produced is more often than not correct and legible.. now this is a different issue for which there isn’t a consensus across software engineer’s.
I don't think that reviewing code is so important as reviewing results. Nobody is reviewing the IL or assembly code when they write in higher level languages. It's the end result that matters in most cases.
But we don't evolve IL or assembly code as the system evolves. We regenerate it from scratch every time.
It is therefore not important whether some intermediate version of that low-level code was completely impossible to understand.
It is not so with LLM-written high-level code. More often than not, it does need to be understood and maintained by someone or something.
These days, I mainly focus on two things in LLM code reviews:
1. Making sure unit tests have good coverage of expected behaviours.
2. Making sure the model is making sound architectural decisions, to avoid accumulating tech debt that'll need to be paid back later. It's very hard to check this with unit tests.
We get stuck reviewing the output assembly when it's broken, and that does happen from time to time. The reason that doesn't happen often is that generation of assembly follows strict rules, which people have tried their best to test. That's not the behavior we're going to get out of a LLM.
Yes, prompts aren't analogous to higher-level code, they're analogous to wizards or something like that which were always rightly viewed with suspicion.
But those are close to deterministic.
> 1) Something happened during 2025 that made the models (or crucially, the wrapping terminal-based apps like Claude Code or Codex) much better. I only type in the terminal anymore.
I have heard say that the change was better context management and compression.
A lot of enhancements came on the model side which in many ways enabled context engineering.
200k and now 1M contexts. Better context management was enabled by improvements in structured outputs/tool calling at the model level. Also reasoning models really upped the game “plan” mode wouldn’t work well without them.
I don't find this very compelling. If you look at the actual graph they are referencing but never showing [1] there is a clear improvement from Sonnet 3.7 -> Opus 4.0 -> Sonnet 4.5. This is just hidden in their graph because they are only looking at the number of PRs that are mergable with no human feedback whatsoever (a high standard even for humans).
And even if we were to agree that that's a reasonable standard, GPT 5 shouldn't be included. There is only one datapoint for all OpenAI models. That data point more indicative of the performance of OpenAI models (and the harness used) than of any progression. Once you exclude it it matches what you would expect from a logistic model. Improvements have slowed down, but not stopped
1: https://metr.org/assets/images/many-swe-bench-passing-prs-wo...
Yes, I think this is basically an instance of the "emergent abilities mirage." https://arxiv.org/abs/2304.15004
If you measure completion rate on a task where a single mistake can cause a failure, you won't see noticeable improvements on that metric until all potential sources of error are close to being eliminated, and then if they do get eliminated it causes a sudden large jump in performance.
That's fine if you just want to know whether the current state is good enough on your task of choice, but if you also want to predict future performance, you need to break it down into smaller components and track each of them individually.
This is fascinating. LLM community discovers PSP/TSP rules that were laid over more than twenty years ago.
What LLM community miss is that in PSP/TSP it is an individual software developer who is responsible to figure out what they need to look after.
What I see is that it is LLM users who try to harness LLMs with what they perceive as errors. It's not that LLMs are learning, it is that users of LLMs are trying to stronghold these LLMs with prompts.
I don’t know it’s fair to characterize the LLM community as being ignorant and rediscovering PSP/TCP. I in fact see that as programmers rediscovering survival analysis, and most LLM folks I know have learned these perspectives from that lens. Could be wrong about PSP, maybe things are more nuanced? But what is there that isn’t already covered by foundational statistics?
What is PSP/TSP?
One of many ways people have branded the idea of process improvement for software engineering.
That's how the public perceive it though.
It's useless and never gets better until it suddenly, unexpecty got good enough.
My robo-chauffer kept crashing into different things until one day he didn't.
Robot vacuum is allowed to crash into things and is still quite useful. You add bumpers, maybe some sort of proximity sensors to make the crash less damaging. It is safe by construction - cant harm humans because it is too small.
Things have improved a bit? Now robot shelves becomes a possibility. Map everything, use more sensors, designate humans to a particular area only. Still quite useful. It is safe by design of areas, where humans rarely walk among robots.
Improved further? Now we can do food delivery service robot. Slow down a bit, use much more sensors, think extra hard how to make it safer. Add a flag on a flagpole. Rounded body. Collisions are probably going to happen. Make the robot lighter than humans so that robot gets more damage than the human in a collision. Humans are vulnerable to falling over - make the robot hight just right to grab onto to regain balance, somewhere near waist hight.
Something like that... Now I wish this would be an actual progress requirement for a robo taxy company to do before they start releasing robo taxies onto our streets. But at least we do it as mankind, algorithm improvements, safety solutuon still benefit the whole chain. And benefit to humanity grows despite it being not quite good enough for one particular task.
I don't know that graph to me shows Sonnet 4.5 as worse than 3.7. Maybe the automated grader is finding code breakages in 3.7 and not breaking that out? But I'd much prefer to add code that is a different style to my codebase than code that breaks other code. But even ignoring that the pass rate is almost identical between the two models.
There is a decent case for this thesis to hold true especially if we look at the shift in training regimes and benchmarking over the last 1-2 years. Frontier labs don't seem to really push pure size/capability anymore, it's an all in focus on agentic AI which is mainly complex post-training regimes.
There are good reasons why they don't or can't do simple param upscaling anymore, but still, it makes me bearish on AGI since it's a slow, but massive shift in goal setting.
In practice this still doesn't mean 50 % of white collar can't be automated though.
> In practice this still doesn't mean 50 % of white collar can't be automated though.
Let me ask you this, though: if we wanted to, what percentage of white collar jobs could have been automated or eliminated prior to LLMs?
Meta has nearly 80k employees to basically run two websites and three mobile apps. There were 18k people working at LinkedIn! Many big tech companies are massive job programs with some product on the side. Administrative business partners, program managers, tech writers, "stewards", "champions", "advocates", 10-layer-deep reporting chains... engineers writing cafe menu apps and pet programming languages... a team working on in-house typefaces... the list goes on.
I can see AI producing shifts in the industry by reducing demand for meaningful work, but I doubt the outcome here is mass unemployment. There's an endless supply of bs jobs as long as the money is flowing.
Meta has 80k employees to run the world's most massive engine of commerce through advertising and matching consumers to products.
They build generative AI tools so people can make ads more easily.
They have some of the most sophisticated tracking out there. They have shadow profiles on nearly everyone. Have you visited a website? You have a shadow profile even if you don't have a Facebook account. They know who your friends are based on who you are near. They know what stores you visit when.
Large fractions of their staff are making imperceptible changes to ads tracking and feed ranking that are making billions of dollars of marginal revenue.
What draws you in as a consumer is a tiny tip of the iceberg of what they actually do.
So like parent said, mostly bs jobs that would improve the product if removed </s>
Totally fair! I think my point might be this is more malice than incompetence.
There are many reasons why we are seeing cuts economically, but the fact that it is possible to make such large cuts is because there were way too many people working at these companies. They had so much cheap money that they over-hired, now money isn't so cheap and they need to reduce headcount. AI need not enter the conversation to get to that point.
This is unfair and dismissive of many roles. Coordination in a massive, technically complex company that has to adhere to laws and regulations is a critical role. I don't get why people shit on certain roles (I'm a SWE). Our PgMs reduce friction and help us be more productive and focused. Technical writers produce customer-facing content and code, and have nothing to do with supporting internal bureaucracy. There are arguments against this in Bullshit Jobs but do you think companies pay PgMs or HR employees hundreds of thousands of dollars a year out of the goodness of their own hearts? Or maybe they actually help the business?
It's also because as you increase organisational complexity, you need to manage it somehow, which generally means hiring more people to do that. And then you need to hire people to manage those new managers. Ad infinitum. The increased complexity begets more complexity.
It sort of reminds me of The Collapse of Complex Societies by Joseph Tainter. These companies are their own microcosms of a complex society and I bet we will see mass layoffs in the future, not from AI but from those companies collapsing into a more sustainable state.
You realize that the reason you need to manage this organizational complexity is largely because the organization is so huge?...
The reality is that you could run LinkedIn with far, far fewer people. You probably need fewer than 100 for core engineering, and likely less than 1,000 overall if you include compliance, sales, and so on - especially since a lot of overseas compliance stuff is outsourced to consulting firms, it's not like you have a team of lawyers in every country in the world.
Before there was so much money in the system, we used to run companies that way. Two decades ago, I worked for a company that had tens of millions of users, maintained its own complex nationwide infra (no AWS back then), and had 400 full-time employees. That made coordination problems a lot easier too. We didn't need ten layers of people and project management because there just wasn't that many of us.
When doubling the number of employees can triple your revenue, you do it.
Keeping a website running with high uptime is not the goal. Maximizing revenue and profit is. The extra people aren't waste, they're what drive the incremental imperceptible changes that make these companies profitable.
This seems like a just-so story.
You can see it happen in reverse with X/Twitter.
Did reducing waste affect the user experience or uptime of Twitter? Not really.
But advertising revenues plummeted, because those extra employees were mostly not about the user experience or keeping the website up, they were about servicing the advertisers that brought the company revenue.
I thought advertising revenues plummeted mostly for content/optics/PR reasons, not ad-buyer-facing feature reasons.
Content moderation was an ad-buyer-facing feature. I really see no evidence Musk actually understood that. When he took it away he was all like surprised Pikachu face that advertisers left.
And how much revenue did that company bring in compared to something like Meta?
Maybe there's a correlation there?
I think the person you're replying to is perfectly aware of the correlation, considering it was a primary feature of their comment.
Not really? The main point of their comment is that companies could be much smaller based on their experience at a much smaller company.
I'm implying that big companies couldn't make as much money as they do without all the employees they have.
Their last para seems to acknowledge the correlation, but flips your assumed causal direction. I.e. they seem to be implying that the that excess money causes the complexity.
I am pretty convinced that for most types of day to day work, any perceived improvements from the latest Claude models for example were total placebo. In blind tests and with normal tasks, people would probably have no idea if they're using Opus 4.5 or 4.6.
This has basically been my experience since Sonnet 3.5. I've been working on a personal project on and off with various models and things since then and the biggest difference between then and now is that it will do larger chunks of work than it did before, but the quality of the code is not particularly better, I still have to do a lot of cleanup and it still goes off the rails pretty frequently. I have to do fewer individual prompts, but the time spent reviewing the code takes longer because I also have to mentally process and fix larger chunks of code too
Is it a better user experience now? Yes. Has it boosted my productivity on this project? Absolutely.
But it still needs a ton of hand holding for anything complicated and I still deal with tons of "OK, this bug is fixed now!" followed by manually confirming a bug still exists.
4.6 has been a very, very slight regression for me, but the tradeoff is they've added better compaction - and now larger context windows. That's a reasonable tradeoff for me.
It's because they are getting so good it's impossible to recognize them.
Haiku 4.5 is already so good it's ok for 80% (95%?) of dev tasks.
I must be writing very different software than you, I keep opus on a tight leash and it still comes to the strangest conclusions.
Very possible. Some things work like a charm on first try for me, others you can spell it out again and again. And then yet again. Something to do with training data, obviously.
I've found Haiku to be truly mediocre for working with. If you want a cheap models, the open source ones are much better
I'd agree with you on 4.5 to 4.6, but going from gpt-5 or 4.0 to 4.5 was night and day.
GPT5 added the router, which was def a downgrade. 4.5 was probably the best non-COT model humanity has made. But too expensive to run.
[dead]
Because post 4.0 dropped the sycophancy?
Interesting article, although with so few data points and such a specific time slice it is difficult to draw serious conclusions about the "improvement" of LLM models.
It's notably lacking newer models (4.5 Opus, 4.6 Sonnet) and models from Gemini.
LLMs appear to naturally progress in short leaps followed by longer plateaus, as breakthroughs are developed such as chain-of-thought, mixture-of-experts, sub-agents, etc.
Focusing on flashy breakthroughs hides the issue that bigger models and merge benchmarks rarely translate to reliability in real codebases. For routine merges, subtle regressions and context quirks matter more than headline progress. Unless evals stress nasty scenarios like multi-file renames with tricky conflicts, the numbers are mostly for show. Progress will plateau until someone tunes for the boring, messy cases that waste dev time.
I feel even if the models are stagnating, the tooling around them, and the integrations and harnesses they have are getting significantly more capable (if not always 'better' - the recent vscode update really handicapped them for some reason). Things like the new agent from booking.com or whatever, if it could integrate with all hotels, activities, mapping tools, flight system, etc could be hugely powerful.
Assuming we get no better than opus 4.6, they're very capable. Even if they make up nonsense 5% of the time!
[flagged]
Controversial opinion from a casual user, but state-of-art LLMs now feel to me more intelligent then the average person on the steet. Also explains why training on more average-quality data (if there's any left) is not making improvements.
But LLMs are hamstrung by their harnesses. They are doing the equivalent of providing technical support via phone call: little to no context, and limited to a bidirectional stream of words (tokens). The best agent harnesses have the equivalent of vision-impairment accessibility interfaces, and even those are still subpar.
Heck, giving LLMs time to think was once a groundbreaking idea. Yesterday I saw Claude Code editing a file using shell redirects! It's barbaric.
I expect future improvements to come from harness improvements, especially around sub agents/context rollbacks (to work around the non-linear cost of context) and LLM-aligned "accessibility tools". That, or more synthetic training data.
Steet? Do you mean street? They're smarter in the same way a search engine is smarter.
Yes, "street". Typing from my phone, sorry.
And search engines are narrow tools that can only output copies of its dataset. An LLM is capable of surprisingly novel output, even if the exact level of creativity is heavily debated.
Remixes aren't novel.
Human cultures are remixes all the way down...
If you haven’t seen that “Everything is a Remix” documentary you should. It’s so good.
No they're not
> But LLMs are hamstrung by their harnesses
entirely so. i think anthropic updated something about the compact algorithm recently, and its gone from working well over long times to basically garbage whenever a compact happens
It's so disrespectful to say an LLM is more intelligent than a person on the street. The LLM has nothing at stake, cares not a sausage about the consequences of what it spits out. People have all kinds of pressures, dependants, and personal issues like health. Our thoughts and actions have real consequences. It's so easy to be intelligent when you're the pretend human that gets switched on for five minutes then switched off again.
It's not a value judgement, I'm no misanthrope. But it's a fact or life that we humans must specialize, while LLMs can afford to have "studied" a staggering variety of topics. It's no different than being slower than a car, or weaker than a hydraulic press.
On a different note, LLMs are still not very wise, as displayed by all the prompt attacks and occasional inane responses like walking to the car wash.
On the contrary, it's the machines that have the luxury of specialisation.
I'm faster than a car when scrambling up a mountain. It's easy to be fast when all you do is drive on smooth roads.
I'm stronger than a hydraulic press when carrying a load over a distance. It's easy to be strong when you're limited to a few centimetres of movement.
Intelligence of LLMs is a trick; they are literally trained to sound intelligent. But it's easy to sound intelligent when all you have to do is sound intelligent. A person has to live in the real world, deal with the weather, with feelings, stress, health and, above all, consequences. We don't have the luxury of just being able to sound intelligent, or have wheels that only work on smooth roads. We have to actually be intelligent in our actions. We have to traverse difficult ground, deal with obstacles we've never encountered before. When we make mistakes, people get hurt.
We're probably thinking about this at very different levels. Here's what I meant: I can ask Claude for "a bilingual German-Russian poem about the side effects of the most common drugs used in anesthesia". I would bet my left shoe that if I asked people on the street, no one will do a better job than Claude. And to me, answering questions correctly is a very good metric for intelligence.
We can debate whether that's real intelligence, and whether the question is fair, but this is still a real, measurable capability, that just eight years ago was a pipe dream. This capability is what OP is tracking, and what I believe is impressive but hamstrung by harnesses.
>I would bet my left shoe that if I asked people on the street, no one will do a better job than Claude.
Because it's unrealistic to expect such a niche population to be out and wandering around. What it sounds like you want is something that masquerades at intelligence, but is really alien.
My experience has been that raw “one-shot intelligence” hasn’t improved as dramatically in the last year, but the workflow around the models has improved massively.
When you combine models with:
tool use
planning loops
agents that break tasks into smaller pieces
persistent context / repos
the practical capability jump is huge.
I don't think it's true, but am I alone in wishing it was? My world is disrupted somewhat but so far I don't think we have a thing that upends our way of life completely yet. If it stayed exactly this good I'd be pretty content.
I agree with your sentiment, but I think we've yet to see the full application of the current technology. (Even if LLMs themselves don't improve, there's significant opportunity for people to use it in ways not currently being done)
The issue with llm’s is trust.
I don’t see that ever going away. Humans have learned to trust other humans over a large time scale with rules in place to control behaviour.
That's a big problem with very specific manifestations. My startup helps customers handle regulatory compliance, also by forwarding complex questions to a pool of consultants.
We've compared now more than a hundred replies to that of GPT Pro, and the quality is roughly the same. Sometimes a little worse, sometimes a little better. Always more detailed. Never unacceptable.
But how to convince our customers that we have the right technology and know how to use it appropriately? We're trying, but it's not easy.
Part of that's accountability. In the event of the LLM producing rubbish, as rare as it may be, who is accountable? There is not a person and her reputation attached to it.
Yup exactly.
Being able to hold someone liable for a F up has been how we have been able to function as a society and get to where we are today.
LLM plus human should be better than either standalone. You won’t be able to make as much money scaling out, though.
You won’t be able to scale out and make as much money though. But surely you’re not only concerned about profit, right? What’s the point of life if you’re just trying to get rich.
When the dust settles, for example if LLM's were to stop improving today, we would come to learn their exact capabilities, what they can do reliably and what they can't.
Once we know what they can do well and how to get them to do it well, and what they can't, you could say we "trust" them to do the first category well and just stop trying to get it to do the second category.
This feeds the adoption problem, though: a lot of companies are thinking "why settle for the current models when even the vendors are saying the models in six months will be exponentially better? Let's let the early adopters work out the bugs and move when these things are more stable"
LLMs are random by nature, they might something done one time but miserably fail the next
I think we're getting to a point where LLM randomness is relevant to someone writing a white paper on LLMs, but not as relevant to consumers of them. Yes the technology uses randomness, but the quality of response somehow still seems very consistent and predictable in 2026.
Yeah, and we will continue to learn to use them where the amount of random failure is acceptable or can be mitigated or reduced with additional tools.
I think what happened with static image generation is happening with LLMs. Basically the tools around are becoming better, but all the AI improvements stall, the error rate stay the same (but external tools curate the results so it won't be noticeable if you don't run your own model), the accuracy is still slightly improving, but slower and slower, and never reach the 'perfect' point. Basically stablediffusion early 2025
Image quality has improved a lot in recent months thanks to better models. The ability of people to notice these improvements is plateauing because they are not trained to spot artifacts, which are becoming more obscure.
Yes, slight increase in that kind of accuracy. And newer models still generate absurd stuff. Ask for an historical picture, like 'a London market in the 18th century', and it is still as historically wrong as it was 2 years ago. It is useful for fantasy/sci-fi though, I use them a lot. But I don't see the point of newer models since late 2024.
I have no idea how a London market in the 18th century should look like so I cannot challenge that but recent models like nano banana ones, Z-image (on a lesser extend) can generate images that are essentially indistinguishable from actual stock photos, this wasn't true for late 2024 models, with wonky backgrounds, too smooth skins and general lack of details (the classic AI-look that AI images had).
Mmm, I mostly generate scenes, not people, and when I do, it's always in a not photorealistic way. It is true that newer models have more details, I think that I put that into 'slight accuracy improvements', is it really major? Or is it mostly for close up people/animals and it is likely that I just didn't notice?
It will be wrong on a lot of details. Basically you would get a market scene that feels 18th century gb, but will use 18th century russian/french/Austrian details, or 19th century/20th century British artefacts, or a mix of both. And the further you go from western places, the higher the error rate is. Basically generating fiction scenes. That's pretty much my usecase, so that's fine, but I won't ever use it to illustrate a historical TTrpg.
That's an interesting claim, but I don't see it in my own work. They have got better but it's very hard to quantify. I just find myself editing their work much less these days (currently using GPT 5.4).
Without meaning to sound dismissive, because I'm really not intending to, there's also the possibility that you've gotten worse after enough time using them. You're treating yourself as a constant in this, but man cannot walk in the same river twice.
This is such a silly response when "You've gotten better at using them and know how to work around their flaws now." is right there and seems a lot more plausible.
That's a possibility, but I doubt it. I've been programming for 35 years and know what I like in code. I've also previously maintained a long review prompt in which I tell the models all the ways in which they get things wrong and to go look for/fix those problems. But those review passes now don't take as long because there are fewer such problems to begin with.
In particular GPT 5.4 is much better at not duplicating code unnecessarily. It'll take the time to refactor, to search for pre-existing utility functions, etc.
The problem with evals is the underlying rubric will always be either subjective, or a quantitative score based on something that is likely now baked into the training set directly.
You kind of have to go on "feels" for a lot of this.
Yeah same, and all my coworkers feel the same.
Most of us have been coding for ages. I actually find it really odd people keep trying to disprove things that are relatively obvious with LLMs
I gave up on trying months ago, you can see the timeline on top of https://fabien.benetou.fr/Content/SelfHostingArtificialIntel...
Truth is I'm probably wrong. I should keep on testing ... but at the same time I precisely gave up because I didn't think the trend was fast enough to keep on investing on checking it so frequently. Now I just read this kind of post, ask around (mainly arguing with comments asking for genuine examples that should be "surprising" and kept on being disappointed) and that seems to be enough for a proxy.
I should though, as I mentioned in another comment, keep track of failed attempts.
PS: I check solely on self-hosted models (even if not on my machine but least on machines I could setup) because I do NOT trust the scaffolding around proprietary closed sources models. I can't verify that nobody is in the loop.
I reckon LLM merge rates will go up, but not necessarily due to quality improvements. Instead I think maintainers will just become fatigued. The amount of code I'm expected to review now is way higher than before. And while I'm reviewing you know more is being generated. I'm sure I've let through more crap due to this fatigue attack on me.
As they become more capable peoples commits will also become more ambitious.
So I’d say fairly flat commit acceptance numbers make sense even in the context of improving LLMs
Indeed. Why is this post down voted? There’s always trade-offs taking place, it’s good to call them out.
Given that it is the general consensus that a step function occurred with Opus 4.5/4.6 only 3 months ago - it seems like an insane omission.
This has been the general consensus for about three years now. "Drastic increases in capability have happened the last 3-6 months" have been a constant refrain.
Without any data from the study past September I think its not unreasonable, if you want to make an argument based on evidence.
For me personally, I agree with you, I'm really seeing it as well.
> "Drastic increases in capability have happened the last 3-6 months" have been a constant refrain.
well, yeah. because that's been the experience for many people.
3 years ago, trying to use ChatGPT 3.5 for coding tasks was more of a gimmick than anything else, and was basically useless for helping me with my job.
today, agentic Opus 4.6 provides more value to me than probably 2 more human engineers on my team would
Yep this has been my experience too.
I tried GPT3.5 for translating code from typescript to rust. It made many mistakes in rust. It couldn't fix borrow checker issues. The context was so small that I could only feed it small amounts of my program at a time. It also introduced new bugs into the algorithm.
Yesterday I had an idea for a simple macos app I wanted. I prompted claude code. It programmed the whole thing start to finish in 10 minutes, no problem. I asked it to optimize the program using a technique I came up with, and it did. I asked it to make a web version and it did. (Though for some reason, the web version needed several rounds of "it doesn't work, here's the console output").
I'm slowly coming to terms with the idea that my job is fundamentally changing. I can get way more done by prompting claude than I can by writing the code myself.
>well, yeah. because that's been the experience for many people.
Yes but this blogpost argues that at least over the course of 2024 to the end of 2025, those people were mistaken.
There's a consensus that SOMETHING changed with Opus 4.5. It might have been the "merge rates" metric, it might have not.
I'm certainly getting faster and cleaner-looking solutions for certain issues on Opus 4.6 than I was 5 months ago, but I'm not sure about the ability to solve (or even weigh in) the actual hard stuff, i.e. the stuff I'm paid for.
And I'm definitely not sure about the supposed big step between 4.5 and 4.6. I'm literally not seeing any.
> This means llms have not improved in their programming abilities for over a year. Isn’t that wild? Why is nobody talking about this?
Because it's not true. They have improved tremendously in the last year, but it looks like they've hit a wall in the last 3 months. Still seeing some improvements but mostly in skills and token use optimization.
> but mostly in skills and token use optimization.
I have heard rumors that token use optimization has been a recent focus to try to tidy up the financials of these companies before they IPO. take that with a grain of salt though
After only 3 months (!) you can claim a plateau, but not a wall.
Benchmaxxing aside, if you are using those tools for programming on a regular basis it should be self-evident that they are improving. I find it very hard to believe that someone using LLMs today vs what was available one year ago (Claude Code released Feb 2025) would have any difficulty answering this question.
I think it is important to try to find more rigorous things to test than the general sentiment of the people using the tools. If only because the more benchmarks we have the more we can improve models without regressions. METR is asking a really interesting question here, "are models improving at making one shot PRs?". The answer seems to be, yes, but slower than benchmarks suggest, if you look at the pass rate of different versions of Claude Sonnet. A reasonable answer is "you're not supposed to use them by making one shot PRs", but then ideally we would need to have some kind of standarized test for the ability of models to incorporate feedback and evolve PRs.
You really can't model these 5 data points with a linear regression or a step function. The models are of different sizes / use cases, and from two different labs. I feel like what we've observed generally is that different labs releasing similarly sized models at similar times are generally pretty similar.
I think the only reasonable thing to read into is Sonnet 3.5 -> 3.7 -> 4.5. But yeah, you just can't draw a line through this thing.
I will die on the hill that LLMs are getting better, particularly Anthropic's releases since December. But I can't point at a graph to prove that, I'm just drawing on my personal experience. I do use Claude Code though, so I think a large part of the improvement comes from the harness.
They are getting better, but they are also hitting diminishing returns.
There's only so much data to train on, and we are unlikely to see giant leaps in performance as we did in 2023/2024.
2026-27 will be the years of primarily ecosystem/agentic improvements and reducing costs.
These studies are always really hard to judge the efficacy of. I would say though the most surprising thing to me about LLMs in the past year is how many people got hyped about the Opus 4.5 release. Having used Claude Code at work since it was released I haven't really noticed any step changes in improvement. Maybe that's because I've never tried to use it to one shot things?
Regardless I'm more inclined to believe that 4.5 was the point that people started using it after having given up on copy/pasting output in 2024. If you're going from chat to agentic level of interaction it's going to feel like a leap.
I used it with Sonnet 4.0 a lot, and there was vastly more back-and-forth and correction of "dumb" things, such as forgetting to add "using" statements in C# files.
I don't know if it's model, or harness improvements, or inbuilt-memory or all of the above, but it often has a step where it'll check itself that is done now before trying to build and getting an inevitable failure.
Those small things add up to a much smoother and richer experience today compared to 6 months ago.
Nah, pre 4.5 it was not comfortable to use agentic coding.
I've been able to supercharge a hobby project of mine over the last couple months using Opus 4.6 in claude code. I had to collaborate and write code still, but claude did like 75% of the work to add meaningful new features to an iOS/Android native mobile app, including Live Activities which is so overly complicated i would not have been able to figure that out. I have it running in a folder that contains both my back end api (express) and my mobile app (nativescript), so it does back end and front end work simultaneously to support new features. this wasnt possible 8 months ago.
I have a similar experience. My hobby project was put on hold after a burnout and lack of motivation. I got a big burst of energy back when I started implementing some long desired features quickly with these new models. I was able to get the project to the point of what I consider is maturity. I did in a month during free time the kind of work that would have burned me up in a good six months fulltime.
I had this suspicion for a while I think we just got way better in harnessing not the models actual reasoning
So we got better in giving it the right context and tools to do the stuff we need to do but not the actual thinking improvements
I feel like anyone used AI coding tools before 11/25 and after 1/26 (with frontier models) will say there has been a massive jump in, there is a difference between whether LLM can do a specific task or pass some arguably arbitrary checks by maintainers vs. what the are capable of.
We still have tons of gaps about how to build and maintain code with AI, but LLM themselves getting better at an unbelievable pace, even with this kind of data analysis I’m surprised anyone can even question it.
> I feel like anyone used AI coding tools before 11/25 and after 1/26 (with frontier models) will say there has been a massive jump in, there is a difference between whether LLM can do a specific task or pass some arguably arbitrary checks by maintainers vs. what the are capable of.
How much of that is the model and how much of that is the tooling built around it? Also why is the tooling, specifically Claude Code, so buggy?
90% model if not more, look at terminal benchmark terminus tool, that mostly proves it
From my personal experience, they have gotten better, but they haven’t unlocked any new capabilities. They’ve just improved at what I was already using them for.
At the end of the day they still produce code that I need to manually review and fully understand before merging. Usually with a session of back-and-forth prompting or manual edits by me.
That was true 2 years ago, and it’s true now (except 2 years ago I was copy/pasting from the browser chat window and we have some nicer IDE integration now).
Well, on one hand they lack new data. Lot's of new code came out of an LLM, so it feeds back.
On the other hand, LLMs tend to go for an average by their nature (if you squint enough). What's more common in their training data, it's more common in the output, so getting them better without fundamental changes, requires one to improve the training data on average too which is hard.
What did improve a lot is the tooling around them. That's gotten way better.
> Well, on one hand they lack new data. Lot's of new code came out of an LLM, so it feeds back.
Supposedly model curation is a Big Deal at Big AI, and they're especially concerned about Ouroboros effects and poisoned data. Also people are still contributing to open source and open sourcing new projects, something that should have slowed to trickle by 2023, once it became clear that from now on, you're just providing the fuel for the machines that will ultimately render you unemployable (or less employable), and that these machines will completely disregard your license terms, including those of the most permissive licenses that seek only attribution, and that you're doing all of this for free.
As someone who builds with LLMs daily without being a developer, I notice quality differences more in practical output than benchmarks. Claude handles complex multi-step tasks better in my experience, but consistency is still the biggest challenge – same prompt can give very different results day to day.
Data is missing on this chart.
It's my experience that opus 4, and then, particularly, 4.5, in Claude code, are head and shoulders above the competition.
I wrote an agentic coder years ago and it yielded trash. (Tried to make it do then what kiro does today).
The models are better. Now, caveat - I don't use anything but opus for coding - Sonnet doesn't do the trick. My experience with Codex and Gemini is that their top models are as good as Sonnet for coding...
I was trying to do something yestesrday and Claude was keep messing it up, after like an hour i realized the model somehow switched to sonet, opus 4.6 is crazy good. It’s very obvious in practice.
Although I feel like for chasing bugs and big systems codex is even better
How the "costant function" result fits the data points better than a slope that has two parameters instead of one.
Cross-validation. The slope overfits when the test set is included from the data the model is fitted on.
Tangential: I've found that having an LLM recreate the full file, with changes appllied, is less mistake-prone, than producing a patch. I wonder if anyone else came to this conclusion too.
>fischer warned us against eyeballing plots proceeds to eyeball it with an arbitrary function
There was a long flat line before the step, models improve, but PR pass rate without human intervention is inherently a staircase function
Yesterday I asked a frontier model to help generate a report. It said great, it can do that, and output a table. I asked it to evaluate its prompt compliance in the result. It concluded that it had failed on every requirement. I asked why it had expressed such confidence, was it analagous to narcissism or psycopathy? It said no, and then said that if I just had to anthropomorphize it, I should think of it as a brilliant friend with severe frontal lobe brain damage.
That actually helps.
I agree completely. I haven't noticed much improvement in coding ability in the last year. I'm using frontier models.
What's been the game changer are tools like Claude Code. Automatic agentic tool loops purpose built for coding. This is what I have seen as the impetus for mainstream adoption rather than noticeable improvements in ability.
My anecdotal experience is rather different.
I write a lot of C++ and QML code. Codex 5.3, only released in Feb, is the the first model I've used that would regularly generate code that passes my 25 years expert smell test and has turned generative coding from a timesap/nuisance into a tool I can somewhat rely on not to set me back.
Claude still wasn't quite there at the time, but I haven't tried 4.6 yet.
QML is a declarative-first markup language that is a superset of the JavaScript syntax. It's niche and doesn't have a giant amount of training data in the corpus. Codex 5.3 is the first model that doesn't super botch it or prefers to write reams of procedural JS embeds (yes, after steering). Much reduced is also the tendency to go overboard on spamming everything with clouds of helper functions/methods in both C++ and QML. It knows when to stop, so to speak, and is either trained or able to reason toward a more idiomatic ideal, with far less explicit instruction / AGENTS.md wrangling.
It's a huge difference. It might be the result of very specific optimization, or perhaps simultaneous advancements in the harness play a bigger role, but in my books my kneck of the woods (or place on the long tail) only really came online in 2026 as far as LLMs are concerned.
As a Qt C++ and QML developer myself[1], Opus 4.6 thinking is much better than any other model I've tested (Codex 5.3/GPT 5.4/Gemini 3.1 Pro).
[1] https://rubymamistvalove.com/block-editor
Maybe n=1, but I disagree? I notice that Sonnet 4.6 follows instructions much better than 4.5 and it generates code much closer to our already in-place production code.
It's just a point release and it isn't a significant upgrade in terms of features or capabilities, but it works... better for me.
Are you using a tool like Claude Code or Codex or windsurf? I ask because I've found their ability to pull in relevant context improves tasks in exactly the way you're describing.
My own experience is that some things get better and some things get worse in perceived quality at the micro-level on each point release. i.e. 4.5->4.6
fwiw the merge rate metric itself might be misleading. most real codebases have implicit conventions and architectural patterns that aren't captured in the issue description, so even if the model writes correct code it might not match what the maintainer actually wanted. imo the bigger signal is how much back-and-forth it takes before merging, not whether the first attempt lands cleanly.
In my niche the Opus 4.6 has been a game changer. In comparison all other LLMs look stupid. I am considering cancelling all other subscriptions.
Yeah I'm not buying the last bit about lower MSE with one term in the model vs two (Brier with one outcome category is MSE of the probabilities). That's the sort of thing that would make me go dig to find where I fucked up the calculation.
With one term it gets more robust in the face of excluding endpoints when constructing the jackknife train/test split, I think. But you're right, it does sound fishy.
What the post is describing is just ANOVA. If removing a category improves the overall fit then fitting the two terms independently has the same optimal solution (with the two independent terms found to be identical). MSE never increases when adding a category.
This is why you have to reach to things that penalize adding parameters to models when running model comparisons.
No, the post is doing cross-validation to test predictive power directly. The error will not decompose as neatly then.
Why would they do that and where do you see evidence they did?
Because it's a direct way to measure predictive power, and it says so: "We’ll use leave-one-out cross-validation"
How do they know? Not everybody includes to coauthored by Claude. I certainly don't.
If you look at a separate trend for the smaller Sonnet models, you can see a rapid trend
3.7 to 4.5 looks pretty flat here.
Even if one-shot LLM performance has plateaued (which I'm not convinced this data shows given omission of recent models that are widely claimed to be better) that missing the point that I see in my own work. The improved tooling and agent-based approaches that I'm using now make the LLM one-shot performance only a small part of the puzzle in terms of how AI tools have accelerated the time from idea to decent code. For instance the planning dialogs I now have with Claude are an important part of what's speeding things up for me. Also, the iterative use of AI to identify, track, and take care of small coding tasks (none of which are particularly challenging in terms of benchmarks) is simply more effective. Could this all have been done with the LLM engines of late 2024. Perhaps, but I think the fine-tuning (and conceivably the system prompts) that make the current LLM's more effective at agent-centered workflows (including tool-use) are a big part of it. One-shot task performance at challenging tasks is an interesting, certainly foundational, metric. But I don't think it captures the important advances I see in how LLM's have gotten better over the last year in ways that actually matter to me. I rarely have a well-defined programming challenge and the obligation to solve it in a single-shot.
>This means the step function has more predictive power (“fits better”) than the linear slope. For fun, we can also fit a function that is completely constant across the entire timespan. That happens to get the best Brier score.
I mean, sure. but it's obvious in that graph that the single openai model is dragging down the right side. Wouldn't it be better to just stick to analyzing models from only one lab so that this was showing change over time rather than differences between models?
From the METR study (https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs...):
>To study how agent success on benchmark tasks relates to real-world usefulness, we had 4 active maintainers from 3 SWE-bench Verified repositories review 296 AI-generated pull requests (PRs). We had maintainers (hypothetically) accept or request changes for patches as well as provide the core reason they were requesting changes: core functionality failure, patch breaks other code or code quality issues.
I would also advise taking a look at the rejection reasons for the PRs. For example, Figure 5 shows two rejections for "code quality" because of (and I quote) "looks like a useless AI slop comment." This is something models still do, but that is also very easily fixable. I think in that case the issue is that the level of comment wanted hasn't been properly formalized in the repo and the model hasn't been able to deduce it from the context it had.
As for the article, I think mixing all models together doesn't make sense. For example, maybe a slope describe the increasing Claude Sonnet better than a step function.
No Gemini. No Opus 4.5. No GPT codex.
As they said, ragebait used to be believable.
LLM's have 100% gotten better, but it's hard to say if it's "intrinsically better", if that makes sense.
> OpenAI’s leading researchers have not completed a successful full-scale pre-training run that was broadly deployed for a new frontier model since GPT-4o in May 2024 [1]
That's evidence against "intrinsically better". They've also trained on the entire internet - we only have 1 internet, so.
However, late 2024 was the introduction of o1 and early 2025 was Deepseek R1 and o3. These were definitely significant reasoning models - the introduction of test time compute and significant RL pipelines were here.
Mid 2025 was when they really started getting integrated with tool calling.
Late 2025 is when they really started to become agentic and integrate with the CLI pretty well (at least for me). For example, codex would at least try and run some smoke tests for itself to test its code.
In early 2026, the trend now appears to be harness engineering - as opposed to "context engineering" in 2025, where we had to preciously babysit 1 model's context, we make it both easier to rebuild context (classic CS trick btw: rebooting is easier than restoring stale state [2]) and really lean into raw cli tool calling, subagents, etc.
[1] https://newsletter.semianalysis.com/p/tpuv7-google-takes-a-s...
[2] https://en.wikipedia.org/wiki/Kernel_panic
FWIW, AI programming has still been as frustrating as it was when it was just TTC in 2025. Maybe because I don't have the "full harness" but it still has programming styles embedded such as silent fallback values, overly defensive programming, etc. which are obvoiusly gleaned from the desire to just pass all tests, rather than truly good programming design. I've been able to do more, but I have to review more slop... also the agents are really unpleasant to work with, if you're trying to have any reasonable conversation with them and not just delegate to them. It's as if they think the entire world revolves around them, and all information from the operator is BS, if you try and open a proper 2-way channel.
It seems like 2026 will go full zoom with AI tooling because the goal is to replace devs, but hopefully AI agents become actually nice to work with. Not sycophantic, but not passively aggressively arrogant either.
Anecdotally, I haven't seen any real improvement from the AI tools I leverage. They're all good-ish at what they do, but all still lie occasionally, and all need babysitting.
I also wonder how much of the jump in early 2025 comes from cultural acceptance by devs, rather than an improvement in the tools themselves.
I think I'm coming to the same conclusion Gpt-3 to 5.3 have had real tangible but incremental improvements with quite diminishing returns.
Perhaps we won't see a phase change like improvement as we did from gpt-2 through to 3 until there is several more orders of magnitude parameters and/or training. Perhaps we will never see it again!
What is getting rapidly better is scaffolding but this seems to be more about understanding and building tools around LLMs than the LLMs themselves improving.
I'm still excited about AI but not constantly hyped to the rafters as some.
I think it depends on what you're using it for. If it is a simple kubernetes config then the model doesn't matter too much. Contract that with writing the scenario for a backtest for an algo that trades on a venue: it is not the same complexity and the basic models are terrible. I've had it tell me that it has added tests to find that they're just stubs! Opus seems to be getting there, but on more complex tasks the others are a complete waste of time.
> If it is a simple kubernetes config then the model doesn't matter too much
I guess at least this person https://www.tomshardware.com/tech-industry/artificial-intell... might disagree. I think already to know what Kubernetes even is requires quite a bit of knowledge. Using a tool that manipulate its configuration files IN PRODUCTION without risking data loss is another ball game entirely.
It’s better pre and post training + better harnessing
[flagged]
Dude. Please stop the fucking LLM comments. GTFO. And actually explicitly against the rules now. @dang
[flagged]
[flagged]
[dead]
[flagged]
[dead]
[dead]
[flagged]
This means llms have not improved in their programming abilities for over a year. Isn’t that wild? Why is nobody talking about this?
Because hype makes money.