I am big on reproducibility (nix aficionado) and determinism (flagging test failures are a red-alert, all-hands-on-deck situation in my world) and correctness.
I am also big on testing (the correct things). And nine-nines (big on Elixir).
And... I'm also big on agent-assisted dev. Which requires pretty much every check in the book to stay productive in. And that's fine to me. I've seen bugs that I wouldn't have made myself. And I've also seen my own bugs fixed. They've all gotten fixed in short order. I don't see why this is a problem.
Raise your personal standards.
Thing is, the unreliable-software situation was already untenable before agents (in poor hands) made it worse.
I don’t think the author (or many people) doubt that one can (and some will) find a way that does not “suck”
But it’s pretty clear that most people are not. For whatever reasons (mgmt pressure, trying to get ahead, skill issues, etc) they half ass it, accept the 10% (silent) fail rate and blame the bad outcomes on the AI as if that absolves them. Or, adopt the attitude that 10% fail is fine, and people who say otherwise are being picky, or are anti-ai luddites or whatever. You should accept that things will suck.
If you took a bad but functional AI generated service and transported it back to 2018 it would have been at worst just mediocre. People do seem forget how dreadful devslop was in the past. I'd take an AI generated mess to disentangle every time over a spaghetti codebase that grew organically in the hands of careless managers.
As far a I can tell, most didn’t bother to see if it worked for the system,p. They just needed to have it compile in their system and then they’d call it done.
Volume has nothing to do with it, this is discussing code quality. But if you mind me saying so, blame this ridiculous amount of codeslop volume on greedy managers and corpocrats. Developers are artists, they usually ship shitcode when they're under pressure
Are the managers of these hypothetical places still interested in keeping the high standards and good design? Enshitification isn't inherent to the technology, if this is what you are deriving from the counterexample, it's a business strategy my dude.
> This is not new, but LLMs further tilt the balance
You speak as if llms had their own minds. Every time anyone talks about AI doing this or that they further reinforce this idea that there isn't a person behind all this. There's always someone watching.
With that said, when you say that llms tilt the balance, who specifically do you say that's driving llms to do that?
It means an ambitious manager can take on more work by having the team slop out. They move up (very effective leader!), standards fall, other managers have to match the changes. The bill comes due years later after they have moved on (and up).
But as more and more things depend on increasingly deep software stacks, everything goes to utter shit if the individual systems don't become more reliable.
Yeah, a good reason to be touchy about AI is that it tips the balance of power to lazy people who don't want to work or think. On the scale of our whole society. Imagining the ideal responsible use by most others is folly. No matter how responsible and conscientious you, the reader, are with your use of it.
> Thing is, the unreliable-software situation was already untenable before agents (in poor hands) made it worse.
I have definitely seen more bafflingly-poor OSS software that just plain doesn't work frequently now than before.
But it's mostly software that wouldn't have existed before because it's trying to do super-niche things. So on the "hobby" side of things, whatever.
But from a "trying to develop software as a business that you want to be a going concern," quality from people who should know better is less tenable than it used to be.
Thing is, the unreliable-software situation was already untenable before agents (in poor hands) made it worse.
Yes but that's the big thing, now isn't it? These are nice tools, used wisely. But their unwise use, oh boy...
The problem is one needs to be in a situation where the incentive is towards quality rather than speed. But that situation rather rare now - thirty years ago, Microsoft won the office wars with crap that had features. And nothing has fundamentally changed in web development since the LPad crisis.
The problem is those companies whose incentive is to allow bugs where it's the involuntary users who suffer will bite you no matter what quality you make your own software.
Excellent post. People always defend agentic/LLM-driven development by saying, "Well it's good enough", or "It works most of the time."
That may be tolerable for some user-facing app. But what if we start normalizing failures in the libraries, the infrastructure, and the compilers? Everything descends into a mess of unreliability, and that slows EVERYTHING and EVERYONE down.
> People always defend agentic/LLM-driven development by saying, "Well it's good enough", or "It works most of the time."
The main argument for LLM-driven development is much simpler: "It will get better".
The current state of LLM coding is about a year old. Imagine if we dismissed human coding efforts after a year. Rust, Python2 -> Python3 transition, Python type checking, Windows, C++, … nothing of that was done in a year and emerged in perfection in the first year. Everything takes ages to mature into a usable product. LLM coding is still in the "throw mud at the wall and see what sticks" stage, give it some more years and see how it will develop and what approaches actually work at. For the time being, LLMs are just the most useful development tool in the history of development tools, that's a pretty solid start in such a short time.
Banking/Finance is the one industry I've seen push back against this type of thinking. Transactions must be handled in a perfect and repeatable way, or the system is unusable as far as the company is concerned.
There's definitely still AI/LLM integration happening, but is kept out of specific areas of the business.
Same with aviation and safety. I honestly believe all programmers should, early in their careers, do a brief "tour of duty" in an industry where the stakes are high and "good enough" isn't good enough. You might not choose to make it your entire career, but at least you're exposed to the discipline, however briefly. Most software developers today have never in their lives worked on a project where defects were taken seriously and where there was process and documentation designed to reduce their occurrence.
To be fair, I'm inclined to believe that most of this pressure to lean (if not rely) on LLMs is not coming from programmers, but rather from middle and upper management: banks are resisting this not because their programmers are holding some kind of line, but because management is not giving-in to the LLM siren call. So that tour of duty, while a good idea, would be more about teaching management skills than anything else.
The normalisation of inexplicable failures is, in my opinion, the consequence of programmers being unable to refuse their managers who have fallen for the LLM siren call, lest they lose their jobs.
I think the failures matter in non-sensitive environments as well albeit with a different threshold.
If you randomly screw up customer orders (think of DoorDash or an online shop or Airbnb). They lose trust in you and you lose your business to the competition. Going happy go lucky and being irresponsible in the business can bankrupt most* businesses.
* well, of course except the criminal empires which are bailed out by our tax money.
I've been seeing the very opposite. Fintech companies treating design of financial systems with the casualness of a frontend aesthetic change. And tons of business people integrating their vibecoded POCs with financially sensitive data sources.
However, we are not (yet) building probabilistic libraries right? Even if AI is used, the code it builds is still deterministic and will fail in a way that can be fixed.
Not unlike legacy code, where the original developers are long gone and no one knows when that system breaking bug is about to wake up.
Even in the best of cases tho, fully human written and reviewed code will still fail eventually.
So I would say we are not normalizing failures (yet) but rather normalizing legacy.
We've been using probabilistic algorithms a long time now, also known as randomised algorithms. You probably won't find a program that is not using a hash table.
Compilers actually actively try to stay deterministic in their output.
It's also worth noting that LLM are fairly well suited to finding and documenting "load bearing" legacy bugs (when set out to do exactly that.)
I know because I've spent the better part of a year having them do exactly this as a precursor to rewrites in more perfomant tech stacks. Once this research is done, it's a fairly safe and mechanical translation job.
Even if AI is used, the code it builds is still deterministic and will fail in a way that can be fixed.
Say that with confidence now. You'll say with shock and incredulity later, "It can be fix? Right?!?!" (Cue some meme). Reproducing bugs is often hard for well designed systems. For randomly designed systems, it's a nightmare.
Fixing software is hard and fixing bad software is harder and fixing software where you didn't even think at the time how to make it maintainable is hardest.
Legacy systems are usually systems that were well designed at the time. That's why it's worth fixing them now.
--> And none of this is saying AI designed systems are bad. But if the only criteria is "code is deterministic by default, it works and we can fix problems later" then AI seems likely a terribleness accelerator.
Code written with AI can be fixed, if well designed, they are not mutually exclusive.
Also legacy systems were absolutely not always well designed, I've picked up a fair share of systems that were absolutely not well thought out, usually due to time pressure, or any other context specific reasons.
I do believe we are creating legacy systems faster now, and it's because teams are unable to keep up with the speed of changes, not because their system is not meant to be maintainable.
My point remains tho, using AI doesn't have to mean normalizing failures.
Write tests first. Have agent iterate until they are satisfied.
The point is that it boils down to writing the tests correctly, regardless of who is implementing the actual code. Hand-written code without test coverage has the same problems as AI generated code.
Counterpoint: software has always had bugs. People are fallible.
Perhaps designing with assumptions that the entire stack is imperfect is the route to higher quality, along the lines of chaos monkey, which randomly kills processes on production boxes… which ensures resilience is real.
Even without AI our tech stack is so deep it’s hard to imagine every feature at every layer being provably correct.
> This leads to a normalization of inexplicability.
It’s also tightly connected to a normalization of lack of accountability.
> This isn't "getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem" -- you still have to do the hard part.
This is probably losing the younger portion of the audience by now. ;)
> When a button breaks on a website, I have a model about what should have happened. Somewhere a contract got broken. [...] I might not have access to debug just an HTTP status 500, but I expect there to be somebody whose job is to understand why the endpoint is 500ing. The ownership is well-defined albeit opaque³.
> For many users, however, the actual experience is roughly just "stupid thing sucks." Software already feels capricious; more failures just change the rate of frustration.
I am betting author does not use cloud services much. It is not just "users", it's developers as well. Github is returning 5xx? AWS service does not work? Your email did not get delivered? Nothing we (developers) can do, "stupid thing sucks".
We see this fatalistic attitude all the time in software. "Bugs are inevitable." No, they aren't! Bugs are a choice. Almost all companies choose bugs because "no bugs" is too expensive. It's sometimes the right choice but we need to acknowledge that it's a choice and not some natural property of software. Imagine if people who built bridges or airplanes thought "bridge collapses and airplane crashes are inevitable, no way to solve it."
I’ve had several occasions where I would build a feature “properly” with defensive guardrails and handling of edge cases “that will never happen (but are technically possible)” and then you run into the coworker that says “it’s overengineered. YAGNI. Just do the one-line fix”
But...were they wrong? There is a difference between "edge case" and "cannot ever happen". I've seen this trend a lot lately where Claude suggests a lot of extra code to handle things that cannot possibly happen. Unless you ask it to, it won't trace the flow of data through the app to confirm the edge case actually exists, if local conditions appear to allow it. Then the developer lets it implement this crap without asking that one question, and I have to say YAGNI in the review.
I've seen so many software bugs in my career where when you traced execution, you'd find the code crashing in a block with the comment //this can never happen
"Confidence scores" have always implied an anthopocentric meaning that doesn't exist. An algorithm doesn't have "confidence" in the way that a person has confidence, but as soon you put something with that name in front of a business person they assume the number is always a meaningful "letter grade curve" or "universal percentage". I still believe so much that the old quote to "there's lies, damned lies, and then statistics" remains a key to understanding so much why ML is leading to dumb outcomes versus hype. People don't understand statistics, so machines that produce nothing but statistics especially confuse people. (I feel this applies to LLMs as well.)
The "normalization of inexplicability" is indeed infuriating. It has always been bad when it comes to computer software, and it's increasingly creeping into other consumer products that depend on embedded software.
I bought a new electric car recently. For the most part I've been quite happy with it. Shortly after I bought it, it started popping up a warning message saying "check EV system" every time I started it. By the time I brought it into the dealership, the warning had gone away, and the technician just told me something to the effect of "eh, I guess it just does that sometimes, let us know if it happens again." Hardware fault? Software bug? Who can say?
Like most modern cars, it has connectivity and Google Maps built into the infotainment system. The vast majority of the time, it works fine. Sometimes it says it has no connectivity (meaning no traffic data and suboptimal routes) for the duration of a drive, even in areas with a strong cell signal where it normally works fine. Sometimes the car says it has connectivity, but Google Maps still thinks it's offline. Sometimes Maps will actually load and display a route, but the "start navigation" button just spins forever as though it's still waiting for something. Are these related issues? Is there a common cause that might be fixable? Who can say?
(Conveniently enough, the warranty specifically does not cover any failures of software or firmware to operate correctly.)
My Kia Niro EV, which I'm otherwise quite happy with, has an occasional failure mode where it locks out the ignition for an arbitrary amount of time (generally 15 minutes). It's sometimes, but not exclusively, triggered by scheduled departure turning on the AC. Kia's response is a big shrug...
I think some of it is an unintended consequence of the backlash against bad or puzzling error dialog boxes in '80s and '90s software. A lot of people seem to have concluded that the easy solution to the challenges of error reporting was just to fail silently instead roll_safe.jpg .
These are some of the most annoying messages a piece of software can possibly display, and I hate them with a passion!
It's like this with a lot of commercial software, where designers don't want to scare users with complicated (useful) error messages, so you just get a "whoopsie! Something went wrong".
I'd much rather have an error message that I _don't_ understand over an error message I _can't_ understand.
In the case of cars, it used to be that a generic "Check engine" light showed up on your dash, and you needed an external tool to ask the computer what caused it to light up, but why would that be nessecary in a modern vehicle? You've got a large, high resolution display (or more!) on the CAN bus, why can't it show the full problem so you could take a picture right as it happens and send it to your dealer for instant diagnosis?
I remember twenty years ago trouble-shooting my father's Internet connections and not being able to discover the misconfiguration. Once I called the technician and they knew nothing but "wade down to this dialogue and choose 'reset'". That didn't work either - but it was a distinct moment in me noticing the Internet and software becoming opaque. And it was only gotten - sometimes those moments of opaque failure are transitory but sometimes they stay indefinitely. Currently, my android phone can't see group texts created by iphones. I'm thankfully semi-retired enough for this to be a blessing but it could a crisis for someone else (I know a couple that fixes the problem temporarily but a text isn't something you want to spend hours debugging to get if you need it).
But my point - this has been increasing for quite a while and it's something like the evil twin of the "it just works" school of design. Because when the default expectation is "just working", the failure mode is "it just doesn't work". I mean, the default assumption that users are morons has de facto made us all morons since we can only deal with the "interface made for morons" and makes failure into opaque failure.
It's world the developers know "all abstractions are leaky abstraction" but where developers still won't give users (or even downstream developers) tools to deal with the leaks 'cause a simplistic interface matters more.
"When there's no emergency exit, there can't be an emergency, can there?"
Late Douglas Adams noticed this phenomenon decades ago:
> The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair.
I’ve validated Jev’s confidence score. Accuracy scales linearly with confidence for the 3 use cases I tested. >.9 it matched a human labeler. I immediately discovered a user behavior I didn’t expect for ~$3. I can now mitigate in real-time due to low cost and latency. This may have a major positive financial impact for all our customers.
Not sure why anyone would feel the need to dunk on this thing without showing a real failure example.
TBF OP makes a good point: skipping out on the whole fine-tune sing and dance and diving directly into a good general classifier is a barrier to developing actual, deep intuition for the problem space (a pretty prevalant anti-AI argument).
If you’ve validated the Jev scores, then it means you’ve built and run the evals. The exact thing the author is arguing needs to be done if you’re using Jev.
So they’re not dunking on that, they’re dunking on the idea that you can just trust Jev confidence scores without actually doing any validation.
I build ai systems for ops automation and don’t understand the author’s pessimism.
I agree with that Evals are a scarce commodity rn. A business needs to define what good looks like. This is a laborious, and sometimes politically controversial, process.
Given good Evals, frontier llms are a magical tool that can automate tasks and do them more accurately than human ops teams.
Exactly which llm to use depends on the mixture of speed, cost and quality of the output.
Jev makes a claim to expand some regions of the Pareto frontier. I look forward to testing if this is true.
There are many areas of work we can’t automate rn. We cannot create good Evals either because time horizons are too long, or it’s too difficult to create good Evals.
That doesn’t mean there’s anything wrong with building good ai engineering systems in areas where it works magically.
> That doesn’t mean there’s anything wrong with building good ai engineering systems in areas where it works magically.
Without the evals, how do you know it works magically? I think that’s the entire point of the authors article, people are using AI tools, and trusting the AI own evaluation of its performance, without making any attempt to validate if the AIs confidence actually correlates to anything meaningful.
From what I’ve seen, the result tends to be very exciting demos, followed by incredibly mediocre results when deployed into a real environment. The lack of critical evaluation then provides a double whammy, because it takes longer than it should to realise the system isn’t actually working correctly, and then the team lacks the skills and understanding needed to improve the system they’ve built. You just kinda have to hope that the next model release is going to magically result in better performance for your particular scenario, which won’t be able to measure due to the lack of evals.
You need a set of Evals, which catch enough of the mistakes your llm makes following your baseline prompt, so you have an equal or lower error rate than humans.
This can be determined by offline benchmarks if you build a system that takes small sequences of actions, and requires live ab test for long sequences of actions.
The more your human ops team work from documented standard operating procedure, rather than tacit knowledge, the less you need Evals except to capture edge cases
If you spend more time with a product, you’re more likely to choose to do it again, even if it’s because of failure or annoyance. You’d justify it somehow (I have a leg up now or something). This actually applies to looking at things as well; a brightly colored box on the supermarket shelf is simply more likely to be chosen because you look at it first and longer.
Wasting your time and resources is a power signifier, but getting you to waste your own time and your own resources is hegemony.
See also "normal accidents" [1], which are an even bigger worst case scenario, which we already struggle with, and will only get worse.
"Normal accidents, or system accidents, are... inevitable in extremely complex systems. Given the characteristic of the system involved, multiple failures that interact with each other will occur, despite efforts to avoid them... while operator error is a very common problem, many failures relate to organizations rather than technology, and major accidents almost always have very small beginnings. Such events appear trivial to begin with before unpredictably cascading through the system to create a large event with severe consequences." [1]
Yeah I think the Jev hype is weird... But this blog post is complete nonsense.
Of course you aren't going to use Jev where you can easily make a decision deterministically with old-school programming. It's going to be used for stuff that might have previously been done by a person, e.g. reviewing app updates in app stores, reviewing CVs, labelling bugs, etc. Obviously that can already have inexplicable failures.
Another of the ways that software "engineering" is drifting even further away from traditional engineering. Failure mode analysis? Root cause analysis? I have no idea, I just talk to this magic box.
This is the problem building "it does everything" machines.
If you are pitching that your service can do potentially "whatever the client wants" you have such a thin basis on which to provide contracts and guarantees as a provider. The narrower the function, the clearer you can be about what's supposed to happen and why things might have gone wrong.
When you're using probabilities as the fundamental approach to computation, all of that goes out the window. Nondeterminism is powerful because it's insanely flexible, but the cost of that flexibility is predictability and expectation. Determinism was humanity's primary choice for formalisms and technology precisely because it reduces complex problems and situations to repeatable mechanics that are easy to understand. Deterministic tools can't do a lot in the grand scheme of things, but it is precisely these limitations that make them work well in concert and keep them comprehensible.
Inexplicable has a range from “I can’t explain that” to “that is not in the realm of possibility”. Windows failures especially have been steadily moving from A (already pretty sad) to B in the past years and I’ve been thinking about how utterly unempowering that is. Tools stop being tools if they have a secret life of their own.
It frustrates me quite a bit that the best solution to many issues is "Just try again tomorrow", trusting that the problem occurring on Azure is caused by some obscure backend service being down six timezones from our office, and the thing™ will start working again in a few days when Microsoft fixes said backend service.
Why is the author pretending that it is only now that inexplicable failures are "being normalized"? They've been normal(ized) ever since this industry has existed.
Literally go no further than the age old advice of "have you tried turning it off and then back on again?", and then that actually working.
Did this person never experience the effects of rocking the boat just a little too much? Daring to do a little too good of a job? How?
Are you referring to "move fast and break things"?
If so, it's definitely newish for those of us who started programming twenty+ years ago. Back then waterfall was the way to develop applications. Once it was burned to CDs or DVDs, that was it. We could not afford shipping broken things.
Ever since internet became ubiquitous we moved to on-the-fly patching for installed software and fast deployment cycles on cloud.
That said, the lack of forethought in design and backward compatibility is nowadays definitely on the rise and occurs likely on an order of magnitude more often.
No, I'm referring to the "we're a massive enterprise and we don't make money by actually figuring out issues, so if service XYZ has troubles after 30 hours of uptime, then we'll restart it every 20 hours and problem solved" phenomenon.
Or the phenomenon where I piss blood explaining why having a p99 that is 30x worse than our p95 is maybe possibly a problem that should be looked at.
Or the phenomenon where concurrency exists, and so issues are no longer reliably reproducible, meaning everyone just throws their hands up and tries to ignore and downplay them as much as humanly possible. That is until a dickhead like me comes around, and does something like a scripted 300 restart cycle test overnight until a clustering resiliency defect finally reproduces, and i can capture enough debugging data that would never be possible on a live environment. All the while the product vendor is twiddling their thumbs, waiting for us to provide said data on a silver platter, because for some reason this completely stock issue doesn't reproduce on their end in a pretty much identical environment, or so they say.
That can't be gold, must be movie props. Gold is stored in reverse pyramid, not upright pyramid, as to be movable at all. And it is not usually stored that high without shelves, since the floor below should not be indented by half a metre and picking up the last gold at floor level would be more cumbersome.
(Yes, storing gold on a what the room door claims is a toilet is a best practice, as thieves won't look for it there. But now the security through obscurity got leaked /s )
I am big on reproducibility (nix aficionado) and determinism (flagging test failures are a red-alert, all-hands-on-deck situation in my world) and correctness.
I am also big on testing (the correct things). And nine-nines (big on Elixir).
And... I'm also big on agent-assisted dev. Which requires pretty much every check in the book to stay productive in. And that's fine to me. I've seen bugs that I wouldn't have made myself. And I've also seen my own bugs fixed. They've all gotten fixed in short order. I don't see why this is a problem.
Raise your personal standards.
Thing is, the unreliable-software situation was already untenable before agents (in poor hands) made it worse.
I don’t think the author (or many people) doubt that one can (and some will) find a way that does not “suck”
But it’s pretty clear that most people are not. For whatever reasons (mgmt pressure, trying to get ahead, skill issues, etc) they half ass it, accept the 10% (silent) fail rate and blame the bad outcomes on the AI as if that absolves them. Or, adopt the attitude that 10% fail is fine, and people who say otherwise are being picky, or are anti-ai luddites or whatever. You should accept that things will suck.
If you took a bad but functional AI generated service and transported it back to 2018 it would have been at worst just mediocre. People do seem forget how dreadful devslop was in the past. I'd take an AI generated mess to disentangle every time over a spaghetti codebase that grew organically in the hands of careless managers.
> People do seem forget how dreadful devslop was in the past.
It was dreadful, but the volume was a single-digit percentage of what you see now.
Even those develoeprs who made a living opying from SO *still needed to make that code work for their system!"
As far a I can tell, most didn’t bother to see if it worked for the system,p. They just needed to have it compile in their system and then they’d call it done.
Volume has nothing to do with it, this is discussing code quality. But if you mind me saying so, blame this ridiculous amount of codeslop volume on greedy managers and corpocrats. Developers are artists, they usually ship shitcode when they're under pressure
That’s true. But there were also places with very high standards and good design. And the LLM slop is pushing in there as well.
Are the managers of these hypothetical places still interested in keeping the high standards and good design? Enshitification isn't inherent to the technology, if this is what you are deriving from the counterexample, it's a business strategy my dude.
It’s a combination of technical capabilities and human nature / incentives in big companies.
Concepts like good design, security, quality are abstract and hard to measure.
Time, cost, revenue etc are easy to measure. The “quality” people eventually get push aside by the money people.
This is not new, but LLMs further tilt the balance
> This is not new, but LLMs further tilt the balance
You speak as if llms had their own minds. Every time anyone talks about AI doing this or that they further reinforce this idea that there isn't a person behind all this. There's always someone watching.
With that said, when you say that llms tilt the balance, who specifically do you say that's driving llms to do that?
It means an ambitious manager can take on more work by having the team slop out. They move up (very effective leader!), standards fall, other managers have to match the changes. The bill comes due years later after they have moved on (and up).
And who pays that bill, actual or technical, is never who created them...
But as more and more things depend on increasingly deep software stacks, everything goes to utter shit if the individual systems don't become more reliable.
If you asked management they'd say that's "business risk"
Yeah, a good reason to be touchy about AI is that it tips the balance of power to lazy people who don't want to work or think. On the scale of our whole society. Imagining the ideal responsible use by most others is folly. No matter how responsible and conscientious you, the reader, are with your use of it.
> Thing is, the unreliable-software situation was already untenable before agents (in poor hands) made it worse.
I have definitely seen more bafflingly-poor OSS software that just plain doesn't work frequently now than before.
But it's mostly software that wouldn't have existed before because it's trying to do super-niche things. So on the "hobby" side of things, whatever.
But from a "trying to develop software as a business that you want to be a going concern," quality from people who should know better is less tenable than it used to be.
Thing is, the unreliable-software situation was already untenable before agents (in poor hands) made it worse.
Yes but that's the big thing, now isn't it? These are nice tools, used wisely. But their unwise use, oh boy...
The problem is one needs to be in a situation where the incentive is towards quality rather than speed. But that situation rather rare now - thirty years ago, Microsoft won the office wars with crap that had features. And nothing has fundamentally changed in web development since the LPad crisis.
The problem is those companies whose incentive is to allow bugs where it's the involuntary users who suffer will bite you no matter what quality you make your own software.
This, 100% I've seen far far worse produced by humans.
Excellent post. People always defend agentic/LLM-driven development by saying, "Well it's good enough", or "It works most of the time."
That may be tolerable for some user-facing app. But what if we start normalizing failures in the libraries, the infrastructure, and the compilers? Everything descends into a mess of unreliability, and that slows EVERYTHING and EVERYONE down.
> People always defend agentic/LLM-driven development by saying, "Well it's good enough", or "It works most of the time."
The main argument for LLM-driven development is much simpler: "It will get better".
The current state of LLM coding is about a year old. Imagine if we dismissed human coding efforts after a year. Rust, Python2 -> Python3 transition, Python type checking, Windows, C++, … nothing of that was done in a year and emerged in perfection in the first year. Everything takes ages to mature into a usable product. LLM coding is still in the "throw mud at the wall and see what sticks" stage, give it some more years and see how it will develop and what approaches actually work at. For the time being, LLMs are just the most useful development tool in the history of development tools, that's a pretty solid start in such a short time.
Banking/Finance is the one industry I've seen push back against this type of thinking. Transactions must be handled in a perfect and repeatable way, or the system is unusable as far as the company is concerned.
There's definitely still AI/LLM integration happening, but is kept out of specific areas of the business.
Same with aviation and safety. I honestly believe all programmers should, early in their careers, do a brief "tour of duty" in an industry where the stakes are high and "good enough" isn't good enough. You might not choose to make it your entire career, but at least you're exposed to the discipline, however briefly. Most software developers today have never in their lives worked on a project where defects were taken seriously and where there was process and documentation designed to reduce their occurrence.
To be fair, I'm inclined to believe that most of this pressure to lean (if not rely) on LLMs is not coming from programmers, but rather from middle and upper management: banks are resisting this not because their programmers are holding some kind of line, but because management is not giving-in to the LLM siren call. So that tour of duty, while a good idea, would be more about teaching management skills than anything else.
The normalisation of inexplicable failures is, in my opinion, the consequence of programmers being unable to refuse their managers who have fallen for the LLM siren call, lest they lose their jobs.
I think the failures matter in non-sensitive environments as well albeit with a different threshold.
If you randomly screw up customer orders (think of DoorDash or an online shop or Airbnb). They lose trust in you and you lose your business to the competition. Going happy go lucky and being irresponsible in the business can bankrupt most* businesses.
* well, of course except the criminal empires which are bailed out by our tax money.
I've been seeing the very opposite. Fintech companies treating design of financial systems with the casualness of a frontend aesthetic change. And tons of business people integrating their vibecoded POCs with financially sensitive data sources.
I’ve had banking transactions fail a number of times for unknowable or ill-defined reasons. One just last week in fact.
So that is a strange choice for repeatable, understandable operations. Might as well use Jev.
Exactly. Reliable abstractions are more important than ever. They're the dues the rest of us must pay to support vibe coding.
However, we are not (yet) building probabilistic libraries right? Even if AI is used, the code it builds is still deterministic and will fail in a way that can be fixed.
Not unlike legacy code, where the original developers are long gone and no one knows when that system breaking bug is about to wake up.
Even in the best of cases tho, fully human written and reviewed code will still fail eventually.
So I would say we are not normalizing failures (yet) but rather normalizing legacy.
We've been using probabilistic algorithms a long time now, also known as randomised algorithms. You probably won't find a program that is not using a hash table.
Compilers actually actively try to stay deterministic in their output.
> So I would say we are not normalizing failures (yet) but rather normalizing legacy.
Nice way of putting it.
It's also worth noting that LLM are fairly well suited to finding and documenting "load bearing" legacy bugs (when set out to do exactly that.)
I know because I've spent the better part of a year having them do exactly this as a precursor to rewrites in more perfomant tech stacks. Once this research is done, it's a fairly safe and mechanical translation job.
Even if AI is used, the code it builds is still deterministic and will fail in a way that can be fixed.
Say that with confidence now. You'll say with shock and incredulity later, "It can be fix? Right?!?!" (Cue some meme). Reproducing bugs is often hard for well designed systems. For randomly designed systems, it's a nightmare.
Fixing software is hard and fixing bad software is harder and fixing software where you didn't even think at the time how to make it maintainable is hardest.
Legacy systems are usually systems that were well designed at the time. That's why it's worth fixing them now.
--> And none of this is saying AI designed systems are bad. But if the only criteria is "code is deterministic by default, it works and we can fix problems later" then AI seems likely a terribleness accelerator.
"Well designed systems" are the key words.
Code written with AI can be fixed, if well designed, they are not mutually exclusive.
Also legacy systems were absolutely not always well designed, I've picked up a fair share of systems that were absolutely not well thought out, usually due to time pressure, or any other context specific reasons.
I do believe we are creating legacy systems faster now, and it's because teams are unable to keep up with the speed of changes, not because their system is not meant to be maintainable.
My point remains tho, using AI doesn't have to mean normalizing failures.
Write tests first. Have agent iterate until they are satisfied.
The point is that it boils down to writing the tests correctly, regardless of who is implementing the actual code. Hand-written code without test coverage has the same problems as AI generated code.
> Have agent iterate until they are satisfied.
Or have them rig the tests so that they always pass.
Ok agentic Ron Jeffries
https://ravimohan.blogspot.com/2007/04/learning-from-sudoku-...
Counterpoint: software has always had bugs. People are fallible.
Perhaps designing with assumptions that the entire stack is imperfect is the route to higher quality, along the lines of chaos monkey, which randomly kills processes on production boxes… which ensures resilience is real.
Even without AI our tech stack is so deep it’s hard to imagine every feature at every layer being provably correct.
And when people have the LLM do the whole design?
> This leads to a normalization of inexplicability.
It’s also tightly connected to a normalization of lack of accountability.
> This isn't "getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem" -- you still have to do the hard part.
This is probably losing the younger portion of the audience by now. ;)
The people accountable are those hosting & writing the algorithms, e.g. Therac-25.
> When a button breaks on a website, I have a model about what should have happened. Somewhere a contract got broken. [...] I might not have access to debug just an HTTP status 500, but I expect there to be somebody whose job is to understand why the endpoint is 500ing. The ownership is well-defined albeit opaque³.
> For many users, however, the actual experience is roughly just "stupid thing sucks." Software already feels capricious; more failures just change the rate of frustration.
I am betting author does not use cloud services much. It is not just "users", it's developers as well. Github is returning 5xx? AWS service does not work? Your email did not get delivered? Nothing we (developers) can do, "stupid thing sucks".
We see this fatalistic attitude all the time in software. "Bugs are inevitable." No, they aren't! Bugs are a choice. Almost all companies choose bugs because "no bugs" is too expensive. It's sometimes the right choice but we need to acknowledge that it's a choice and not some natural property of software. Imagine if people who built bridges or airplanes thought "bridge collapses and airplane crashes are inevitable, no way to solve it."
I’ve had several occasions where I would build a feature “properly” with defensive guardrails and handling of edge cases “that will never happen (but are technically possible)” and then you run into the coworker that says “it’s overengineered. YAGNI. Just do the one-line fix”
But...were they wrong? There is a difference between "edge case" and "cannot ever happen". I've seen this trend a lot lately where Claude suggests a lot of extra code to handle things that cannot possibly happen. Unless you ask it to, it won't trace the flow of data through the app to confirm the edge case actually exists, if local conditions appear to allow it. Then the developer lets it implement this crap without asking that one question, and I have to say YAGNI in the review.
I've seen so many software bugs in my career where when you traced execution, you'd find the code crashing in a block with the comment //this can never happen
> It is not just "users", it's developers as well.
One can be simultaneously a developer and a user. Distributed systems [0] weren't invented five years ago, after all. ;)
"Github owns this part that we rely on for correct operation and we can do fuckall about it when it fails." is a well-defined ownership model.
[0] ...implying the existence of distinct parts that can be independently developed and independently fail...
"Confidence scores" have always implied an anthopocentric meaning that doesn't exist. An algorithm doesn't have "confidence" in the way that a person has confidence, but as soon you put something with that name in front of a business person they assume the number is always a meaningful "letter grade curve" or "universal percentage". I still believe so much that the old quote to "there's lies, damned lies, and then statistics" remains a key to understanding so much why ML is leading to dumb outcomes versus hype. People don't understand statistics, so machines that produce nothing but statistics especially confuse people. (I feel this applies to LLMs as well.)
"Appeal to adult animated show" is lame enough on Reddit but it's particularly odious blogspam here.
The "normalization of inexplicability" is indeed infuriating. It has always been bad when it comes to computer software, and it's increasingly creeping into other consumer products that depend on embedded software.
I bought a new electric car recently. For the most part I've been quite happy with it. Shortly after I bought it, it started popping up a warning message saying "check EV system" every time I started it. By the time I brought it into the dealership, the warning had gone away, and the technician just told me something to the effect of "eh, I guess it just does that sometimes, let us know if it happens again." Hardware fault? Software bug? Who can say?
Like most modern cars, it has connectivity and Google Maps built into the infotainment system. The vast majority of the time, it works fine. Sometimes it says it has no connectivity (meaning no traffic data and suboptimal routes) for the duration of a drive, even in areas with a strong cell signal where it normally works fine. Sometimes the car says it has connectivity, but Google Maps still thinks it's offline. Sometimes Maps will actually load and display a route, but the "start navigation" button just spins forever as though it's still waiting for something. Are these related issues? Is there a common cause that might be fixable? Who can say?
(Conveniently enough, the warranty specifically does not cover any failures of software or firmware to operate correctly.)
My Kia Niro EV, which I'm otherwise quite happy with, has an occasional failure mode where it locks out the ignition for an arbitrary amount of time (generally 15 minutes). It's sometimes, but not exclusively, triggered by scheduled departure turning on the AC. Kia's response is a big shrug...
I think some of it is an unintended consequence of the backlash against bad or puzzling error dialog boxes in '80s and '90s software. A lot of people seem to have concluded that the easy solution to the challenges of error reporting was just to fail silently instead roll_safe.jpg .
> a warning message saying "check EV system"
These are some of the most annoying messages a piece of software can possibly display, and I hate them with a passion!
It's like this with a lot of commercial software, where designers don't want to scare users with complicated (useful) error messages, so you just get a "whoopsie! Something went wrong".
I'd much rather have an error message that I _don't_ understand over an error message I _can't_ understand.
In the case of cars, it used to be that a generic "Check engine" light showed up on your dash, and you needed an external tool to ask the computer what caused it to light up, but why would that be nessecary in a modern vehicle? You've got a large, high resolution display (or more!) on the CAN bus, why can't it show the full problem so you could take a picture right as it happens and send it to your dealer for instant diagnosis?
Even if the user error is opaque, why not log a short internal error to flash or the cloud?
I remember twenty years ago trouble-shooting my father's Internet connections and not being able to discover the misconfiguration. Once I called the technician and they knew nothing but "wade down to this dialogue and choose 'reset'". That didn't work either - but it was a distinct moment in me noticing the Internet and software becoming opaque. And it was only gotten - sometimes those moments of opaque failure are transitory but sometimes they stay indefinitely. Currently, my android phone can't see group texts created by iphones. I'm thankfully semi-retired enough for this to be a blessing but it could a crisis for someone else (I know a couple that fixes the problem temporarily but a text isn't something you want to spend hours debugging to get if you need it).
But my point - this has been increasing for quite a while and it's something like the evil twin of the "it just works" school of design. Because when the default expectation is "just working", the failure mode is "it just doesn't work". I mean, the default assumption that users are morons has de facto made us all morons since we can only deal with the "interface made for morons" and makes failure into opaque failure.
It's world the developers know "all abstractions are leaky abstraction" but where developers still won't give users (or even downstream developers) tools to deal with the leaks 'cause a simplistic interface matters more.
"When there's no emergency exit, there can't be an emergency, can there?"
Late Douglas Adams noticed this phenomenon decades ago:
> The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair.
There's that, but even he didn't note "now, soon nearly everything is going to be in that category of things that can't possibly go wrong"
I’ve validated Jev’s confidence score. Accuracy scales linearly with confidence for the 3 use cases I tested. >.9 it matched a human labeler. I immediately discovered a user behavior I didn’t expect for ~$3. I can now mitigate in real-time due to low cost and latency. This may have a major positive financial impact for all our customers.
Not sure why anyone would feel the need to dunk on this thing without showing a real failure example.
TBF OP makes a good point: skipping out on the whole fine-tune sing and dance and diving directly into a good general classifier is a barrier to developing actual, deep intuition for the problem space (a pretty prevalant anti-AI argument).
If you’ve validated the Jev scores, then it means you’ve built and run the evals. The exact thing the author is arguing needs to be done if you’re using Jev.
So they’re not dunking on that, they’re dunking on the idea that you can just trust Jev confidence scores without actually doing any validation.
I build ai systems for ops automation and don’t understand the author’s pessimism.
I agree with that Evals are a scarce commodity rn. A business needs to define what good looks like. This is a laborious, and sometimes politically controversial, process.
Given good Evals, frontier llms are a magical tool that can automate tasks and do them more accurately than human ops teams.
Exactly which llm to use depends on the mixture of speed, cost and quality of the output.
Jev makes a claim to expand some regions of the Pareto frontier. I look forward to testing if this is true.
There are many areas of work we can’t automate rn. We cannot create good Evals either because time horizons are too long, or it’s too difficult to create good Evals.
That doesn’t mean there’s anything wrong with building good ai engineering systems in areas where it works magically.
> That doesn’t mean there’s anything wrong with building good ai engineering systems in areas where it works magically.
Without the evals, how do you know it works magically? I think that’s the entire point of the authors article, people are using AI tools, and trusting the AI own evaluation of its performance, without making any attempt to validate if the AIs confidence actually correlates to anything meaningful.
From what I’ve seen, the result tends to be very exciting demos, followed by incredibly mediocre results when deployed into a real environment. The lack of critical evaluation then provides a double whammy, because it takes longer than it should to realise the system isn’t actually working correctly, and then the team lacks the skills and understanding needed to improve the system they’ve built. You just kinda have to hope that the next model release is going to magically result in better performance for your particular scenario, which won’t be able to measure due to the lack of evals.
You need a set of Evals, which catch enough of the mistakes your llm makes following your baseline prompt, so you have an equal or lower error rate than humans.
This can be determined by offline benchmarks if you build a system that takes small sequences of actions, and requires live ab test for long sequences of actions.
The more your human ops team work from documented standard operating procedure, rather than tacit knowledge, the less you need Evals except to capture edge cases
If you spend more time with a product, you’re more likely to choose to do it again, even if it’s because of failure or annoyance. You’d justify it somehow (I have a leg up now or something). This actually applies to looking at things as well; a brightly colored box on the supermarket shelf is simply more likely to be chosen because you look at it first and longer.
Wasting your time and resources is a power signifier, but getting you to waste your own time and your own resources is hegemony.
There is no way intentionally increasing failure rates improves user retention... right? Please tell me it doesn't.
See also "normal accidents" [1], which are an even bigger worst case scenario, which we already struggle with, and will only get worse.
"Normal accidents, or system accidents, are... inevitable in extremely complex systems. Given the characteristic of the system involved, multiple failures that interact with each other will occur, despite efforts to avoid them... while operator error is a very common problem, many failures relate to organizations rather than technology, and major accidents almost always have very small beginnings. Such events appear trivial to begin with before unpredictably cascading through the system to create a large event with severe consequences." [1]
[1] https://en.wikipedia.org/wiki/Normal_Accidents
That’s not even an attempt to be charitable to jev. It can make decisions from unstructured data. Nobody uses anything just because it is cheap…
Yeah I think the Jev hype is weird... But this blog post is complete nonsense.
Of course you aren't going to use Jev where you can easily make a decision deterministically with old-school programming. It's going to be used for stuff that might have previously been done by a person, e.g. reviewing app updates in app stores, reviewing CVs, labelling bugs, etc. Obviously that can already have inexplicable failures.
Another of the ways that software "engineering" is drifting even further away from traditional engineering. Failure mode analysis? Root cause analysis? I have no idea, I just talk to this magic box.
This is the problem building "it does everything" machines.
If you are pitching that your service can do potentially "whatever the client wants" you have such a thin basis on which to provide contracts and guarantees as a provider. The narrower the function, the clearer you can be about what's supposed to happen and why things might have gone wrong.
When you're using probabilities as the fundamental approach to computation, all of that goes out the window. Nondeterminism is powerful because it's insanely flexible, but the cost of that flexibility is predictability and expectation. Determinism was humanity's primary choice for formalisms and technology precisely because it reduces complex problems and situations to repeatable mechanics that are easy to understand. Deterministic tools can't do a lot in the grand scheme of things, but it is precisely these limitations that make them work well in concert and keep them comprehensible.
What happens when non deterministic machines become better than humans at translating requirements to deterministic machines?
Inexplicable has a range from “I can’t explain that” to “that is not in the realm of possibility”. Windows failures especially have been steadily moving from A (already pretty sad) to B in the past years and I’ve been thinking about how utterly unempowering that is. Tools stop being tools if they have a secret life of their own.
Excellent post!
It frustrates me quite a bit that the best solution to many issues is "Just try again tomorrow", trusting that the problem occurring on Azure is caused by some obscure backend service being down six timezones from our office, and the thing™ will start working again in a few days when Microsoft fixes said backend service.
Why is the author pretending that it is only now that inexplicable failures are "being normalized"? They've been normal(ized) ever since this industry has existed.
Literally go no further than the age old advice of "have you tried turning it off and then back on again?", and then that actually working.
Did this person never experience the effects of rocking the boat just a little too much? Daring to do a little too good of a job? How?
Are you referring to "move fast and break things"?
If so, it's definitely newish for those of us who started programming twenty+ years ago. Back then waterfall was the way to develop applications. Once it was burned to CDs or DVDs, that was it. We could not afford shipping broken things.
Ever since internet became ubiquitous we moved to on-the-fly patching for installed software and fast deployment cycles on cloud.
That said, the lack of forethought in design and backward compatibility is nowadays definitely on the rise and occurs likely on an order of magnitude more often.
No, I'm referring to the "we're a massive enterprise and we don't make money by actually figuring out issues, so if service XYZ has troubles after 30 hours of uptime, then we'll restart it every 20 hours and problem solved" phenomenon.
Or the phenomenon where I piss blood explaining why having a p99 that is 30x worse than our p95 is maybe possibly a problem that should be looked at.
Or the phenomenon where concurrency exists, and so issues are no longer reliably reproducible, meaning everyone just throws their hands up and tries to ignore and downplay them as much as humanly possible. That is until a dickhead like me comes around, and does something like a scripted 300 restart cycle test overnight until a clustering resiliency defect finally reproduces, and i can capture enough debugging data that would never be possible on a live environment. All the while the product vendor is twiddling their thumbs, waiting for us to provide said data on a silver platter, because for some reason this completely stock issue doesn't reproduce on their end in a pretty much identical environment, or so they say.
That can't be gold, must be movie props. Gold is stored in reverse pyramid, not upright pyramid, as to be movable at all. And it is not usually stored that high without shelves, since the floor below should not be indented by half a metre and picking up the last gold at floor level would be more cumbersome.
(Yes, storing gold on a what the room door claims is a toilet is a best practice, as thieves won't look for it there. But now the security through obscurity got leaked /s )
skipped