> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming.
In theory. The only difference between today and "the aughts" is that we have machines that can spit out a ton of code very quickly.
Nothing has changed about the discipline or honesty around testing (you can skip automated tests even faster now if you wish). You can and should work with AI to write tests, but you have to know the difference between a good test and a "looks good on paper" test in order for it to truly be effective and raise the quality of what you're building.
> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming.
I've been building a compiler with LLMs for a memory safe language like Rust with near zero cost abstractions (no GC), but with WAY less cognitive overhead.
I can tell you right now:
1) It's 100x more than I could have achieved with zero compiler design experience.
2) I'm HIGHLY skeptical that LLMs can build something of this complexity (in some ways it's more difficult than implementing a Rust compiler) - so the testing is quite robust - 3 different systems (unit, integration, fuzz tests) each with mutant testing, each with between ~65-90% line coverage and ~50-80% branch coverage, combined with ~99% line coverage and ~86% branch coverage.
There is ZERO chance I could get something even close to this level of "working" by myself ever - let alone with minimal effort.
The test is kind of simple - if LLM's can do this... They should be able to do just about anything... Compilers are notoriously difficult to verify they actually work, rather than just kind of work sometimes...
People can say I'm wasting my time all they want.
But, one, it's been enlightening. I'm literally in awe of what they can do and have done.
Two, I've developed a bunch of tooling / metrics necessary to get them to be able to do something at this level of complexity without falling over themselves. And I think it can work at scale pretty easily.
Nearly all of the research comes from the 80s or farther back for the complexity metrics.
Hate to be a pedant, but that's really not what "zero cost abstractions" means. The idea behind those is that you get a cleaner interface to some gross machine functionality/OS API/etc. layer, but don't pay a performance cost vs. using the gross lower-level layer. E.g. Rust's Option, unlike C++'s std::optional.
What you're thinking of is "no runtime" or "lightweight runtime", which does often mean "no garbage collector".
Rust's zero cost abstractions mainly stem from its affine ownership model managing memory lifetimes safely and correctly with zero cost - as that is the killer feature... That's what I do.
When people think of "zero cost" they don't think about std::optional. They think about not having to manage memory lifetimes AND NOT having to pay for a Garbage Collector to do it for you. That was always the trade you made until Rust.
I add on some cost to locks to prevent deadlock, and some cost to loops to insert co-operative yields in concurrent contexts unless you turn it off.
Affine as in substructural linear types. They correspond to linear logic [0], and affine logic is named such because the way it's defined corresponds to affine functions. You don't literally need to scale your pointers though.
You're not wasting your time; LLMs have written plenty of compilers. Compilers are easy for LLMs to work on, because their level of verifiability is very high. That is, an LLM can easily determine whether what a compiler is doing is correct or incorrect.
Automated verifiability goes down once a software project incorporates things like:
Writing unit tests used to be the bane of my existence. I used to hate them. Often times, the LoC for unit tests was 3X the LoC of the actual code.
But not any more! Now I point the LLM to the code and order it to write unit tests, covering all edge cases, etc. I'd rather spend 3 hours arguing with the LLM than writing unit tests! :-D
I am curious in your experience how often the LLM must also update the tests. I find that if LLMs write tests after the implementation exists, they are either extremely brittle because they are coupled to the implementation, or they cover little of value because they mock everything to the point of testing nothing.
I have found a decent trick to be to write a parameterized test with e.g. a `cases` array that tests a function how you want it tested. Then ask the LLM to fill out more cases. It’s not perfect, but results in much less brittleness since you’ve already defined the specifics of what gets tested and what doesn’t.
Some companies (e.g. Microsoft) used to have "Software Engineers in Test" who's job was writing such tests all day long, so that those developers who were developing features wouldn't waste time on it.
If your tests align with the spec and the tooling is good it isnt tedious.
If you find writing tests tedious enough to make using an LLM to write them seem like a good idea you're probably churning out repetitive tests, unnecessary tests, tests which aren't great at catching bugs.
I heard people say this before. I'm wondering, how do you instruct the LLM to generate the tests? Do you tell it the scenarios that would be covered, or do you just tell it to write tests for the code?
If you are working with a web application, playwright + frontier LLM is incredibly capable. They added some recent features to make this sort of use case go a lot smoother:
If you set this up correctly, you can have a main agent issue natural language testing instructions to this playwright agent which returns a natural language summary of what it experiences. This is the sort of thing where I begin to get interested in the idea of agents working while I sleep.
> The idea is to create a markdown file where an AI agent is asked to work as a QA engineer
Given your code-base is mature enough, please don't have a single Skill/Steering/Persona/Ruleset (or whatever) for your "QA Engineer." This is just the same "my behavioral file can one-shot the entire system build" kind of thinking that will give you expensive, marginal results as the system grows.
If you want to have success in this space, get really fine-grained. Every single test scope needs its own behavioral files.
Have your core behavioral file define some simple specifics around Test Pyramid, Test Purposes, checks for tautological tests, etc. Then get _really_ specific;
<test-type>-architect (plan)
<test-type>-engineer (execute)
<test-type>-resolver (problem solver, maintenance, how to manage a failure, etc.)
e.g., playwright-architect, etc.
Then create additional ones for Unit tests, API tests, contract tests, or any other required test layer for the SUT.
Overengineered? Maybe given the size of your codebase. But for anything significant, you are codifying what humans and their skillsets do.
Isn’t this explicitly the one place you’d never want to use AI? Like, the only actual problem with AI is that it sometimes ignores errors in output like it has a PHD in Blindness To Problems. I always figured the path forward was strictly enforced and managed tests written by hand, because who gives a shit about the code behind it as long as you can prove that the output is real?
Ten million blackboxes with ten billion tests or whatever. Otherwise it’s literally the blind leading the blind
Scenario testing is the new word for it and I think this is a game changer.
Two of the reasons I never liked writing tests is
- they didn’t seem to usually assert much internal logic
- they would have to be maintained along with the original code
I think scenario testing is much better instead because the actual way a person uses a feature hardly changes but the internals might change a lot.
So imagine I’m making an e-commerce website. There are lots of internal mechanisms. I’ll have an agent testing all the functionalities as if it were a customer. This gives me much much more confidence while writing code because it is more uncorellated with the code.
Tomorrow I can change a lot of internals but the testing agent stays the same.
There’s something to note though: not all code is possible to be scenario tested. Like data engineering and other things where the feedback time is huge.
Also the merits of documentation and specs. It’s been eye-opening to see the subset of developers who were almost disdainful about writing documentation for their colleagues but are now tripping over themselves to do so for their clanker.
People falling all over themselves to write docs for their pile-of-linear-algebra-with-a-smiley-face-painted-on-it [0] don't read the docs, no. People who give a shit about writing solid software that doesn't get them paged at three in the damn morning do.
[0] The face is there to provide social-trustworthiness signals to engage the human pack-bonding instinct, natch.
Your sarcasm is unwarranted, because what I said is true and reflects the experience of a lot of people.
A decade ago I left a job and spent the last week thoroughly documenting every flow and code section of an app that I worked with, which was the core value proposition of the company. A couple years later I ask around and nobody even took a look at that.
People just don't read, and there are actually good reasons for that, one of them being that documentation is outdated in most orgs and the effort to keep it up to date is greater than reading the code.
That’s a rather stunning comparison: racism is a problem because it’s unfairly treating sentient beings but a pile of linear algebra is not even sentient, much less your peer. That’s part of why I used the term: “agent” isn’t current because agents have, well, agency and can be held accountable.
People are rediscovering everything. Some people have proposed using a more formal language to tell the AI precisely what code to write. That's a compiler.
Ťhose are all technology variations of “automated web ui tests”, which is a subset of “automated ui tests”, which is itself almost (but not quite exactly) a subset of “automated user acceptance tests”, none of which are new categories.
So, throw out the traditional test pyramid, shift right, and rely more on persona testing than fine-grained atomic tests? I would hope teams don't need to re-learn that lesson for themselves, but...
My clankers are instructed to use "Outside In development" with "red/green TDD" at all times.
They write really good Gherkin features and then work inwards writing unit tests as they go - checking that they fail before implementation so it's actually testing something worthwhile.
And the code they ship is decent quality (not as good as me most of the time - but a LOT better than me when I'm tired or I'm pissed off about something or the work is really boring).
This already exists. You mean capturing user flows which should already be supplied by product to the developer. A decent system is Behavior Driven Development (though honestly a poor acronym for it’s use).
Are you an engineer ? You must test your "creation". Or would you expect that the microwave owen you just bougth will be tested by your child while getting burned ?
I ran mutation testing on a side project recently and found a test that passed even if the production method returned an empty string. AI-generated tests at scale will have exactly this problem. High coverage, confident test names, zero actual verification.
Is the use of the term "automatic programming" a deliberate parallel to the development of COBOL [1]?
If so, if this is meant to imply that LLMs are just another step towards higher-level abstractions, the analogy doesn't quite work. Unlike a COBOL compiler, the LLMs output can't be predicted or reasoned about, so you can't really fix bugs in your program (i.e. your prompt) but only try to permute it haphazardly and hope for the best.
I believe this can work if done on top of traditional testing. I would feel very uneasy to replace deterministic (ok, not always but mostly) test suites with something that is not deterministic at all
I think this is just TDD or unit test dogma and I’m personally not a fan.
Unit tests and deterministic tests are hard to get right and need to be done at the correct boundary.
I have seen many people dogmatically pushing unit tests religiously but this often leads to very hard to maintain tests that mostly exist just to change along with the main code itself.
A good way to understand if your unit tests are good: are you changing them along with changing your actual code? Then it’s a bad test. I think the argument for “it’s just documentation” is weak.
I don’t disagree with your point, but there is still value in having unit tests that change along with the code. It’s less than a “proper” test, but when these tests break _unexpectedly_, it’s still more signal than you’d have without them. Like, always changing `file.go` alongside `file_test.go` may be acceptable if you catch errors that impact `serve_test.go` unexpectedly.
Of course, if you’re just watching Claude changing both and saying “LGTM” then it’s not very valuable.
> A good way to understand if your unit tests are good: are you changing them along with changing your actual code? Then it’s a bad test. I think the argument for “it’s just documentation” is weak.
Unit tests are great for pure algorithms, like file format, data encoding, crypto,… etc. Everything with a specs that will rarely changes. You write your tests once and basically never have to update them.
But for requirements that changes often like in a enterprise settings or applications, maintaining a suite of unit tests is expensive. Integration tests are better because contracts between modules don’t change that much. Even if the suite are not exhaustive, they’re useful enough to catch some failures.
> But for requirements that changes often like in a enterprise settings or applications, maintaining a suite of unit tests is expensive. Integration tests are better because contracts between modules don’t change that much. Even if the suite are not exhaustive, they’re useful enough to catch some failures.
> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming.
In theory. The only difference between today and "the aughts" is that we have machines that can spit out a ton of code very quickly.
Nothing has changed about the discipline or honesty around testing (you can skip automated tests even faster now if you wish). You can and should work with AI to write tests, but you have to know the difference between a good test and a "looks good on paper" test in order for it to truly be effective and raise the quality of what you're building.
> I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming.
I've been building a compiler with LLMs for a memory safe language like Rust with near zero cost abstractions (no GC), but with WAY less cognitive overhead.
I can tell you right now:
1) It's 100x more than I could have achieved with zero compiler design experience.
2) I'm HIGHLY skeptical that LLMs can build something of this complexity (in some ways it's more difficult than implementing a Rust compiler) - so the testing is quite robust - 3 different systems (unit, integration, fuzz tests) each with mutant testing, each with between ~65-90% line coverage and ~50-80% branch coverage, combined with ~99% line coverage and ~86% branch coverage.
There is ZERO chance I could get something even close to this level of "working" by myself ever - let alone with minimal effort.
The test is kind of simple - if LLM's can do this... They should be able to do just about anything... Compilers are notoriously difficult to verify they actually work, rather than just kind of work sometimes...
People can say I'm wasting my time all they want.
But, one, it's been enlightening. I'm literally in awe of what they can do and have done.
Two, I've developed a bunch of tooling / metrics necessary to get them to be able to do something at this level of complexity without falling over themselves. And I think it can work at scale pretty easily.
Nearly all of the research comes from the 80s or farther back for the complexity metrics.
Hate to be a pedant, but that's really not what "zero cost abstractions" means. The idea behind those is that you get a cleaner interface to some gross machine functionality/OS API/etc. layer, but don't pay a performance cost vs. using the gross lower-level layer. E.g. Rust's Option, unlike C++'s std::optional.
What you're thinking of is "no runtime" or "lightweight runtime", which does often mean "no garbage collector".
Rust's zero cost abstractions mainly stem from its affine ownership model managing memory lifetimes safely and correctly with zero cost - as that is the killer feature... That's what I do.
When people think of "zero cost" they don't think about std::optional. They think about not having to manage memory lifetimes AND NOT having to pay for a Garbage Collector to do it for you. That was always the trade you made until Rust.
I add on some cost to locks to prevent deadlock, and some cost to loops to insert co-operative yields in concurrent contexts unless you turn it off.
> affine ownership
huh? you can rotate and scale the ownership?
Affine as in substructural linear types. They correspond to linear logic [0], and affine logic is named such because the way it's defined corresponds to affine functions. You don't literally need to scale your pointers though.
[0] https://en.wikipedia.org/wiki/Linear_logic
You're not wasting your time; LLMs have written plenty of compilers. Compilers are easy for LLMs to work on, because their level of verifiability is very high. That is, an LLM can easily determine whether what a compiler is doing is correct or incorrect.
Automated verifiability goes down once a software project incorporates things like:
- Concurrency
- Networking / distributed systems
- Visuals / animations
- Domain knowledge (e.g. banking, finance)
Yes and no, they are easy to verify they work sometimes, hard to verify they work all times.
Writing unit tests used to be the bane of my existence. I used to hate them. Often times, the LoC for unit tests was 3X the LoC of the actual code.
But not any more! Now I point the LLM to the code and order it to write unit tests, covering all edge cases, etc. I'd rather spend 3 hours arguing with the LLM than writing unit tests! :-D
I am curious in your experience how often the LLM must also update the tests. I find that if LLMs write tests after the implementation exists, they are either extremely brittle because they are coupled to the implementation, or they cover little of value because they mock everything to the point of testing nothing.
I have found a decent trick to be to write a parameterized test with e.g. a `cases` array that tests a function how you want it tested. Then ask the LLM to fill out more cases. It’s not perfect, but results in much less brittleness since you’ve already defined the specifics of what gets tested and what doesn’t.
Same for me. I actively ask the LLM to write as few tests as possible. Otherwise you end up redundant and low value ttests.
Yep, and wasted token spend on an ongoing basis.
I instruct the LLM to follow TDD practices in certain areas, but otherwise prioritize integration style tests at the edges.
[flagged]
[dead]
Some companies (e.g. Microsoft) used to have "Software Engineers in Test" who's job was writing such tests all day long, so that those developers who were developing features wouldn't waste time on it.
If your tests align with the spec and the tooling is good it isnt tedious.
If you find writing tests tedious enough to make using an LLM to write them seem like a good idea you're probably churning out repetitive tests, unnecessary tests, tests which aren't great at catching bugs.
I heard people say this before. I'm wondering, how do you instruct the LLM to generate the tests? Do you tell it the scenarios that would be covered, or do you just tell it to write tests for the code?
>cover all edge cases
That’s probably the extent of the prompt.
I have only one thing to say....
what
If you are working with a web application, playwright + frontier LLM is incredibly capable. They added some recent features to make this sort of use case go a lot smoother:
https://playwright.dev/docs/release-notes#version-159
If you set this up correctly, you can have a main agent issue natural language testing instructions to this playwright agent which returns a natural language summary of what it experiences. This is the sort of thing where I begin to get interested in the idea of agents working while I sleep.
> The idea is to create a markdown file where an AI agent is asked to work as a QA engineer
Given your code-base is mature enough, please don't have a single Skill/Steering/Persona/Ruleset (or whatever) for your "QA Engineer." This is just the same "my behavioral file can one-shot the entire system build" kind of thinking that will give you expensive, marginal results as the system grows.
If you want to have success in this space, get really fine-grained. Every single test scope needs its own behavioral files.
Have your core behavioral file define some simple specifics around Test Pyramid, Test Purposes, checks for tautological tests, etc. Then get _really_ specific;
<test-type>-architect (plan)
<test-type>-engineer (execute)
<test-type>-resolver (problem solver, maintenance, how to manage a failure, etc.)
e.g., playwright-architect, etc.
Then create additional ones for Unit tests, API tests, contract tests, or any other required test layer for the SUT.
Overengineered? Maybe given the size of your codebase. But for anything significant, you are codifying what humans and their skillsets do.
[dead]
Isn’t this explicitly the one place you’d never want to use AI? Like, the only actual problem with AI is that it sometimes ignores errors in output like it has a PHD in Blindness To Problems. I always figured the path forward was strictly enforced and managed tests written by hand, because who gives a shit about the code behind it as long as you can prove that the output is real?
Ten million blackboxes with ten billion tests or whatever. Otherwise it’s literally the blind leading the blind
Scenario testing is the new word for it and I think this is a game changer.
Two of the reasons I never liked writing tests is
- they didn’t seem to usually assert much internal logic
- they would have to be maintained along with the original code
I think scenario testing is much better instead because the actual way a person uses a feature hardly changes but the internals might change a lot.
So imagine I’m making an e-commerce website. There are lots of internal mechanisms. I’ll have an agent testing all the functionalities as if it were a customer. This gives me much much more confidence while writing code because it is more uncorellated with the code.
Tomorrow I can change a lot of internals but the testing agent stays the same.
There’s something to note though: not all code is possible to be scenario tested. Like data engineering and other things where the feedback time is huge.
are we just re-inventing playwright tests except 10x slower and infinity times more expensive?
i feel like im going insane
since the rise of agentic coding tools, it feels like we're in a new "eternal september" of people discovering ui end-to-end test automation.
Also the merits of documentation and specs. It’s been eye-opening to see the subset of developers who were almost disdainful about writing documentation for their colleagues but are now tripping over themselves to do so for their clanker.
Agents read the docs. People don't. That's the underlying reason.
> People don't.
People falling all over themselves to write docs for their pile-of-linear-algebra-with-a-smiley-face-painted-on-it [0] don't read the docs, no. People who give a shit about writing solid software that doesn't get them paged at three in the damn morning do.
[0] The face is there to provide social-trustworthiness signals to engage the human pack-bonding instinct, natch.
Your sarcasm is unwarranted, because what I said is true and reflects the experience of a lot of people.
A decade ago I left a job and spent the last week thoroughly documenting every flow and code section of an app that I worked with, which was the core value proposition of the company. A couple years later I ask around and nobody even took a look at that.
People just don't read, and there are actually good reasons for that, one of them being that documentation is outdated in most orgs and the effort to keep it up to date is greater than reading the code.
> ...what I said is true and reflects the experience of a lot of people.
Wow. What I said is true and reflects the experience of a lot of people. Amazing!
That's nice, but if you write docs and people don't read them then that's clearly not a winning strategy in many orgs.
[flagged]
That’s a rather stunning comparison: racism is a problem because it’s unfairly treating sentient beings but a pile of linear algebra is not even sentient, much less your peer. That’s part of why I used the term: “agent” isn’t current because agents have, well, agency and can be held accountable.
https://lucumr.pocoo.org/2026/5/26/clankers/
Positing an equivalence between a dismissive term for AI bots and a racial slur against black people is, like, super racist.
People are rediscovering everything. Some people have proposed using a more formal language to tell the AI precisely what code to write. That's a compiler.
Well playwright tests used to be called puppeteer tests which used to be called selenium tests, so you tell me.
Ťhose are all technology variations of “automated web ui tests”, which is a subset of “automated ui tests”, which is itself almost (but not quite exactly) a subset of “automated user acceptance tests”, none of which are new categories.
So, throw out the traditional test pyramid, shift right, and rely more on persona testing than fine-grained atomic tests? I would hope teams don't need to re-learn that lesson for themselves, but...
>Scenario testing is the new word
How is scenario different from a behavior (as in Behavior-Driven Development)?
Gherkin and things like Cucumber are not something new, are they?
My clankers are instructed to use "Outside In development" with "red/green TDD" at all times.
They write really good Gherkin features and then work inwards writing unit tests as they go - checking that they fail before implementation so it's actually testing something worthwhile.
And the code they ship is decent quality (not as good as me most of the time - but a LOT better than me when I'm tired or I'm pissed off about something or the work is really boring).
Well, scenario contains the letters s, c and n, while behavior doesn't.
This already exists. You mean capturing user flows which should already be supplied by product to the developer. A decent system is Behavior Driven Development (though honestly a poor acronym for it’s use).
> Two of the reasons I never liked writing tests
Are you an engineer ? You must test your "creation". Or would you expect that the microwave owen you just bougth will be tested by your child while getting burned ?
'I never liked writing tests' is not the same as 'I don't write tests'.
I ran mutation testing on a side project recently and found a test that passed even if the production method returned an empty string. AI-generated tests at scale will have exactly this problem. High coverage, confident test names, zero actual verification.
Don't worry, AI maximalists have a solution: create tests for the tests.
That's what mutation testing is.
IME there are these levels of tests:
- If you call the setter, the getter returns the same value - these are kinda bullshit and would be caught by the next level anyway
- Testing basic normal use
- Testing known difficulties of the implementation
- Exhaustive or randomized (if necessary) testing of the state space, ~= property-based testing
I expect AI to have very different levels of ability for these, not necessarily in strictly descending order as listed.
Is the use of the term "automatic programming" a deliberate parallel to the development of COBOL [1]?
If so, if this is meant to imply that LLMs are just another step towards higher-level abstractions, the analogy doesn't quite work. Unlike a COBOL compiler, the LLMs output can't be predicted or reasoned about, so you can't really fix bugs in your program (i.e. your prompt) but only try to permute it haphazardly and hope for the best.
[1] https://ethw.org/Milestones:A-0_Compiler_and_Initial_Develop...
I believe this can work if done on top of traditional testing. I would feel very uneasy to replace deterministic (ok, not always but mostly) test suites with something that is not deterministic at all
I think this is just TDD or unit test dogma and I’m personally not a fan.
Unit tests and deterministic tests are hard to get right and need to be done at the correct boundary.
I have seen many people dogmatically pushing unit tests religiously but this often leads to very hard to maintain tests that mostly exist just to change along with the main code itself.
A good way to understand if your unit tests are good: are you changing them along with changing your actual code? Then it’s a bad test. I think the argument for “it’s just documentation” is weak.
I don’t disagree with your point, but there is still value in having unit tests that change along with the code. It’s less than a “proper” test, but when these tests break _unexpectedly_, it’s still more signal than you’d have without them. Like, always changing `file.go` alongside `file_test.go` may be acceptable if you catch errors that impact `serve_test.go` unexpectedly.
Of course, if you’re just watching Claude changing both and saying “LGTM” then it’s not very valuable.
> A good way to understand if your unit tests are good: are you changing them along with changing your actual code? Then it’s a bad test. I think the argument for “it’s just documentation” is weak.
Unit tests are great for pure algorithms, like file format, data encoding, crypto,… etc. Everything with a specs that will rarely changes. You write your tests once and basically never have to update them.
But for requirements that changes often like in a enterprise settings or applications, maintaining a suite of unit tests is expensive. Integration tests are better because contracts between modules don’t change that much. Even if the suite are not exhaustive, they’re useful enough to catch some failures.
> But for requirements that changes often like in a enterprise settings or applications, maintaining a suite of unit tests is expensive. Integration tests are better because contracts between modules don’t change that much. Even if the suite are not exhaustive, they’re useful enough to catch some failures.
Yes this is what I'm trying to say.
The idea of injecting more indeterminacy in pipelines is beyond me.
Well you see, you just run the same test 10,000 times, and then...
What is the point of asking LLM to do manual testing? IMHO it would be much better to make it write automated tests. So you can just rerun them?
Please use a more readable variable-width font
[dead]
[dead]
[flagged]
That crosses into personal attack. You can't do that on HN so we've banned the account.
Not a single word of this post was written by AI. You are hallucinating.