> It’s actually fine if agents make a lot of boneheaded mistakes. What’s not ok is if they keep making the same mistakes.
I worked in life sciences for a bit. There is a process in clinical trials called corrective and preventative actions (CAPA). You'll also find this in other areas where failure tolerance is low (e.g. aircraft).
It's simple: when a mistake happens, you run your CAPA process (Google CAPA form and see examples to extrapolate what that process might look like) and determine the root cause and the correction to the process that allowed the mistake to happen in the first place.
(At least as a SaaS vendor in life sciences, when we had a CAPA (e.g. after a SEV0 failure), it would be folded into our SOPs and then we would be required to retrain on the SOP. Auditors would want to see our evidence of CAPAs, the versions of our SOPs, the records of training. All to extreme for most shops, but I add this for context/color)
This is something most eng shops do not have the discipline for since it requires some diligence.
Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
You stop relying on the agents following instructions exactly.
You need two pieces:
a) prompts, that tell the agents what to do and how to do it (and ideally, the why, where, etc, the full picture) - that's the positive half, that drives behavior the way you want it.
b) deterministic tooling that prevents negative outcomes, like linters, compilers, static analysis, fuzzing, testing, the more the better. This side should either be firewalled off from the AI or very carefully watched so that it doesn't drift.
The part that you put in the deterministic side is the "never do x" stuff - I have lint for long comments (which AI hits every single time it commits), all my dev scripts are in typescript, precommit hooks, massive CI, and I lint even for things like redirecting error to standard out, tiny stuff, and also e.g. static migration analysis so the AI never ships an exclusive full table lock in a migration, for example.
yes, this is a very helpful approach. I take a similar approach, partly because I find bringing up negative directions seem to bring the undesired action/direction into the agent's view and makes it more likely that they will go there
You can’t deterministically keep them from making even a tiny fraction of all the possible errors they can and do make though.
You can’t keep humans from making those errors either but you also don’t let an error prone human crank out 20k LOC per day without forcing other humans to understand it.
C# Roslyn Analyzers[0], for example, are quite powerful and can identify complex patterns in code. One approach to deterministic enforcement would be to ensure that the project is set up with an analyzers library and mistakes that can be deterministically flagged are
I don't know if that's necessarily true in practice the way it is in theory. If you knock down 95% of the issues they create over a certain period of time, in practice you'll have better code than a human would write, and it's all upside from there.
I've been using languages with stronger type systems and that's also a huge boon.
Why couldn't you express all those as test cases rather than instructions?
In test cases i can do anything, a test framework is just a way of discovering and then scheduling functions to run. I can emit useful instructions to the agent from the failed test case: "After walking the AST of all use of state machine X, a branch was found at Y which reused stale state. Ensure stale references are dropped..."
I can force the agent to pass the test suite before it considers itself done. I can reject edits of such test cases to partially mitigate reward hacking. etc etc
You can express them as tests, but you also need a feedback mechanism that creates the rule that when the LLM generates some net new code or performs some refactor, that there are these CAPAs that it needs to cover with test cases.
The CAPA is a learning that sits outside of the mechanism of verification; it is a record of problem:root_cause:preventative_action. I see it as the instruction that would be required to generate the test case to prevent the next occurrence of a class of failures.
In a real-world process, for example, there is usually a QA lead that is verifying that the process is followed by looking at the paperwork and evidence.
> Why couldn't you express all those as test cases rather than instructions?
How would prevent removing the test case and the code that it tests? That's a legitimate thing to do when you're modifying a codebase. My intuition is "corrective and preventative actions (CAPA)" is a level or two up from test cases.
This is a question of context management and I suppose some would classify this as "harness engineering" as the trend of the moment.
One approach, for example, might be to have the a standalone code reviewer agent that is solely responsible for interfacing with the CAPA system (e.g. via a tool, via MCP) and acts as a back stop. When it finds a new type of CAPA, it stores it (and the backend indexes it with enough metadata to support broad types of retrieval). When it reviews a piece of code, it finds past CAPAs. By file locality. By business domain in the application. By keywords.
Same tool and repository available to both building agents and review agents, but use the review agent as a dedicated back stop as part of the verification process.
Every place I've worked at has positioned their SEV review process as 'blameless' with respect to the engineers who created the code, processes, etc that led to the incident. On paper, it's not about pointing fingers (though I've found this to be somewhat idealistic), but about finding faults in the processes that let bad code slip through the cracks.
It will be interesting to see how the notion of 'responsibility' carries over as agents handle higher and higher levels of abstraction. Agents built using the popular frameworks of today are somewhere in between code, formerly written by engineers, and an actor (like an engineer). When an agent wrote the framework to audit and monitor ops agents who are monitoring the performance and reliability of the code that coding agents wrote, who is responsible for ultimately ensuring it doesn't happen again? Should the findings just be added to whatever RAG and a few prompts/hooks/skills changed by another agent? When does a human follow up and to what degree?
This is the sort of thing that makes me believe that software engineers will never truly go away; at the very least, they serve the very useful function of being ultimately responsible for something going wrong.
> Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
I think a point we are grappling with is, what necessitates human intervention, like philosophically. Is it accountability? I was thinking about this in terms of code review and it's not like we would fire someone if they broke prod, so at what point do we need someone accountable. Maybe it's for approvals for certain classes of risk (like those associated with actual harm to life). It's also not lost on me that many human systems lack accountability.
I think there's a pretty clear and wide gulf between "nobody could have seen that coming", or "I just wasn't on my A game", and frank negligence.
I once worked at a very large company, and one of my coworkers was like "eh, I'm not going to do any of this work, because I don't have to", and that was a clear indicator that the problem wasn't (just) with the system, but with him individually.
I've had a similar experience, also at a very large organization. In my case this person chose that kind of environment because it allowed them to work on side stuff while waiting for approvals, etc. A lot of people in very large companies don't view the grind of organizational friction inhibiting their actions as problematic.
This is a general principal in the eng shop I work for, its written into every step, including the SCRUM etc.. What when well, why did it fail, what could be done better. Its all part of root cause analysis.
I personally use it for the agents I use, so why did it fail, what was the root cause, what can we do to prevent it again.
Agreed that domain experts should be making these kinds of decisions, and the right way for an agent mistake to get caught is going to depend a lot on the cost and probability of the mistake. My only point here is that the agent side of the system needs to learn. The software production machine needs to improve over time.
Sorry to get hung up on what is probably just flavor text, but the first sentence.. "Agents can oneshot games that are actually fun", uhhh such as?
Would love to be proven wrong here, feels like I've just seen the same basic one-level kart racer in like 3 model announcements. Not even sure if they were one-shotted, I certainly would not describe them as "fun"
As someone who builds games, particularly games meant to be played in the browser or apps, it can 100% one-shot a game-like experience, but it will not be very good.
You need to playtest the hell out of games. Play a bit, iterate, play a bit more, iterate, etc. You will get way better results if you start small and don’t try to one-shot it. Start small, vertical slice, playtest, and go fro there.
Yeah, that goes against my experience. Agents can absolutely one-shot games, but the gap between something that looks like a game and something that is fun to play and worth trying for more than a few minutes is gaping wide.
Probably fun defined in the ephemeral context. As-in it was "fun" to play in the context of the short form video attention span, social media brain rotted individual. Then it was thrown away and never played again.
It's definitely not fun if you consider building an actual community around it and have a long term plan for it.
Agreed, I read this and almost instinctively decided to close the page, agents can be used to build games for sure, but that "one shot" and "fun" in the same sentence is really misleading
FYI, it looks like you've been banned for the last ~8-9 months, for no obvious reason I can see. I vouched for your comment to make it visible, but they appears to show up as [dead] by default.
Kind of feels like we're in a similar place with AI programming as we are with self driving vehicles. Good enough for the typical case, maybe even better than most humans, but bad when dealing with novel situations or situations which require a really cohesive model of the system (code, business requirements, physical world, whatever).
Makes sense to me - I see it going down two separate routes in parallel:
One is to take the human out the OODA loop for cyber defense. Servlet libs (for example) are going to become fluid, self-modifying things w/ contracts that operate much differenlty from how they do today. The engineering practice around these things will need to change.
The second is that UX will be self-modifying. Just like how pi can modify itself, I can see this being a general practice for user-facing applications. Perhaps a text-box in which users can describe tweaks in can request changes to how the application functions. Engineering an application will focus on modeling the non-negotiables of how an application works, and providing the correct primitives for user-driven LLMs to modify software on the fly, as well as track and rollback changes. There'll also be funsies around how to ensure that ads get delivered regardless of the user trying to get rid of them.
What would you prefer: 99% of user requests to succeed but w/ 5% risk of getting O-day'd, or have 90% of user requests succeed but w/ 0.1% risk of getting 0-day'd.
If you have a logo in the top left corner of your company blog the logo should route to the home page so I can see what product you're actually building.
This would be more convincing if there were a simple stat on the home page of the dispositions of the reported bugs (fixed/no-repro/won't-fix).
As a random sample of one, I looked at one of the bugs this reported on Tailscale (first thing on the homepage) [0], and the pull request ends with "Apologies for the lack of due diligence here. I'll go ahead and close this out."
We should publish some stats. The fix rates on bugs are around 45% baseline, increasing over time as we learn from behavior within a given codebase. The merge rates for the codebase health work we do (e.g. deleting dead code) are very high, at least 90%.
My answer to this is adding an engineering assurance module to my specification and verification toolkit. Every project has a different definition of success and quality. My goal is to provide a consistent way to define, analyze, and report quality metrics tailored to each project.
My bet is that we'll see a second layer of harness emerge, as self-driving codebases become the target. There will be an application facing harness, orthogonal to the agent facing harness. The app harness will represent the software factory that is emergent for the specific application being developed.
I suspect it won't be a harness, but just a more specific LLM trained in the universe of user-selected context of vetted resources.
Why? Because LLMs are always going to be dumb when they're trained at scale. Their ability to speak software diverges from their friendly user input layer. A harness won't overcome that, but an LLM saddle ontop of a larger model would provide the type of feedback loops you'd want to look into.
I don't think you'll find two deterministic systems will produce much.
I think that dissolves the "self driving" distinction, though? The mechanisms for driving the codebase must be present in the codebase itself. Otherwise its just a regular out-of-band development process.
LLMs also do not have anything like memory, by design. Everything bolted on that smells like memory or is being called memory is a crutch, at best. Not being pedantic, I just think this aspect is lost on a lot of people.
Also, when we humans are tricked into perceiving a remote mind, we automatically assume it has a memory like us, or at least like large animals.
In other words, there's a story-document between a SherlockHolmesBot and UserPlaceholder that's growing like a crystal formation. Some software sees "Sherlock Homes Says" and then "performs" the dialogue at us, and we then assume Sherlock Holmes exists with a mind and memory, rather than being a facet of a text-story.
Sometimes, but I think there are a few problems with this:
1. Some context doesn't have an obvious place to write it in the code. If you're explaining why a tricky function is implemented a certain way, you can leave the explanation above the function or within the function. If you're explaining something more general, there might not be a natural place to put it.
2. Relatedly, some context doesn't have an obvious location to read in the code. E.g. you can comment on a schema that a table is intended to be append-only, but an agent could easily miss this comment if it doesn't gather context all the way down to the raw schema. Any research process has unknown unknowns. There may not be a canonical place to look.
3. Adding comments for every single human intent might be a bit noisy. E.g., if a code reviewer flags something that I know isn't a bug, I want the system to learn from this, and I want that knowledge to take effect outside of just code review, but I might not want every single code review thread to yield a codebase comment.
1: There are corner cases, no doubt, but the main source of information should remain near the code.
2 & 3: How to be sure that the agent will pick the "append only hint" from the memory? Especially when it starts growing and cannot be part of the context?
I think the bigger need is for outside concerns, like: what is the infrastructure like? How much volume does this feature handle? How should changes to live code be rolled out to prevent running processes from failing?
In addition to all the product details that aren't in the codebase or docs, like "keep this logical path because it is used by our one big client."
Mostly what I think is needed is a richer worldview available to the LLM so it understands not just the code but can understand the product and its real world usage and constraints.
But how you are going to link the "keep this logical path ..." with the code? You cannot feed all the memory to the agent because of the context rot, and you are not sure the agent will always pick it when needed.
I have integrated AI into my application. The source code is embedded into the binary. When someone questions the numbers in my app it can debug itself. If it truly finds a bug in itself it will tell the user and open a ticket. Soon we are going to let the application fix itself.
How do you protect intellectual property? Or is this a case of the value being somewhere else, such as in your backend? If so, how does the agent debug frontend and backend? I presume it stops at frontend
I recently built a calculator to try to quantify the gains an org could expect as they become more "AI-native." I tried to account for a bunch of things including env, which I call "AI roadway," but I'm definitely missing codebase readiness (thank you, author), and also bottleneck analysis. Even without those considerations, though, gains are usually modest. It's hard to get to 2x.
I suspect that if you're not careful with agent memory it creates a danger of agent-driven cargo-cult behavior. I've watched this in my own ad-hoc agent loops where it starts with something basic, like the first agent tried to run some gigantic dependency inspection command and OOMed the local JVM and eventually recorded a workaround (to enable it to run gigantic dependency inspection commands...), and by time I get a few more agents into the loop, agents have written entire paragraphs about testing and validating local dev environment memory configurations that are mostly irrelevant to whatever is being worked on.
In general I've seen other issues like this where small errors and irrelevant comments in the codebase spin out into larger problems that consume annoying amounts of time/tokens. Maybe Anthropic and OpenAI don't notice this because they're in an "infinite monkeys with typewriters" scenario, but it's noticeable to me when the agent in my CLI has been spinning for 15 minutes contemplating irrelevant details
This happens way too often in my experience and I tried a lot of things. If the agent has access to an existing codebase it will always try to stay as close to it as possible. It doesn’t diverge from the initial direction the project/code went into, even if it’s already clear that this direction goes nowhere. Somehow agents aren’t able to detach themselves from existing code to come up with completely new solutions.
Sometimes I feel like it would be best to only give the agent access to clear API boundaries (say public interfaces to certain modules) and let it work out a new system from scratch given the expected inputs and outputs. Then plug this independent solution into the system. Of course it can still overfit these interfaces but it’s less than having access to the entire codebase.
Thinking about it, maybe it’s possible to let one agent extract those interfaces, then use them as grounding for a new session.
One less nice way of achieving the same is to tell the agent once in a while that it should think completely from scratch (from first principles). But this relies heavily on instruction following in the reasoning part, which sometimes works and sometimes fails.
It's a losing proposition. Self-driving cars go from point A to point B, and it's really easy to specify point B. My experience so far using agentic AI 24/7 for almost a year has been the AI can drive almost anywhere, so the whole problem is specifying where to drive. This doesn't get solved with faster, better AI, because all that will do is allow it to autonomously drive your codebase to random places you didn't want it to go. Then you're still left with the problem of specifying the problem.
But then there's the cases where the AI can't actually drive. What if you tell the agent to invent AGI? Or time travel? Where does it drive to? Does it tell you that it can't? How does it know it can't just drive here? How you you know? At the end of the day these are not wish granting machines, so someone connected to reality is still going to have to make decisions. And that person is going to be the one held liable for whatever the AI does so would they want a self-driving codebase in the first place?
Well put. I've found that agents are great at adding features to areas of the code that I already understand, where I can phrase my goals coherently and keep them on the rails.
Any attempt to describe software by "specification" leaves enough ambiguity that the agent will do deranged stuff like add a ton of code to satisfy error cases that can't happen. Or, like you imply, it will confidently create a bunch of nonsense to "solve" a problem in a way that is not really possible.
I'm hoping the "blogging about doing" format will fade away, now that doing is cheaper than blogging. It used to be that the doing part was expensive, and risky. Now you can just do 5 things and throw away 4. So I'd rather see the finished product than talk about it.
The missing piece in most versions of this is the feedback loop, not the harness. A codebase can only drive itself as far as it can tell whether it's still on the road: tests that fail for the right reason, error budgets, a way to notice that a change made something slower rather than broken.
Without that, "self-driving" degrades into applying diffs that compile. With it, the interesting question becomes what the system is allowed to do when the signal goes red - revert, retry, or stop and ask - and that's a policy decision nobody has good defaults for yet.
I'm playing around with ecosystem driven development (hear me out ;)):
So i want claude to build a small game for me so of course i create one big file in which i write everything then i tell claude to analyse it and grill me (grill me skill) to clarify all smaller details.
This alone might lead to really good small sfotware but I still have to push it sometimes.
Now instead of doing this, i tell claude to build a small tool which generates a dashboard and memory and which can save specs and ask me choicses (do you prefer this color over the other, what do you think about problem A? What solution would you prefer? A, b, c or something else).
This does a few things:
1. claude doesn't has to save/store everything in a context
2. claude can now talk to that tool to ask it stuff
3. claude can now use the tool as a todo list
4. claude now can more easily spin up more agents in parallel
5. i have a nice interface and i can solve issues while claude works on unblocked tasks
6. I actually can follow the progress a lot easier
The only problem with this is: with the next update, you have to reinvestigate how claude was finetuned and adjust. A few month ago /goal was really good, now you need it a lot less because claude will do something for an hour without /goal
And the spec file only started to work after November/Opus moment but it got so good, that i can pack A LOT of stuff in a half structured markdown file and let it code what I need.
The progress is still too fast for the whole ecosystem
It's simple: when a mistake happens, you run your CAPA process (Google CAPA form and see examples to extrapolate what that process might look like) and determine the root cause and the correction to the process that allowed the mistake to happen in the first place.
(At least as a SaaS vendor in life sciences, when we had a CAPA (e.g. after a SEV0 failure), it would be folded into our SOPs and then we would be required to retrain on the SOP. Auditors would want to see our evidence of CAPAs, the versions of our SOPs, the records of training. All to extreme for most shops, but I add this for context/color)
This is something most eng shops do not have the discipline for since it requires some diligence.
Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
The more important question - how would you actually get LLM agents to follow the instructions in your ever-growing CAPA reliably?
It’s all very well having a list of actions to avoid but that doesn’t help if your agents won’t reliably follow it.
You stop relying on the agents following instructions exactly.
You need two pieces:
a) prompts, that tell the agents what to do and how to do it (and ideally, the why, where, etc, the full picture) - that's the positive half, that drives behavior the way you want it.
b) deterministic tooling that prevents negative outcomes, like linters, compilers, static analysis, fuzzing, testing, the more the better. This side should either be firewalled off from the AI or very carefully watched so that it doesn't drift.
The part that you put in the deterministic side is the "never do x" stuff - I have lint for long comments (which AI hits every single time it commits), all my dev scripts are in typescript, precommit hooks, massive CI, and I lint even for things like redirecting error to standard out, tiny stuff, and also e.g. static migration analysis so the AI never ships an exclusive full table lock in a migration, for example.
yes, this is a very helpful approach. I take a similar approach, partly because I find bringing up negative directions seem to bring the undesired action/direction into the agent's view and makes it more likely that they will go there
You can’t deterministically keep them from making even a tiny fraction of all the possible errors they can and do make though.
You can’t keep humans from making those errors either but you also don’t let an error prone human crank out 20k LOC per day without forcing other humans to understand it.
This may be platform dependent.
C# Roslyn Analyzers[0], for example, are quite powerful and can identify complex patterns in code. One approach to deterministic enforcement would be to ensure that the project is set up with an analyzers library and mistakes that can be deterministically flagged are
[0] https://learn.microsoft.com/en-us/visualstudio/code-quality/...
I don't know if that's necessarily true in practice the way it is in theory. If you knock down 95% of the issues they create over a certain period of time, in practice you'll have better code than a human would write, and it's all upside from there.
I've been using languages with stronger type systems and that's also a huge boon.
Why couldn't you express all those as test cases rather than instructions?
In test cases i can do anything, a test framework is just a way of discovering and then scheduling functions to run. I can emit useful instructions to the agent from the failed test case: "After walking the AST of all use of state machine X, a branch was found at Y which reused stale state. Ensure stale references are dropped..."
I can force the agent to pass the test suite before it considers itself done. I can reject edits of such test cases to partially mitigate reward hacking. etc etc
You can express them as tests, but you also need a feedback mechanism that creates the rule that when the LLM generates some net new code or performs some refactor, that there are these CAPAs that it needs to cover with test cases.
The CAPA is a learning that sits outside of the mechanism of verification; it is a record of problem:root_cause:preventative_action. I see it as the instruction that would be required to generate the test case to prevent the next occurrence of a class of failures.
In a real-world process, for example, there is usually a QA lead that is verifying that the process is followed by looking at the paperwork and evidence.
> Why couldn't you express all those as test cases rather than instructions?
How would prevent removing the test case and the code that it tests? That's a legitimate thing to do when you're modifying a codebase. My intuition is "corrective and preventative actions (CAPA)" is a level or two up from test cases.
This is a question of context management and I suppose some would classify this as "harness engineering" as the trend of the moment.
One approach, for example, might be to have the a standalone code reviewer agent that is solely responsible for interfacing with the CAPA system (e.g. via a tool, via MCP) and acts as a back stop. When it finds a new type of CAPA, it stores it (and the backend indexes it with enough metadata to support broad types of retrieval). When it reviews a piece of code, it finds past CAPAs. By file locality. By business domain in the application. By keywords.
Same tool and repository available to both building agents and review agents, but use the review agent as a dedicated back stop as part of the verification process.
The same way you do for humans, regular training and audits.
I think the only way for now it through distillation of your own models, which can get expensive fast.
Every place I've worked at has positioned their SEV review process as 'blameless' with respect to the engineers who created the code, processes, etc that led to the incident. On paper, it's not about pointing fingers (though I've found this to be somewhat idealistic), but about finding faults in the processes that let bad code slip through the cracks.
It will be interesting to see how the notion of 'responsibility' carries over as agents handle higher and higher levels of abstraction. Agents built using the popular frameworks of today are somewhere in between code, formerly written by engineers, and an actor (like an engineer). When an agent wrote the framework to audit and monitor ops agents who are monitoring the performance and reliability of the code that coding agents wrote, who is responsible for ultimately ensuring it doesn't happen again? Should the findings just be added to whatever RAG and a few prompts/hooks/skills changed by another agent? When does a human follow up and to what degree?
This is the sort of thing that makes me believe that software engineers will never truly go away; at the very least, they serve the very useful function of being ultimately responsible for something going wrong.
> Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
I think a point we are grappling with is, what necessitates human intervention, like philosophically. Is it accountability? I was thinking about this in terms of code review and it's not like we would fire someone if they broke prod, so at what point do we need someone accountable. Maybe it's for approvals for certain classes of risk (like those associated with actual harm to life). It's also not lost on me that many human systems lack accountability.
I think there's a pretty clear and wide gulf between "nobody could have seen that coming", or "I just wasn't on my A game", and frank negligence.
I once worked at a very large company, and one of my coworkers was like "eh, I'm not going to do any of this work, because I don't have to", and that was a clear indicator that the problem wasn't (just) with the system, but with him individually.
I've had a similar experience, also at a very large organization. In my case this person chose that kind of environment because it allowed them to work on side stuff while waiting for approvals, etc. A lot of people in very large companies don't view the grind of organizational friction inhibiting their actions as problematic.
This is a general principal in the eng shop I work for, its written into every step, including the SCRUM etc.. What when well, why did it fail, what could be done better. Its all part of root cause analysis.
I personally use it for the agents I use, so why did it fail, what was the root cause, what can we do to prevent it again.
Agreed that domain experts should be making these kinds of decisions, and the right way for an agent mistake to get caught is going to depend a lot on the cost and probability of the mistake. My only point here is that the agent side of the system needs to learn. The software production machine needs to improve over time.
Sorry to get hung up on what is probably just flavor text, but the first sentence.. "Agents can oneshot games that are actually fun", uhhh such as?
Would love to be proven wrong here, feels like I've just seen the same basic one-level kart racer in like 3 model announcements. Not even sure if they were one-shotted, I certainly would not describe them as "fun"
As someone who builds games, particularly games meant to be played in the browser or apps, it can 100% one-shot a game-like experience, but it will not be very good.
You need to playtest the hell out of games. Play a bit, iterate, play a bit more, iterate, etc. You will get way better results if you start small and don’t try to one-shot it. Start small, vertical slice, playtest, and go fro there.
Yeah, that goes against my experience. Agents can absolutely one-shot games, but the gap between something that looks like a game and something that is fun to play and worth trying for more than a few minutes is gaping wide.
Probably fun defined in the ephemeral context. As-in it was "fun" to play in the context of the short form video attention span, social media brain rotted individual. Then it was thrown away and never played again.
It's definitely not fun if you consider building an actual community around it and have a long term plan for it.
Agreed, I read this and almost instinctively decided to close the page, agents can be used to build games for sure, but that "one shot" and "fun" in the same sentence is really misleading
FYI, it looks like you've been banned for the last ~8-9 months, for no obvious reason I can see. I vouched for your comment to make it visible, but they appears to show up as [dead] by default.
Kind of feels like we're in a similar place with AI programming as we are with self driving vehicles. Good enough for the typical case, maybe even better than most humans, but bad when dealing with novel situations or situations which require a really cohesive model of the system (code, business requirements, physical world, whatever).
Makes sense to me - I see it going down two separate routes in parallel:
One is to take the human out the OODA loop for cyber defense. Servlet libs (for example) are going to become fluid, self-modifying things w/ contracts that operate much differenlty from how they do today. The engineering practice around these things will need to change.
The second is that UX will be self-modifying. Just like how pi can modify itself, I can see this being a general practice for user-facing applications. Perhaps a text-box in which users can describe tweaks in can request changes to how the application functions. Engineering an application will focus on modeling the non-negotiables of how an application works, and providing the correct primitives for user-driven LLMs to modify software on the fly, as well as track and rollback changes. There'll also be funsies around how to ensure that ads get delivered regardless of the user trying to get rid of them.
> fluid, self-modifying things w/ contracts
This smells like worm food. Those contracts had better be airtight.
What would you prefer: 99% of user requests to succeed but w/ 5% risk of getting O-day'd, or have 90% of user requests succeed but w/ 0.1% risk of getting 0-day'd.
I could imagine it going so far as to have an LLM re-impl core libs every time a request is processed. Lightning never strikes the same place twice.
If you have a logo in the top left corner of your company blog the logo should route to the home page so I can see what product you're actually building.
This would be more convincing if there were a simple stat on the home page of the dispositions of the reported bugs (fixed/no-repro/won't-fix).
As a random sample of one, I looked at one of the bugs this reported on Tailscale (first thing on the homepage) [0], and the pull request ends with "Apologies for the lack of due diligence here. I'll go ahead and close this out."
[0] https://github.com/tailscale/tailscale/pull/17843
We should publish some stats. The fix rates on bugs are around 45% baseline, increasing over time as we learn from behavior within a given codebase. The merge rates for the codebase health work we do (e.g. deleting dead code) are very high, at least 90%.
My answer to this is adding an engineering assurance module to my specification and verification toolkit. Every project has a different definition of success and quality. My goal is to provide a consistent way to define, analyze, and report quality metrics tailored to each project.
https://github.com/agent-ix/engineering-assurance https://github.com/agent-ix/quoin
My bet is that we'll see a second layer of harness emerge, as self-driving codebases become the target. There will be an application facing harness, orthogonal to the agent facing harness. The app harness will represent the software factory that is emergent for the specific application being developed.
Anyway, here's mine, still wip:
https://hale-lang.org/docs/dna/
https://github.com/hale-lang/hale/issues/690
I suspect it won't be a harness, but just a more specific LLM trained in the universe of user-selected context of vetted resources.
Why? Because LLMs are always going to be dumb when they're trained at scale. Their ability to speak software diverges from their friendly user input layer. A harness won't overcome that, but an LLM saddle ontop of a larger model would provide the type of feedback loops you'd want to look into.
I don't think you'll find two deterministic systems will produce much.
I think that dissolves the "self driving" distinction, though? The mechanisms for driving the codebase must be present in the codebase itself. Otherwise its just a regular out-of-band development process.
fine-tuning may be a more scalable approach to LLM personalization than sending all the same context to two LLMs
I'm working towards both in my homelab to see which works better with little qwen
Author here, hello! Happy to expand on how we're thinking about this if any of it is unclear.
We wrote this post as part of a launch, which you can check out here: https://x.com/danlovesproofs/status/2095182189499711759
> Global memory
I don't get why we need global memory for code? Aren't code comments (even if invented for humans) the ideal place where to put "memories"?
LLMs also do not have anything like memory, by design. Everything bolted on that smells like memory or is being called memory is a crutch, at best. Not being pedantic, I just think this aspect is lost on a lot of people.
Also, when we humans are tricked into perceiving a remote mind, we automatically assume it has a memory like us, or at least like large animals.
In other words, there's a story-document between a SherlockHolmesBot and UserPlaceholder that's growing like a crystal formation. Some software sees "Sherlock Homes Says" and then "performs" the dialogue at us, and we then assume Sherlock Holmes exists with a mind and memory, rather than being a facet of a text-story.
Sometimes, but I think there are a few problems with this:
1. Some context doesn't have an obvious place to write it in the code. If you're explaining why a tricky function is implemented a certain way, you can leave the explanation above the function or within the function. If you're explaining something more general, there might not be a natural place to put it.
2. Relatedly, some context doesn't have an obvious location to read in the code. E.g. you can comment on a schema that a table is intended to be append-only, but an agent could easily miss this comment if it doesn't gather context all the way down to the raw schema. Any research process has unknown unknowns. There may not be a canonical place to look.
3. Adding comments for every single human intent might be a bit noisy. E.g., if a code reviewer flags something that I know isn't a bug, I want the system to learn from this, and I want that knowledge to take effect outside of just code review, but I might not want every single code review thread to yield a codebase comment.
1: There are corner cases, no doubt, but the main source of information should remain near the code.
2 & 3: How to be sure that the agent will pick the "append only hint" from the memory? Especially when it starts growing and cannot be part of the context?
I think the bigger need is for outside concerns, like: what is the infrastructure like? How much volume does this feature handle? How should changes to live code be rolled out to prevent running processes from failing?
In addition to all the product details that aren't in the codebase or docs, like "keep this logical path because it is used by our one big client."
Mostly what I think is needed is a richer worldview available to the LLM so it understands not just the code but can understand the product and its real world usage and constraints.
But how you are going to link the "keep this logical path ..." with the code? You cannot feed all the memory to the agent because of the context rot, and you are not sure the agent will always pick it when needed.
I have integrated AI into my application. The source code is embedded into the binary. When someone questions the numbers in my app it can debug itself. If it truly finds a bug in itself it will tell the user and open a ticket. Soon we are going to let the application fix itself.
How do you protect intellectual property? Or is this a case of the value being somewhere else, such as in your backend? If so, how does the agent debug frontend and backend? I presume it stops at frontend
Not sure about you, but I've been writing real world software using agents for a while now. https://github.com/jaksa76/microfactory
Is this the "real world software" or a tool that creates "real world software"?
This article hints at it, but I think the blocker is environment.
How can you self drive an app on windows? There's no clear UI framework, design pattern, nothing which can bullet proof your app.
I bet you'd have better luck on Plan 9.
Remove the choice. Make frameworks which have limited options which are usable by default.
Spot on about environment.
I recently built a calculator to try to quantify the gains an org could expect as they become more "AI-native." I tried to account for a bunch of things including env, which I call "AI roadway," but I'm definitely missing codebase readiness (thank you, author), and also bottleneck analysis. Even without those considerations, though, gains are usually modest. It's hard to get to 2x.
Tool is here if useful: https://timvasil.com/ai-native
I suspect that if you're not careful with agent memory it creates a danger of agent-driven cargo-cult behavior. I've watched this in my own ad-hoc agent loops where it starts with something basic, like the first agent tried to run some gigantic dependency inspection command and OOMed the local JVM and eventually recorded a workaround (to enable it to run gigantic dependency inspection commands...), and by time I get a few more agents into the loop, agents have written entire paragraphs about testing and validating local dev environment memory configurations that are mostly irrelevant to whatever is being worked on.
In general I've seen other issues like this where small errors and irrelevant comments in the codebase spin out into larger problems that consume annoying amounts of time/tokens. Maybe Anthropic and OpenAI don't notice this because they're in an "infinite monkeys with typewriters" scenario, but it's noticeable to me when the agent in my CLI has been spinning for 15 minutes contemplating irrelevant details
This happens way too often in my experience and I tried a lot of things. If the agent has access to an existing codebase it will always try to stay as close to it as possible. It doesn’t diverge from the initial direction the project/code went into, even if it’s already clear that this direction goes nowhere. Somehow agents aren’t able to detach themselves from existing code to come up with completely new solutions.
Sometimes I feel like it would be best to only give the agent access to clear API boundaries (say public interfaces to certain modules) and let it work out a new system from scratch given the expected inputs and outputs. Then plug this independent solution into the system. Of course it can still overfit these interfaces but it’s less than having access to the entire codebase.
Thinking about it, maybe it’s possible to let one agent extract those interfaces, then use them as grounding for a new session.
One less nice way of achieving the same is to tell the agent once in a while that it should think completely from scratch (from first principles). But this relies heavily on instruction following in the reasoning part, which sometimes works and sometimes fails.
It's a losing proposition. Self-driving cars go from point A to point B, and it's really easy to specify point B. My experience so far using agentic AI 24/7 for almost a year has been the AI can drive almost anywhere, so the whole problem is specifying where to drive. This doesn't get solved with faster, better AI, because all that will do is allow it to autonomously drive your codebase to random places you didn't want it to go. Then you're still left with the problem of specifying the problem.
But then there's the cases where the AI can't actually drive. What if you tell the agent to invent AGI? Or time travel? Where does it drive to? Does it tell you that it can't? How does it know it can't just drive here? How you you know? At the end of the day these are not wish granting machines, so someone connected to reality is still going to have to make decisions. And that person is going to be the one held liable for whatever the AI does so would they want a self-driving codebase in the first place?
Well put. I've found that agents are great at adding features to areas of the code that I already understand, where I can phrase my goals coherently and keep them on the rails.
Any attempt to describe software by "specification" leaves enough ambiguity that the agent will do deranged stuff like add a ton of code to satisfy error cases that can't happen. Or, like you imply, it will confidently create a bunch of nonsense to "solve" a problem in a way that is not really possible.
I'm hoping the "blogging about doing" format will fade away, now that doing is cheaper than blogging. It used to be that the doing part was expensive, and risky. Now you can just do 5 things and throw away 4. So I'd rather see the finished product than talk about it.
“how can we make ourselves even MORE replaceable?”
Reminds me of that Richard Scarry children's book illustration of a fox busily sawing off a tree branch he's sitting on.
I currently see a shit ton of work involved with "scaling" (aka cleaning up) the AI generated slop that was output in the past 2-3 years.
The missing piece in most versions of this is the feedback loop, not the harness. A codebase can only drive itself as far as it can tell whether it's still on the road: tests that fail for the right reason, error budgets, a way to notice that a change made something slower rather than broken.
Without that, "self-driving" degrades into applying diffs that compile. With it, the interesting question becomes what the system is allowed to do when the signal goes red - revert, retry, or stop and ask - and that's a policy decision nobody has good defaults for yet.
I'm playing around with ecosystem driven development (hear me out ;)):
So i want claude to build a small game for me so of course i create one big file in which i write everything then i tell claude to analyse it and grill me (grill me skill) to clarify all smaller details.
This alone might lead to really good small sfotware but I still have to push it sometimes.
Now instead of doing this, i tell claude to build a small tool which generates a dashboard and memory and which can save specs and ask me choicses (do you prefer this color over the other, what do you think about problem A? What solution would you prefer? A, b, c or something else).
This does a few things:
1. claude doesn't has to save/store everything in a context 2. claude can now talk to that tool to ask it stuff 3. claude can now use the tool as a todo list 4. claude now can more easily spin up more agents in parallel 5. i have a nice interface and i can solve issues while claude works on unblocked tasks 6. I actually can follow the progress a lot easier
The only problem with this is: with the next update, you have to reinvestigate how claude was finetuned and adjust. A few month ago /goal was really good, now you need it a lot less because claude will do something for an hour without /goal
And the spec file only started to work after November/Opus moment but it got so good, that i can pack A LOT of stuff in a half structured markdown file and let it code what I need.
The progress is still too fast for the whole ecosystem