Maybe I missed it, but it look like this has just a single metric. Maybe instead of making a new project, you could try to get this metric added to a existing tool like https://dekobon.github.io/big-code-analysis/index.html which already has dozens of metrics.
Yes, I'm doing my own research on AI augmented pipelines https://blog.officefloor.net . I actually found most code quality tools look for bugs and complexity, but nothing much about cohesive erosion. The nice thing about this metric, is that it determine the files where the erosion is occurring. I turned it into a GitHub action to make it easier to access to get wider feedback on the metric. The GitHub action triggers on your merge request and tells you the files where erosion is occurring to refactor. This stops erosion before it gets too expensive to change (big refactors or rewrite). Yes, happy to work with others to get the metric into other tools.
I just dump all user messages from all sessions in a project into a flat .md file and have agents synthesize the user's intent. Then, using that extracted intent, the agents review code and tests. I call this a retro/reflection pass. It checks whether the code matches the intent and whether the tests match the code.
Compactly formatted user messages are something an agent can ingest in a few minutes, even if they are thousands of lines long. And the quality of those messages is great: they don't track what the agent does well, only what changes and what breaks.
Having this top-down view helps a lot. Usually, within a session and deep into a task, the agent loses the global perspective and optimizes for local success. I find it weird there is no harness that treats user messages as high value signal (except my own, of course, I have it, https://github.com/horiacristescu/playbook-harness).
The problem with specification and user discussion is they still have errors that code has. But unlike code, there are no tests to confirm correctness.
So now we have a definition of the system in a non-exact language with no ability to test to confirm it's correctness. The code holds the essential complexity and now we are adding accidental complexity on top to manage.
Again agree the specifications and user discussion provides context for the AI. However, a well written test suite provides similar context that can actually confirm correctness of the system.
However, saying all the above. Focus of ImpactGate ( https://impactgate.officefloor.net ) is about erosion of the code, not correctness.
Comes from the basic Computer Science principals of High Cohesion and Low Coupling.
High cohesion means the functionality of a component are closely related and focused on performing a single well defined task. Basically single classes for single purposes.
Erosion of this is when classes start doing to many things, in the case of God classes.
The Change Impact formula looks at a way of detecting when the cohesion is eroding and flagging it on a change (as the pull/merge request itself should generally be single focus cohesive change)
I've never encountered that term before (cohesive erosion) but I like it, if I'm interpreting it correctly.
Do you mean like the hyper focus an LLM puts on the task in front of it so you end up with drift (duplicated concepts/multiple ways of doing things, terminology drift (e.g., now we have "customer" and "client"). That sort of thing?
When you think about a god class or god method, it occurs over time by adding more than a single responsibility.
Yes, there are generally complex algorithms but they usually are not things developers write (imported from libraries).
What is usually going on in the god class/method is that things keep getting added to it. These things should be separated out. So the cohesiveness of the class/method erodes into doing too many things.
The idea of the Change Impact formula is to catch this early so you start refactoring to separate out into classes with single cohesive purposes.
The problem with AI is it handles complexity really well and will happily keep piling changes into god classes/methods reaching ridiculous CC levels (have see over 200). Previously developers would get annoyed and do the refactor. But with AI these days, changes are happening faster. So Change Impact is to try to monitor the cohesive erosion.
That project in itself looks very interesting. How are people using it, any examples of how people get this into an actual report / CI test / benchmark / whatever ?
Code metrics in general aren’t that widely used. I’ve only ever worked at one place (a bank) that tracked it, and that was only because sonarcube had it built in.
While a lot of metrics make intuitive sense, we don’t have that much hard evidence to prove or disprove their value. Part of it is the whole “if a metric becomes a target, it ceases to be a good metric” thing. Adding the checks to a large existing project probably has negative value. But I think it’s worth doing for greenfield projects.
For humans, these should just be advisory. But for LLMs I’m happy enough to make it a blocking check.
I keep thinking of doing an experiment where I give the same LLM the same problem, and only change which metric is enforced. And then see if any of them have a noticeable effect on correctness/maintainability.
> any examples of how people get this into an actual report / CI test / benchmark / whatever ?
Yeah they have examples of adding it to CI, or local checks, generate html reports, etc in their docs.
I think the future of development will be a lot of automated quality checks like these on AI-drafted code that humans review and ensures that it doesn't muck up the business logic and actually fulfills the acceptance criteria. that said, I don't think existing toolsets are really great at actually measuring code quality
I've been in the process of reviewing and validating a lot of tools like this (qlty, Sonarqube, fallow, etc) and the false positive rate is anywhere from 20% to 80% for a lot of our sniff tests (zizmor produces an overwhelming majority of false positives here for what feels like arbitrary and very context-dependent GHA requirements)
the last thing I want to do is to annoy the hell out of our devs by requiring checks like these to pass especially since it's only a small percentage of them who vibe code everything and then also vibe response to code reviews. I feel like that's the anti-pattern that we'd push people towards by requiring checks like these to pass
another avenue of exploration has been requiring test coverage but also good test quality metrics (eg are there negative tests? mutation testing? empty asserts?) something that seems quite easy to spin up into a skill and pair with a deterministic harness. trash-tests is a neat little project that incorporates some of this: https://github.com/frangelbarrera/trash-tests (disclosure: I am not the repo owner or even a contributor, just a quality nerd who loves underdogs lol)
all in all, it really does feel like we'll need a revamp of the SDLC with our current expected velocities
Yep, agree on annoying developers. So just to cover usability it can run in warn and block mode to address this.
Though to the bigger point of your comment, yes SDLC are becoming faster. We can churn out code at a ridiculous rate. However, doesn't mean it's good code. And hence, there are studies showing things actually slowing down because reviews pile up.
I guess I look at the Change Impact formula (and https://impactgate.officefloor.net implementation of it) as threshold tool. Small changes that aren't contributing to god classes, just let through. When things start to smell, the files involved get marked for review.
Ideally this then can cut down on review time and allow overall increased velocity.
But yes relies on trusting the AI to do "simple" things
Yep, this all actually started because of experimenting with my own open source project https://officefloor.net (giving full disclosure)
I was testing the additive pipeline style of OfficeFloor against the mutative handler style of Spring. I was looking to see what factors could be used to allow AI to make long on going changes (experiment is 60 changes to an end point, where all add functionality and every 4th change is mutative on existing rules). Then I watch how AI manages to make the 60 changes in each architecture.
I've done many runs and you are quite right about Goodhart effect in giving it the metric. Never knew Spring code could be written so badly.
I've tried runs with better prompting also and I'm starting to find the key factor is actually the architecture itself.
From my initial findings, it's seeming that additive pipeline architectures hold up much better against AI slop than our typically single method web handler architectures.
Nice idea. Our new CTO brought in a tool he made for analyzing cyclomatic complexity and it’s been useful since we’re a heavily AI-forward shop.
BTW, you can avoid your comments being flagged and killed by writing them yourself! I know it’s tempting to offshore it to AI (especially after you’ve vibe-coded a whole project) but some genuine human communication goes a long way.
I know cyclomatic complexity has been heavily debated for a long time, but I do think it’s valuable. It’s really good at highlighting common annoyances like overly clever code, nested ternaries, dense functions with too many branches…
The only thing is that these issues seem like human code problems and IME LLMs don’t really write code like this anymore. It’s almost the opposite in python, actually, where Claude leans on writing lots of 2-3 liner private utils which is a separate kind of complexity and organization problem.
I still find it useful specifically for React where it’s frustratingly normalized to write many branches in your JSX though.
The difference to previous CC use, is the the change impact formula looks at the complexity already in the class/file. Typical CC just looks at the function it is change and not the context of the change. The Change Impact formula incorporates that to avoid god class and god method issues. Plus multiplying by number of files punishes for non-cohesive code bases. For me it puts the intuition of high cohesion and low coupling into a measurable metric.
What score do we get for a group of fixes to single line bugs that are obvious once spotted?
There is an implication that all AI changes add complexity and reduce quality, but it is obvious that the quality and complexity is a property of the code not the writer, so all equivalent changes should be equal no matter what the shource.
How do clear improvements to make something simpler yet more functional score under this system?
The problem with cohesion is understanding what is "single purpose". Nothing can determine this without interpreting the code and making a judgement call on whether it needs to be refactored into two or more classes.
So I'm looking to approximate this. Instead of looking for cohesion, Change Impact formula approximates this by looking at existing complexity in the file.
If the change adds more complex functions to an already complex class, it scores high. Smell of it doing too many things.
If the change adds a complex function to an empty file, it's likely just a complex single problem.
If the change adds simple change to complex file, then if this happens too many times, we flag it for concern. Especially if it changes a lot of other files too.
So it's not mean to measure your code exactly. It's meant to highlight smell that needs investigation. And the formula points out the classes that need looking at.
Plus I'm not sure that all changes are equivalent. I'm not looking at single one off changes. I'm looking at 60 changes in a row. The first 20 might go through fine. However, after the changes pile on, there needs to be some refactoring to keep the code clean. This is trying to catch it before it becomes a mess.
> If the change adds a complex function to an empty file, it's likely just a complex single problem.
How do you determine that this is correct = adding complexity to a single file vs adding less complexity to multiple files that this one file then orchestrates.
Yes, you are right that looking at one commit doesn't tell you much.
It's looking at the nature of on going changes.
A single complex file written once and never touched is likely a good cohesive file about a single problem solved.
However, if we start making many changes to that file increasing it's complexity, it's likely not the problem getting more complex. It's likely we're adding more purposes to the file, eroding it's cohesion. This is the signal I'm trying to catch to avoid a mess starting.
Right, so really any fresh file is more or less fine (would still evaluate cyclomatic complexity). It's the quantity/frequency of changes that matters (or CRAP)
- mutative architecture (Spring) which as single method handling the request. It grows in complexity as changes are added
- additive architecture (OfficeFloor) which does function injection. A pipeline of functions, where new functionality is an injected function as a step in the pipeline (i.e. adding functions, rather than mutating a single handler function)
What I've found is the complexity of the overall system is what it is. Between both architectures, the overall CC is the same. However, the distribution is very different. Mutative architectures pile changes into god handler method. Additive architectures spread the complexity into smaller manageable files.
This, however, does not come with perfect wins. Spread also means AI doesn't read the dispersed files. AI is good with complexity and tends to fix things when it sees them (so does fix bugs in god classes). But this is only to a certain point when the complexity then comes to overwhelm the AI.
The formula is trying to catch the mess of on going changes at a point before the expensive refactoring or rewriting becomes necessary.
Personally I’m ok with vibe coded projects - certainly feels like the future of things, and I think the line between vibe coded and “professionally” coded is increasingly blurring - but I completely agree on the marketing/communications piece. Ideally your communication about a project conveys real expertise and ownership, signaling that you really understand the problem you’re trying to solve and the tradeoffs you made in your approach to do so. I am very hesitant to use a project where it feels like the eng couldn’t pass a pop quiz about how it works + why.
Most static code analysis looks for bugs or just too complex functions.
The novelty of Change Impact is it looks at the context of the complexity. Adding complexity to something already complex should be flagged for review.
I've not come across any static code analysis so far that has provided this.
And in my own research of architectures, I came to this formula.
Kind of ironic that this thing purports to “gate” slop, yet the entire project is slop, including the README and even the authors comments here. I hate this timeline.
Claude is pretty good at adding focused changes and if a fix is already present, then it correctly points it out rather than adding unnecessary refactors.
Yes, I've run it against a bunch of open source projects with long histories (before AI) to see if it predicts bugs. Seems file size is still a better predictor. However, for the projects where good coding was strictly adhered to and others that were not, it showed the differences appropriately. So I've found it useful in general for Software erosion.
Maybe I missed it, but it look like this has just a single metric. Maybe instead of making a new project, you could try to get this metric added to a existing tool like https://dekobon.github.io/big-code-analysis/index.html which already has dozens of metrics.
Yes, I'm doing my own research on AI augmented pipelines https://blog.officefloor.net . I actually found most code quality tools look for bugs and complexity, but nothing much about cohesive erosion. The nice thing about this metric, is that it determine the files where the erosion is occurring. I turned it into a GitHub action to make it easier to access to get wider feedback on the metric. The GitHub action triggers on your merge request and tells you the files where erosion is occurring to refactor. This stops erosion before it gets too expensive to change (big refactors or rewrite). Yes, happy to work with others to get the metric into other tools.
I just dump all user messages from all sessions in a project into a flat .md file and have agents synthesize the user's intent. Then, using that extracted intent, the agents review code and tests. I call this a retro/reflection pass. It checks whether the code matches the intent and whether the tests match the code.
Compactly formatted user messages are something an agent can ingest in a few minutes, even if they are thousands of lines long. And the quality of those messages is great: they don't track what the agent does well, only what changes and what breaks.
Having this top-down view helps a lot. Usually, within a session and deep into a task, the agent loses the global perspective and optimizes for local success. I find it weird there is no harness that treats user messages as high value signal (except my own, of course, I have it, https://github.com/horiacristescu/playbook-harness).
Keeping all the specifications and user discussion does create more context, which is useful for AI.
However, I'd bring in Brooks discussion on essential and accidental complexity. In other words, there being No Silver Bullet https://www.cs.unc.edu/techreports/86-020.pdf
The problem with specification and user discussion is they still have errors that code has. But unlike code, there are no tests to confirm correctness.
So now we have a definition of the system in a non-exact language with no ability to test to confirm it's correctness. The code holds the essential complexity and now we are adding accidental complexity on top to manage.
Again agree the specifications and user discussion provides context for the AI. However, a well written test suite provides similar context that can actually confirm correctness of the system.
However, saying all the above. Focus of ImpactGate ( https://impactgate.officefloor.net ) is about erosion of the code, not correctness.
What exactly is “cohesive erosion”?
Comes from the basic Computer Science principals of High Cohesion and Low Coupling.
High cohesion means the functionality of a component are closely related and focused on performing a single well defined task. Basically single classes for single purposes.
Erosion of this is when classes start doing to many things, in the case of God classes.
The Change Impact formula looks at a way of detecting when the cohesion is eroding and flagging it on a change (as the pull/merge request itself should generally be single focus cohesive change)
I've never encountered that term before (cohesive erosion) but I like it, if I'm interpreting it correctly.
Do you mean like the hyper focus an LLM puts on the task in front of it so you end up with drift (duplicated concepts/multiple ways of doing things, terminology drift (e.g., now we have "customer" and "client"). That sort of thing?
When you think about a god class or god method, it occurs over time by adding more than a single responsibility.
Yes, there are generally complex algorithms but they usually are not things developers write (imported from libraries).
What is usually going on in the god class/method is that things keep getting added to it. These things should be separated out. So the cohesiveness of the class/method erodes into doing too many things.
The idea of the Change Impact formula is to catch this early so you start refactoring to separate out into classes with single cohesive purposes.
The problem with AI is it handles complexity really well and will happily keep piling changes into god classes/methods reaching ridiculous CC levels (have see over 200). Previously developers would get annoyed and do the refactor. But with AI these days, changes are happening faster. So Change Impact is to try to monitor the cohesive erosion.
That project in itself looks very interesting. How are people using it, any examples of how people get this into an actual report / CI test / benchmark / whatever ?
Code metrics in general aren’t that widely used. I’ve only ever worked at one place (a bank) that tracked it, and that was only because sonarcube had it built in.
While a lot of metrics make intuitive sense, we don’t have that much hard evidence to prove or disprove their value. Part of it is the whole “if a metric becomes a target, it ceases to be a good metric” thing. Adding the checks to a large existing project probably has negative value. But I think it’s worth doing for greenfield projects.
For humans, these should just be advisory. But for LLMs I’m happy enough to make it a blocking check.
I keep thinking of doing an experiment where I give the same LLM the same problem, and only change which metric is enforced. And then see if any of them have a noticeable effect on correctness/maintainability.
> any examples of how people get this into an actual report / CI test / benchmark / whatever ?
Yeah they have examples of adding it to CI, or local checks, generate html reports, etc in their docs.
I think the future of development will be a lot of automated quality checks like these on AI-drafted code that humans review and ensures that it doesn't muck up the business logic and actually fulfills the acceptance criteria. that said, I don't think existing toolsets are really great at actually measuring code quality
I've been in the process of reviewing and validating a lot of tools like this (qlty, Sonarqube, fallow, etc) and the false positive rate is anywhere from 20% to 80% for a lot of our sniff tests (zizmor produces an overwhelming majority of false positives here for what feels like arbitrary and very context-dependent GHA requirements)
the last thing I want to do is to annoy the hell out of our devs by requiring checks like these to pass especially since it's only a small percentage of them who vibe code everything and then also vibe response to code reviews. I feel like that's the anti-pattern that we'd push people towards by requiring checks like these to pass
another avenue of exploration has been requiring test coverage but also good test quality metrics (eg are there negative tests? mutation testing? empty asserts?) something that seems quite easy to spin up into a skill and pair with a deterministic harness. trash-tests is a neat little project that incorporates some of this: https://github.com/frangelbarrera/trash-tests (disclosure: I am not the repo owner or even a contributor, just a quality nerd who loves underdogs lol)
all in all, it really does feel like we'll need a revamp of the SDLC with our current expected velocities
Yep, agree on annoying developers. So just to cover usability it can run in warn and block mode to address this.
Though to the bigger point of your comment, yes SDLC are becoming faster. We can churn out code at a ridiculous rate. However, doesn't mean it's good code. And hence, there are studies showing things actually slowing down because reviews pile up.
I guess I look at the Change Impact formula (and https://impactgate.officefloor.net implementation of it) as threshold tool. Small changes that aren't contributing to god classes, just let through. When things start to smell, the files involved get marked for review.
Ideally this then can cut down on review time and allow overall increased velocity.
But yes relies on trusting the AI to do "simple" things
Yep, this all actually started because of experimenting with my own open source project https://officefloor.net (giving full disclosure)
I was testing the additive pipeline style of OfficeFloor against the mutative handler style of Spring. I was looking to see what factors could be used to allow AI to make long on going changes (experiment is 60 changes to an end point, where all add functionality and every 4th change is mutative on existing rules). Then I watch how AI manages to make the 60 changes in each architecture.
I've done many runs and you are quite right about Goodhart effect in giving it the metric. Never knew Spring code could be written so badly.
I've tried runs with better prompting also and I'm starting to find the key factor is actually the architecture itself.
From my initial findings, it's seeming that additive pipeline architectures hold up much better against AI slop than our typically single method web handler architectures.
Nice idea. Our new CTO brought in a tool he made for analyzing cyclomatic complexity and it’s been useful since we’re a heavily AI-forward shop.
BTW, you can avoid your comments being flagged and killed by writing them yourself! I know it’s tempting to offshore it to AI (especially after you’ve vibe-coded a whole project) but some genuine human communication goes a long way.
I know cyclomatic complexity has been heavily debated for a long time, but I do think it’s valuable. It’s really good at highlighting common annoyances like overly clever code, nested ternaries, dense functions with too many branches…
The only thing is that these issues seem like human code problems and IME LLMs don’t really write code like this anymore. It’s almost the opposite in python, actually, where Claude leans on writing lots of 2-3 liner private utils which is a separate kind of complexity and organization problem.
I still find it useful specifically for React where it’s frustratingly normalized to write many branches in your JSX though.
The difference to previous CC use, is the the change impact formula looks at the complexity already in the class/file. Typical CC just looks at the function it is change and not the context of the change. The Change Impact formula incorporates that to avoid god class and god method issues. Plus multiplying by number of files punishes for non-cohesive code bases. For me it puts the intuition of high cohesion and low coupling into a measurable metric.
[dead]
Is it public? Would love to try it!
Sonarqube has existed for like 20 years. There's doezens of cyclomatic complexity, linter, and cve scanner tools though.
What score do we get for a group of fixes to single line bugs that are obvious once spotted?
There is an implication that all AI changes add complexity and reduce quality, but it is obvious that the quality and complexity is a property of the code not the writer, so all equivalent changes should be equal no matter what the shource.
How do clear improvements to make something simpler yet more functional score under this system?
Yes, the formula does make some approximations.
The problem with cohesion is understanding what is "single purpose". Nothing can determine this without interpreting the code and making a judgement call on whether it needs to be refactored into two or more classes.
So I'm looking to approximate this. Instead of looking for cohesion, Change Impact formula approximates this by looking at existing complexity in the file.
If the change adds more complex functions to an already complex class, it scores high. Smell of it doing too many things.
If the change adds a complex function to an empty file, it's likely just a complex single problem.
If the change adds simple change to complex file, then if this happens too many times, we flag it for concern. Especially if it changes a lot of other files too.
So it's not mean to measure your code exactly. It's meant to highlight smell that needs investigation. And the formula points out the classes that need looking at.
Plus I'm not sure that all changes are equivalent. I'm not looking at single one off changes. I'm looking at 60 changes in a row. The first 20 might go through fine. However, after the changes pile on, there needs to be some refactoring to keep the code clean. This is trying to catch it before it becomes a mess.
> If the change adds a complex function to an empty file, it's likely just a complex single problem.
How do you determine that this is correct = adding complexity to a single file vs adding less complexity to multiple files that this one file then orchestrates.
Yes, you are right that looking at one commit doesn't tell you much.
It's looking at the nature of on going changes.
A single complex file written once and never touched is likely a good cohesive file about a single problem solved.
However, if we start making many changes to that file increasing it's complexity, it's likely not the problem getting more complex. It's likely we're adding more purposes to the file, eroding it's cohesion. This is the signal I'm trying to catch to avoid a mess starting.
Right, so really any fresh file is more or less fine (would still evaluate cyclomatic complexity). It's the quantity/frequency of changes that matters (or CRAP)
Yes
The research I'm actually doing is comparing:
- mutative architecture (Spring) which as single method handling the request. It grows in complexity as changes are added
- additive architecture (OfficeFloor) which does function injection. A pipeline of functions, where new functionality is an injected function as a step in the pipeline (i.e. adding functions, rather than mutating a single handler function)
What I've found is the complexity of the overall system is what it is. Between both architectures, the overall CC is the same. However, the distribution is very different. Mutative architectures pile changes into god handler method. Additive architectures spread the complexity into smaller manageable files.
This, however, does not come with perfect wins. Spread also means AI doesn't read the dispersed files. AI is good with complexity and tends to fix things when it sees them (so does fix bugs in god classes). But this is only to a certain point when the complexity then comes to overwhelm the AI.
The formula is trying to catch the mess of on going changes at a point before the expensive refactoring or rewriting becomes necessary.
The overuse of “gate” in this post title makes me think the entire thing was vibe coded even the marketing.
Personally I’m ok with vibe coded projects - certainly feels like the future of things, and I think the line between vibe coded and “professionally” coded is increasingly blurring - but I completely agree on the marketing/communications piece. Ideally your communication about a project conveys real expertise and ownership, signaling that you really understand the problem you’re trying to solve and the tradeoffs you made in your approach to do so. I am very hesitant to use a project where it feels like the eng couldn’t pass a pop quiz about how it works + why.
How the change impact formula works: https://blog.officefloor.net/2026/08/measuring-blast-radius-...
Seems like the posts on this site are largely LLM-generated though? Correct me if I’m wrong.
"That is a lot to ask of one number. Here is how it fits into one. Built one decision at a time." it sure sounds like Claude.
[dead]
It is called Impact"Gate" https://github.com/marketplace/actions/impact-gate
Uncanny how any coding model I use will employ the word "gate".
Probably because I've had Quality Gates in my pipelines for so long :)
What languages does this support? I'm guessing it's Python only, but it would be nice if that was mentioned somewhere.
It uses lizard for parsing, so Python, Java, JavaScript, TypeScript, C, C#, Go, Scala, and more.
What's the structural decay score of this HN title? Apparently not high enough to be gated.
Isn't this just static code analysis? We have tools for this.
Most static code analysis looks for bugs or just too complex functions.
The novelty of Change Impact is it looks at the context of the complexity. Adding complexity to something already complex should be flagged for review.
I've not come across any static code analysis so far that has provided this.
And in my own research of architectures, I came to this formula.
Kind of ironic that this thing purports to “gate” slop, yet the entire project is slop, including the README and even the authors comments here. I hate this timeline.
Great way of detecting AI slop.
Claude is pretty good at adding focused changes and if a fix is already present, then it correctly points it out rather than adding unnecessary refactors.
I'm not looking at building prototypes. Looking at ways to manage code bases after they've gone through hundreds if not thousands of changes.
Interesting idea even for non-AI code.
Yes, I've run it against a bunch of open source projects with long histories (before AI) to see if it predicts bugs. Seems file size is still a better predictor. However, for the projects where good coding was strictly adhered to and others that were not, it showed the differences appropriately. So I've found it useful in general for Software erosion.
A sloppy project to find slop in projects. It sure works well
[flagged]
[dead]
[flagged]