I'm running a server on AWS with TimescaleDB on the disk because I don't need much. I figure I'll move it when the time comes.
Claude Code this morning was about to create an account with NeonDB and Fly.io although it has been very successful managing the AWS EC2 service.
Claude Code likely is correct that I should start to use NeonDB and Fly.io which I have never used before and do not know much about, but I was surprised it was hawking products even though Memory.md has the AWS EC2 instance and instructions well defined.
> Claude Code likely is correct that I should start to use NeonDB and Fly.io which I have never used before and do not know much about
I wouldn't be so sure about that.
In my experience, agents consistently make awful architectural decisions. Both in code and beyond (even in contexts like: what should I cook for a dinner party?). They leak the most obvious "midwit senior engineer" decisions which I would strike down in an instant in an actual meeting, they over-engineer, they are overly-focused on versioning and legacy support (from APIs to DB schemas--even if you're working on a brand new project), and they are absolutely obsessed with levels of indirection on top of levels of indirection. The definition of code bloat.
Unless you're working on the most bottom-of-the-barrel problems (which to be fair, we all are, at least in part: like a dashboard React app, or some boring UI boilerplate, etc.), you still need to write your own code.
I find they are very concerned about ever pulling the trigger on a change or deleting something. They add features and codepaths that weren't asked for, and then resist removing them because that would break backwards compatibility.
In lieu of understanding the whole architecture, they assume that there was intent behind the current choices... which is a good assumption on their training data where a human wrote it, and a terrible assumption when it's code that they themselves just spit out and forgot was their own idea.
From what you said it sounds like the conclusion should be "you still need to design the architecture yourself", not necessarily "you still need to write your own code".
Yeah, I actually wanted to write an addendum, so I'll just do it here. I think that going from pseudocode -> code is a pretty neat concept (which is kind of what I mean by "write your own code"), but not sure if it's economically viable if the AI industry weren't so heavily subsidized by VC cash. So we might end back up at writing actual code and then telling the AI agent "do another thing, and make it kinda like this" where you point it to your own code.
I'm doing it right now, and tbh working on greenfield projects purely using AI is extremely token-hungry (constantly nudging the agent, for one) if you want actual code quality and not a bloated piece of garbage[1][2].
> even though Memory.md has the AWS EC2 instance and instructions well defined
I will second that, despite the endless harping about the usefulness of CC, it's really not good at anything that hasn't been done to death a couple thousand times (in its training set, presumably). It looks great at first blush, but as soon as you start adding business-specific constraints or get into unique problems without prior art, the wheels fall off the thing very quickly and it tries to strongarm you back into common patterns.
Advertisers will only pay if AI providers will provide them data on the equivalent of “ad impressions”. And unlabeled/non-evident advertisements are illegal in many (most?) countries.
It doesn't necessarily have to be advertisers paying AI providers. It could be advertisers working to ensure they get recommended by the latest models. The next form of SEO.
What coding with LLMs have taught me, particularly in a domain that's not super comfortable for me (web tech), is that how many npm packages (like jwt auth, or build plugins) can be replaced by a dozen lines of code.
And you can actually make sense of that code and be sure it does what you want it to.
Good report, very important thing to measure and I was thinking of doing it after Claude kept overriding my .md files to recommend tools I've never used before.
The vercel dominance is one I don't understand. It isn't reflected in vercel's share of the deployment market, nor is it one that is likely overwhelming prevalent in discourse or recommended online (possible training data). I'm going to guess it's the bias of most generated projects being JS/TS (particularly Next.js) and the model can't help but recommend the makers of Next.js in that case.
I'm already seeing a degradation in experience in Gemini's response since they've started stuffing YouTube recommendations at the end of the response. Anthropic is right in not adding these subtle(or not) monetization incentives.
This is funny to me because when I tell Claude how I want something built I specify which libraries and software patents I want it to use, every single time. I think every developer should be capable of guiding the model reasonably well. If I'm not sure, I open a completely different context window and ask away about architecture, pros and cons, ask for relevant links or references, and make a decision.
Patents aren't vulnerable to cleanroom reverse engineering. You can create something yourself in your bedroom and use it yourself without knowing the patented thing exists, and still violate the patent. That's why they're so scary.
You won't get caught if you write something yourself and use it yourself, but programmers (contrary to entrepreneurs) have a pattern of avoiding illegal things instead of avoiding getting caught.
The sad part is that most software patents are so woefully underspecified and content-free that even Claude might have trouble coming up with an actual implementation.
Unrelated to the topic at hand but related to the technologies mentioned. I weep for Redux. It's an excellent tool, powerful, configurable, battle tested with excellent documentation and maintainer team. But the community never forgave it for its initial "boilerplate-y" iterations. Years passed, the library evolved and got more streamlined and people would still ask "redux or react context?" Now it seems this has carried over to Claude as well. A sad turn of events.
Redux is boring tech and there is a time and place for it. We should not treat it as a relic of the past. Not every problem needs a bazooka, but some problems do so we should have one handy.
Redux should not be used for 1 person projects. If you need redux you'll know it because there will be complexity that is hard to handle. Personally I use a custom state management system that loosely resembles RecoilJS.
Well, the tech du jour now is whatever's easier for the AI to model. Of course it's a chicken and egg problem, the less popular a tech is the harder it is to make it into the training data set. On the other hand, from an information theoretic point of view, tools that are explicit and provides better error messages and require less assumptions about hidden state is definitely easier for the AI when it tries to generalize to unknowns that doesn't exist in its training data.
This seems web centric and I expect that colors the decision making during this analysis somewhat.
People are using it for all kinds of other stuff, C/C++, Rust, Golang, embedded.
And of course if you push it to use a particular tool/framework you usually won't get much argument from it.
Worth reading alongside recent research on AGENTS.md file effectiveness. The clearest use case for these files isn't describing your codebase, it's overriding default behavior. If your project has specific requirements around tooling (common in government and regulated industries), that's exactly what belongs in the AGENTS.md files.
I found it a remarkable transition to not use Redis for caching from Sonnet 4.5 to Opus 4.6. I wonder why that is the case? Maybe I need to see the code to understand the use case of the cache in this context better.
It really disappointing to see it so strongly preferring Github Actions which is in my experience terrible. Almost everything about GHA pushes you in the direction of constantly blowing out the 10GB cache limit in an attempt to have CI not run for ages. I also feel like the standard cache action using git works poorly with any tools that use mtime on files to determine freshness.
I guess at least Opus can help you muddle through GHA being so crappy.
LLMs are going to keep React alive for the indefinite future.
Especially with all the no-code app building tools like Lovable which deal with potential security issues of an LLM running wild on a server, by only allowing it to build client-side React+Vite app using Supabase JWT.
Not sure what to make of this. React is missing entirely. Or is this report also assuming that React is the default for everything and not worth mentioning at all? Just like shadcn/ui's first mention of React is somewhere down the page or hidden in the docs?
Furthermore, what's the point of "no tools named"? Why would I restrict myself like that? If I put "use Nodejs, Hono, TypeScript and use Hono's html helper to generate HTML on the server like its 2010, write custom CSS, minimize client-side JS, no Tailwind" in CLAUDE.md, it happily follows this.
As someone who runs a small dev agency, I'm very interested in research like this.
Let's say some Doctor decides to vibecode an app on the weekend, with next to 0 exposure to software development until she started hearing about how easy it was to create software with these tools. She makes incredible progress and is delighted in how well it works, but as she considers actually opening it up the world she keeps running into issues. How do I know this is secure? How do I keep this maintained and running?
I want to be in a position where she can find me to get professional help, so it's very helpful to know what stacks these kinds of apps are being built in.
claudecode _loves_ shadcn/ui. I hadn't even heard of it until i was playing around with claudecode. It seems fine to me and if the coding agent loves it then more power to it, i don't really care. That's the problem.
I think that makes coding agent choices extremely suspect, like i don't really care what it uses as long as what's produced works and functions inline with my expectations. I can totally see companies paying Anthropic to promote their tool of choice to the top of claudecodes preferences. After thinking about it, i'm not sure if that's a problem or not. I don't really care what it uses as long as my requirements (all of them) are met.
I didn't read the report just the "finding" - but at least for launchdarkly it's nice that it chose a roll-your-own, i hate feature flag SaaS, but that's just me
I'm running a server on AWS with TimescaleDB on the disk because I don't need much. I figure I'll move it when the time comes.
Claude Code this morning was about to create an account with NeonDB and Fly.io although it has been very successful managing the AWS EC2 service.
Claude Code likely is correct that I should start to use NeonDB and Fly.io which I have never used before and do not know much about, but I was surprised it was hawking products even though Memory.md has the AWS EC2 instance and instructions well defined.
> Claude Code likely is correct that I should start to use NeonDB and Fly.io which I have never used before and do not know much about
I wouldn't be so sure about that.
In my experience, agents consistently make awful architectural decisions. Both in code and beyond (even in contexts like: what should I cook for a dinner party?). They leak the most obvious "midwit senior engineer" decisions which I would strike down in an instant in an actual meeting, they over-engineer, they are overly-focused on versioning and legacy support (from APIs to DB schemas--even if you're working on a brand new project), and they are absolutely obsessed with levels of indirection on top of levels of indirection. The definition of code bloat.
Unless you're working on the most bottom-of-the-barrel problems (which to be fair, we all are, at least in part: like a dashboard React app, or some boring UI boilerplate, etc.), you still need to write your own code.
I find they are very concerned about ever pulling the trigger on a change or deleting something. They add features and codepaths that weren't asked for, and then resist removing them because that would break backwards compatibility.
In lieu of understanding the whole architecture, they assume that there was intent behind the current choices... which is a good assumption on their training data where a human wrote it, and a terrible assumption when it's code that they themselves just spit out and forgot was their own idea.
How do you make an LLM that’s was trained on average Internet code not end up as a midwit?
Mediocrity in, mediocrity out.
From what you said it sounds like the conclusion should be "you still need to design the architecture yourself", not necessarily "you still need to write your own code".
Yeah, I actually wanted to write an addendum, so I'll just do it here. I think that going from pseudocode -> code is a pretty neat concept (which is kind of what I mean by "write your own code"), but not sure if it's economically viable if the AI industry weren't so heavily subsidized by VC cash. So we might end back up at writing actual code and then telling the AI agent "do another thing, and make it kinda like this" where you point it to your own code.
I'm doing it right now, and tbh working on greenfield projects purely using AI is extremely token-hungry (constantly nudging the agent, for one) if you want actual code quality and not a bloated piece of garbage[1][2].
[1] https://imgur.com/a/BBrFgZr
[2] https://imgur.com/a/9Xbk4Y7
But he did design the architecture:
> even though Memory.md has the AWS EC2 instance and instructions well defined
I will second that, despite the endless harping about the usefulness of CC, it's really not good at anything that hasn't been done to death a couple thousand times (in its training set, presumably). It looks great at first blush, but as soon as you start adding business-specific constraints or get into unique problems without prior art, the wheels fall off the thing very quickly and it tries to strongarm you back into common patterns.
This is where LLM advertising will inevitably end up: completely invisible. It's the ultimate "influencer".
Or not even advertising, just conflict of interest. A canary for this would be whether Gemini skews toward building stuff on GCP.
Richard Thaler must be proud. This is the ultimate implementation of "Nudge"
I wonder if aggregators will emerge (something like Ground News does for news sources)
LLM pattern [0] will probably eventually emerge as the best way to fight those biases. This way everyone benefits from token burn!
[0](https://github.com/karpathy/llm-council)
Advertisers will only pay if AI providers will provide them data on the equivalent of “ad impressions”. And unlabeled/non-evident advertisements are illegal in many (most?) countries.
It doesn't necessarily have to be advertisers paying AI providers. It could be advertisers working to ensure they get recommended by the latest models. The next form of SEO.
That's called LLM SEO now I believe.
What coding with LLMs have taught me, particularly in a domain that's not super comfortable for me (web tech), is that how many npm packages (like jwt auth, or build plugins) can be replaced by a dozen lines of code.
And you can actually make sense of that code and be sure it does what you want it to.
Good report, very important thing to measure and I was thinking of doing it after Claude kept overriding my .md files to recommend tools I've never used before.
The vercel dominance is one I don't understand. It isn't reflected in vercel's share of the deployment market, nor is it one that is likely overwhelming prevalent in discourse or recommended online (possible training data). I'm going to guess it's the bias of most generated projects being JS/TS (particularly Next.js) and the model can't help but recommend the makers of Next.js in that case.
I just got an incredible idea about how foundation model providers can reach profitability
I'm already seeing a degradation in experience in Gemini's response since they've started stuffing YouTube recommendations at the end of the response. Anthropic is right in not adding these subtle(or not) monetization incentives.
is it anything like the OpenAI ad model but for tool choice haha
Claude Free suggests Visual Studio.
Claude Plus suggests VSCode.
Claude Pro suggests emacs.
I'm not quite sure if you're making fun of emacs or actually praising it.
Stallman paying for advertising, now that is good one :)
Copilot suggests leftpad
> ~~Claude Pro suggests emacs.~~
Claude Pro asks you about your preferences and needs instead of pushing an opinionated solution?
I'd thought about model providers taking payment to include a language or toolkit in the training set.
Hence the claw partnership.
This is funny to me because when I tell Claude how I want something built I specify which libraries and software patents I want it to use, every single time. I think every developer should be capable of guiding the model reasonably well. If I'm not sure, I open a completely different context window and ask away about architecture, pros and cons, ask for relevant links or references, and make a decision.
You specify which software patents you want it to use?
AI reading the patent is basically cleanroom reverse engineering according to current AI IP standards :D
Patents aren't vulnerable to cleanroom reverse engineering. You can create something yourself in your bedroom and use it yourself without knowing the patented thing exists, and still violate the patent. That's why they're so scary.
You won't get caught if you write something yourself and use it yourself, but programmers (contrary to entrepreneurs) have a pattern of avoiding illegal things instead of avoiding getting caught.
It's not a perfect joke I'll admit.
The sad part is that most software patents are so woefully underspecified and content-free that even Claude might have trouble coming up with an actual implementation.
Patterns?
Yeah patterns. lol!
Unrelated to the topic at hand but related to the technologies mentioned. I weep for Redux. It's an excellent tool, powerful, configurable, battle tested with excellent documentation and maintainer team. But the community never forgave it for its initial "boilerplate-y" iterations. Years passed, the library evolved and got more streamlined and people would still ask "redux or react context?" Now it seems this has carried over to Claude as well. A sad turn of events.
Redux is boring tech and there is a time and place for it. We should not treat it as a relic of the past. Not every problem needs a bazooka, but some problems do so we should have one handy.
Redux should not be used for 1 person projects. If you need redux you'll know it because there will be complexity that is hard to handle. Personally I use a custom state management system that loosely resembles RecoilJS.
More like redux vs zustand. Picking zustand was one of the good standout picks for me.
Well, the tech du jour now is whatever's easier for the AI to model. Of course it's a chicken and egg problem, the less popular a tech is the harder it is to make it into the training data set. On the other hand, from an information theoretic point of view, tools that are explicit and provides better error messages and require less assumptions about hidden state is definitely easier for the AI when it tries to generalize to unknowns that doesn't exist in its training data.
> Traditional cloud providers got zero primary picks
Good - all of them have a horrible developer experience.
Final straw for me was trying to put GHA runners in my Azure virtual net and spent 2 weeks on it.
This seems web centric and I expect that colors the decision making during this analysis somewhat.
People are using it for all kinds of other stuff, C/C++, Rust, Golang, embedded. And of course if you push it to use a particular tool/framework you usually won't get much argument from it.
Worth reading alongside recent research on AGENTS.md file effectiveness. The clearest use case for these files isn't describing your codebase, it's overriding default behavior. If your project has specific requirements around tooling (common in government and regulated industries), that's exactly what belongs in the AGENTS.md files.
It still ignores it. I always have to say 'Isn't this mentioned in AGENTS??' and it will concede that it is.
Have any links?
I'll be interested to hear stories - down the line - from the participants in the the LLM SEO war [1].
Interesting that tailwind won out decisively in their niche, but still has seen the business ravaged by LLMs.
[1] https://paritybits.me/copilot-seo-war/
It's like tailwindcss was purposely designed to be managed my LLM.
I found it a remarkable transition to not use Redis for caching from Sonnet 4.5 to Opus 4.6. I wonder why that is the case? Maybe I need to see the code to understand the use case of the cache in this context better.
It really disappointing to see it so strongly preferring Github Actions which is in my experience terrible. Almost everything about GHA pushes you in the direction of constantly blowing out the 10GB cache limit in an attempt to have CI not run for ages. I also feel like the standard cache action using git works poorly with any tools that use mtime on files to determine freshness.
I guess at least Opus can help you muddle through GHA being so crappy.
LLMs are going to keep React alive for the indefinite future.
Especially with all the no-code app building tools like Lovable which deal with potential security issues of an LLM running wild on a server, by only allowing it to build client-side React+Vite app using Supabase JWT.
Not sure what to make of this. React is missing entirely. Or is this report also assuming that React is the default for everything and not worth mentioning at all? Just like shadcn/ui's first mention of React is somewhere down the page or hidden in the docs?
Furthermore, what's the point of "no tools named"? Why would I restrict myself like that? If I put "use Nodejs, Hono, TypeScript and use Hono's html helper to generate HTML on the server like its 2010, write custom CSS, minimize client-side JS, no Tailwind" in CLAUDE.md, it happily follows this.
As someone who runs a small dev agency, I'm very interested in research like this.
Let's say some Doctor decides to vibecode an app on the weekend, with next to 0 exposure to software development until she started hearing about how easy it was to create software with these tools. She makes incredible progress and is delighted in how well it works, but as she considers actually opening it up the world she keeps running into issues. How do I know this is secure? How do I keep this maintained and running?
I want to be in a position where she can find me to get professional help, so it's very helpful to know what stacks these kinds of apps are being built in.
claudecode _loves_ shadcn/ui. I hadn't even heard of it until i was playing around with claudecode. It seems fine to me and if the coding agent loves it then more power to it, i don't really care. That's the problem.
I think that makes coding agent choices extremely suspect, like i don't really care what it uses as long as what's produced works and functions inline with my expectations. I can totally see companies paying Anthropic to promote their tool of choice to the top of claudecodes preferences. After thinking about it, i'm not sure if that's a problem or not. I don't really care what it uses as long as my requirements (all of them) are met.
Because the primary and future audience of Claude et al don’t know the tools they want, or even that a choice exists.
> Furthermore, what's the point of "no tools named"?
There are vibe coders out there that don't know anything about coding.
I mean, i guess that will shortly put an end to the "no code" movement.
I didn't read the report just the "finding" - but at least for launchdarkly it's nice that it chose a roll-your-own, i hate feature flag SaaS, but that's just me