We studied this question pretty systematically in the MirrorCode paper [1], comparing Python, C, Rust, Go, OCaml, and Ada across 19 very long-horizon tasks, for Claude Opus 4.7 and GPT-5.5.
> In our results, there was little sign of inter-language differences in solve rates, for any model (Figure 5b). This suggests that AI models have learned generalized programming skills, rather than pattern-matching syntax. This does not mean that implementation language is irrelevant. Conditional on solving a target, we found a small effect on token usage: successful Python solutions tended to use fewer tokens than average, while successful Ada solutions tended to use more (Appendix C). We consider these to be small differences, given that these six programming languages vary widely in how concise they are, and in how much functionality is provided by their standard library (recall that agents cannot download dependencies in MirrorCode, they must solve the task using only the standard library).
In Appendix C, Ada tended to use only about 25% more tokens than the average language. Ada is a language used mainly in safety-critical aerospace and defense systems, which has ~200x less pre-training data available than C or Python.
We're also comparing more recent language models (on just Go vs Ada, for cost reasons), on our leaderboard [2].
Reading your quote really gets me wondering who the people making those kinds of analysis are...
It's like they haven't maintained any actual software, because the criteria they choose is... Completely irrelevant?
The things that matter are tooling, orchestration and ecosystem - as well as how the LLM will actually implement the solution for a task
LLMs constantly do idiotic things. If you have good libraries to utilize, the likelihood of the solution actually working goes up because they no longer need to implement the hard part.
If you have orchestration for dependency injection, code generation, meta analysis etc
Tooling like the way otel tracing is integrated, openapi generation etc is also invaluable because every time the LLM does something the likelihood of it being hallucinated/wrong increases etc
You'd need to implement a nontrivial project in different languages, then add nontrivial features across them and only then start by rating eg correctness and incident occurrence after the final output
But token use on a one shot? Completely irrelevant as far as I see it.
I'm not sure I trust a source that says "just 70 tokens average, nearly half of Clojure (109 tokens)".
There's no reason to add the phrase "nearly half of", and there's especially no reason to add it when it's significantly far away from half.
But on the main topic, I still feel that Go is an excellent choice for LLMs. There is pretty much just one way of doing most things, and the available training data is pretty consistent. This is very different from Python, where training data is polluted (I presume) with tons of code written by non-software engineers and demonstrating many different ways of doing the same thing.
Also a big plus for Go is the tooling. Fast compiles and good linting shortens the iteration cycle time, resulting in less need for me to tell the LLM to correct mistakes.
For some reason, most LLMs I've used default to wanting to write Python. I have to repeatedly teach them to use Go unless there is a very compelling reason to choose otherwise.
I would personally rather see and use Clojure, but I don't feel its ecosystem would provide the same benefits as Go, including obviously the easy single binary distribution.
Go is absolutely one of the best programming languages for LLMs for the reason you say, and Python is just what LLMs like to use to write short throwaway scripts. Frontier LLMs are generally pretty good at most programming languages and can pick up new ones pretty quickly. Training data seems to mostly just increase the speed which they write code, for example, GPTs tend to write Rust and Python faster than other programming languages.
For actual output quality, the main deciding factor is simply how much tooling it is there for the LLMs to check their own work, as LLMs seemed to avoid using a lot of libraries in general. That's why C# is underrated due to the tooling strength of the .NET ecosystem, as long as you tell LLMs to avoid using reflections unless absolutely necessary.
C++ is also surprisingly good, but you pretty much have to tell the LLMs to treat it like Go and don't use any of the dangerous features for normal code.
I think a lot of people are sleeping on the advantages of "batteries included" ecosystems.
The need to select an appropriate 3rd party library represents an entire dimension of the search space that can be eliminated. Imagine having to make this choice multiple times per day when your competition is just mindlessly using System.* types. The fact that the .NET ecosystem is curated by one entity should not be underestimated.
Even when we do need to import 3rd party nugets, the models seem to follow this highly structured pattern. They scan the xml docs, and failing that they will build a throwaway console app to reflect over all the unique types and build a report. The fact that we can easily do this with a simple powershell command makes a big difference. How many other ecosystems can even consider doing this? Reflection is a superpower, not something to be avoided.
Reflection is good for prototyping and get something setup quickly, but if you build your architecture around it, not only do you lose access to NativeAOT, the code becomes very hard to debug, and if you code with LLM a lot, you either have to spend time trying to debug reflections or just rewrite it with source generation to begin with, which is at least honest about the metaprogramming there.
Reflection is just such a dangerous feature that looks like ordinary code, which is why it is something to be avoided, and having an LLM write/analyze the code for eliminates the need to use a lot of reflective code to begin with.
> C++ is also surprisingly good, but you pretty much have to tell the LLMs to treat it like Go and don't use any of the dangerous features for normal code.
For existing codebases I made the experience, that LLMs are very good at replicating their style.
At work most of our C++ codebases use a fairly consistent style and subset of C++ features and to my initial surprise specifying style conventions etc explicitly turned out to be mostly superfluous.
Of course, we also have some legacy projects originally, written in ANSI C, which only received a few changes in the last 15 years to compile with a C++ compiler. Here a style guide is helpful, bit I consider it more like a temporary instruction for refactoring.
If you want the opposite experience, try LLMs with an older Oracle Database. Even with the official Oracle SqlDeveloper MCP Server they struggle a lot.
However, I struggle as well and the error messages of Oracle are some of the worst I've ever experienced.
> Go is absolutely one of the best programming languages for LLMs for the reason you say, and Python is just what LLMs like to use to write short throwaway scripts.
And yet this article has pretty strong empirical data to show that your intuition here is incorrect. You should back up your statement with something more than vibes.
What is output quality without correctness? That seems like a distinction without a difference.
Is the claim that LLMs produce Go code that is superficially nice looking but in fact fail to solve the stated problem? Because that's an anti-Go position I'd say.
Correctness is a suitable property to act as a multiplier in your formula, where incorrect is 0 and correct is 1, but you also need other facets to find a quality gradient.
> This is very different from Python, where training data is polluted (I presume) with tons of code written by non-software engineers and demonstrating many different ways of doing the same thing.
Counter-example: agents with Django-related stuff. Excellent output.
I think you will find that is a supporting example. Django pushes for a particular style and structure, which is a similar property found in the Go community.
LLMs seem to fall apart where human written projects of the same nature had no particular way about them. It is especially apparent when treading into waters where beginners are found. Like the earlier comment suggests, this is presumably because the LLMs struggle to find any kind of pattern to latch onto. Django offers a pattern, but one not shared by rest of the Python ecosystem. Whereas virtually all Go codebases look the same.
> "They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt."
This makes it a great language not just for young Googlers programmers, but also for LLM Agents!
IMO, the next big language will be similar philosophy, but without garbage collection. (is Zig the closest to filling that niche?)
> This is very different from Python, where training data is polluted (I presume) with tons of code written by non-software engineers and demonstrating many different ways of doing the same thing.
Python's philosophy is there is one way to do it, as opposed to Perl's TIMTOWTDI.
Your statement also assumes that 'software engineers' write the best code, and from my experience, this is definitely not true
I believe the training data should simply be limited to only code written by someone like Fabrice Ballard, or whoever you think writes the best code.
But that's the trouble, Python has the slogan about only one way, but it's really not true in practice. Or maybe there's the one way that "should" be done, and then the half dozen other ways you'll encounter in the wild, as gp alluded.
Python is the language where in practice this is most untrue - maybe outside C++.
As an example, there are dicts, tuples, classes, NamedTuples, dataclasses, attr.s, Pydantic, and pretty much all of those solve similar problem (hold my data) but have slightly different properties and use cases.
Aren't LLMs a way to somehow extract the one way it should be done (or to be more precise, the more common way), over the half other ways? That correct way might be more difficult to extract from another languages that encourage multiple valid ways.
Also, if you trust the benchmarks, it seems that Python is, at the very least, decent enough for LLMs. There seems to be "no trouble" in practice, unless you show us better proof than "I feel like it must be bad for this and that".
Contrary to what most comments seem to indicate, my takeaway from this is that it doesn't really matter all that much for the agents what language you pick. If humans are still to be involved in the process at some point, then its imperative that the language can be read by them, so the preference or skills of the developer(s) are of primary concern, not the agent.
Ive been amazed at how well LLMs are at writing Gleam[1] and Lustre[2]. Compared to a mainstream language, there is basically zero gleam code in the training data.
I have no evidence to back this up, but I suspect that languages that are good for humans[3] will be good for LLMs. Compiled, strongly typed, statically typed, immutable, pure functions, pattern matched, memory safe, etc.
[1] https://gleam.run
[2] https://lustre.hexdocs.pm
[3] Yes I realize that languages features that are "good for humans" is a hotly debated topic. That's just my personal list for what I like in a language.
I used to hold this opinion but since changing to Rust on the server and Typescript on the client, I can confidently tell you agents are so much better at Rust, especially at producing idiomatic code, than they are at Gleam.
There are reasons to love Gleam and Lustre (I like Gleam a lot), but LLMs just aren't one of them. I made the switch to Rust around May this year. Also the community is super anti-AI, arguably with good reason (how it impacts open source), and I'd recommend keeping your AI code to yourself.
I’ve been developing a language for a few years, and even with incomplete semantics and a simple one page example LLMs don’t have much trouble writing it.
I think the language/syntax has an impact, but the tooling around it will be most important for LLMs, in the same way it is for humans.
Gleam has been stable for over two years, so maybe it's been long enough that LLMs have internalised the documentation.
Given it's a language that doesn't really contain any groundbreaking ideas[1] (the closest is 'use' IMO), it's possible LLMs can reuse patterns from other functional language.
[1] This isn't criticism. I love how Gleam turned out.
That's not a take I was expecting to find here. I've found most LLMs absolutely dreadful when it comes to Gleam, to the point that I most often disable even inline autocomplete when working in Gleam codebases.
Too often I find them getting pulled into larger ruts in the training data and trying to insert language features that don't exist (ifs, loops, and syntactic constructs) from more popular languages like TypeScript and Rust. Do you not experience other languages getting partially substituted in when you have LLMs write Gleam?
I suspect it depends a lot on the llm/harness being used. But when I use Opus/cc or sol/codex, at the end of the turn everything compiles, passes tests, and passes lint. I never even look at code that can't compile. Maybe the LLM is generating weird stuff in-between, but I don't see it.
What you're describing feels like my experience back in 2024/25. Back then I was using a llm auto complete or the chat interface, and I would get weird stuff all the time. (not just gleam but any language).
For an even more niche language, Roc basically became usable in the new syntax about two months ago (still has compiler crashes, there's a good reason it hasn't had a real release) but Opus writes it just fine after a couple corrections to handle the language's quirks.
I don't care so much about token efficiency and cost, within reason. I care about whether the quality of the code is maintainable over time and through many iterations. My gut feeling is that very strict languages with good types, a standardized style, and very strong static analysis tools, is what helps make that happen. Of course it also has to be well-represented in the training data.
That leaves Go, Rust, Python with type annotations, and Typescript. And, I choose them in roughly that order unless there's a reason to choose otherwise. Rapid iterations on scripty tasks get Python. Most CLI, system services, and web apps are Go. Desktop apps and games are Rust. Typescript if I don't have a choice (i.e. it runs in a browser).
I wonder if there's correlations between tasks and languages, e.g. maybe R is better for bioinformatics tasks, python for webdev, C for CLIs, etc. I'd expect to see it because some languages are used more often in some tasks than others, but on the other hand LLMs can learn across languages and it's not clear if task-language use patterns are just historical or if the language is genuinely better at the task.
It's not clear to me how useful of a signal replicating existing pieces of well-known software is for this kind of evaluation, given what we know about how effectively LLMs can retrieve data from their training corpus and style-transfer it across different settings (programming languages here). That would explain their convergence in ability across different languages on the tasks in this post. I'd be far more interested in people's real-world experiences.
I tried writing an AI harness in Python. Seemed the obvious way to go. Tons of libraries. Libraries for talking to model APIs. Libraries for context and conversation management. Libraries for talking to MCPs. It is the language for LLMs!
It was a shit show and just couldn't write anything that would not crash. Super confident it had done a good job. Full of random bugs. A UI needs interactivity, interruption, handling exceptions. It produced some of the worst code I've ever seen. And looking at the libraries' code: also some of the worst code I've ever seen.
I switched to rust + tauri. In about three person weeks of work I have UI with forking conversations, tool use with built in grepping, tons of quality tools. It's more productive (for me) than Claude Code (CLI or desktop).
I was recently working on an AI harness too, but I wasn't using AI to code it. It's really easy and requires little code. I wasn't even using langchain - that would require even less code.
UI is harder for sure, but it's not that bad. You need to think though where to catch which exceptions.
LLMs might opt for langchain which has had multiple breaking changes after the knowledge cutoff, making it hard for the LLM to work with it. This is probably going to lead to the LLM having to make many changes to it's code, making it messy and leading to further code being less maintainable.
Yeah, I've had similar experiences, also starting out with dynamic languages and migrating to Rust. If the LLM will write a lot of the code for me, why not choose something (1) super fast, and (2) which has types I can use to understand and specify the code I want without having to read all the output?
I've been trying out Lean for related reasons, to good effect. It's really interesting there since it can crank out proofs that would've been completely infeasible for a dedicated team of PhDs before, whereas I haven't seen any LLM projects written in Python that I couldn't have slung out in a few months myself. I personally think it's a lot more interesting to focus on the new things you can now do with LLMs that weren't possible before, as opposed to doing the same old stuff at moderately higher velocity.
In my experience, Dart/Flutter has been so much better than React. Go has been really good for the backend. Basically if the framework/language gives you structure and one way to do things, agents tend to create less mess with less guardrails from you.
For me c wins here. It is compact, there are all language parts one needs and available and it well fitted to transport knowledge without much syntax hussle
What's optimal for LLMs and for people is probably not going to be the same. People are a bit lazy.
Coding agents do much more than generating code though. Much of what they do relates to validating that what was generated is a valid solution. That includes everything from type checking, running tests, static code analysis, linting, running code in a headless browser, etc. The more tools agents have at their disposal, the better the feedback loop gets. But of course some of these tools are costly to run.
Statically compiled languages have a head start here as they simply exclude entire categories of bugs that a dynamically typed language might have. And with things like type inference, their token overhead can be pretty minimal. Modern languages like Kotlin or Swift are pretty compact and don't really add a lot of bloat relative to say typescript/javascript. Go is a bit more verbose but tends to work well. Rust seems pretty popular with LLM users as well. The main challenge with languages like this is the performance hit you take running their build tools. Doing that a lot slows you down and it burns a lot of tokens as well.
Zstd gets rather easier from dotnet 11, it becomes a near one-limer since it's getting added into System.IO.comoression.
I know this because my agent already knew this the other day when I was evaluating compression, but that's because it has access to search.
That's a key part of what makes agents good coders too, mine is often looking up and downloading the source for how libraries are implemented.
It seems unnatural to air-gap them for evaluation.
I guess they didn't want them just finding an existing library to copy, but it's not very "real-world" to deny the ability to search quickly.
That said, the best language is still just the one you know. No amount of token saving is worth getting a bunch of code back you can't easily understand and review.
Cool line of questioning, but one piece of information is pivotal and critically not-yet-included: equivalent accomplishments in each language. For example, if I want to write standard things: web server, memoized fibonnaci, recipe search engine, what's the length-and-density of these outputs for each language? I think that would add in some ~normalization.
Strongly agree- this is how I “evaluated” languages pre-agents. though I suspect this would bias results in favor of whatever has best signal to noise for boilerplate from stackoverflow/reddit , rather than what LLM’s “””reason””” best with. (Presuming those aren’t quite one-and-the-same)
This is a great discussion: I wonder though if we are asking the right question. Yes, absolutely language choice can play a large role in the efficiency of coding agents. The point about Rust is right: static typing provides a fast verification loop at compile time. I would argue though that the way the codebase is composed could actually generalize the concept of "easy verifiability" past the actual coding language.
For instance, if an application can be broken down into components that have a verifiable contract in how they are to be used, then an LLM can load only the relevant modules into its context and fully understand how to use them and fix them if needed. It is also easier for the LLM to verify the functionality of a component rather than the entire system.
Additionally, in an application composed of functioning components, issues are more likely to occur at the boundaries between them, which the LLM can focus on rather than having to always consider the entire application that it most likely can't load fully into its context.
A well designed componentized Python application will likely be far more efficient for modification by an LLM than a large Rust monolith.
C and C++ do well because there is most literature and code out there to help them reason about it. C is helpful because it has little hidden runtime for them to trip over.
that being said, those languages obviously have limits in applicability looking at the entire spectrum of software. JS, python and others still have useful domains.
i dont think newer languages as rust are better for LLMs as they might be for new programmers. for new programmers they offer extra features but for an LLM this is added potential to make mistakes. Also a lot of newer languages are less stable so you can realise their current implementations might not be fully trained on by the models or even be after their cutoff date..
One thing worth noting is that syntactic density doesn't necessarily mean cheaper because because symbols don't chunk/tokenize as well as plain English
What I see from results like this is that the delta between languages is small enough now that it's hard to justify not not using something like Rust for the performance and correctness benefits if you're using LLMs and it fits the domain
How good or bad are LLMs on languages that have evolved over the years and aren't popular enough to get hand tuned?
Asking because for non programming, if you use them instead of a wiki for a topic that has had yearly changes for like 10 years they get confused and mix releases like crazy.
I discovered last week that Fable 5 can write perfect xTensa LX7 assembler code without tools or references. Mind blown.
But, when working on a creative graphics task, the results were best in Lua, middling in integer-only C, and underwhelming in ASM in terms of creative depth.
It's unclear that this is an advantage, certainly not in terms of "logic vs syntax".
First of all programs written in curly-brace languages still also have indentation to indicate statement grouping / blocks / scope, even if it's not required, so for a correct program (and that's not deliberately obfuscated), and one that's in the process of being written by an LLM, any advantage there disappears. Furthermore, having both indentation and explicit block markers provides redundancy which could be a significant advantage for an LLM (it being a probabilistic text / program generator). And for an incorrect program that redundancy is a big advantage for the LLM because it should be very easy for it to notice a mismatch of indentation and braces.
The only downside would be a very slightly higher token cost for the redundancy. I realize that Python comes out on or near the top in most of the comparisons in the linked article, but I doubt that's the reason.
The thing with LLM is they don't automatically pair parenesis/curly braces like we do with editors/IDEs.
The closing } ) ] token has to be generated to match exactly the beginning { ( [ many lines before. You can challenge yourself writing Lisp by hand without cursor moving backwards, and try close correctly by counting ))))))) you'd have a big headache.
A long, nested sub-routine with many () {} will cost LLM's context and makes it underperform, because the attention head have to track the state. On the other hand the indentation level can be infered as a single token[1] and saves reasoning effort.
Note these discussion is about "code generation", not parsing.
One thing that often happens with board games is rule issues in translations. Specifics that are clear in one language get lost in translation. Wolff Designa is out of Latvia. So not surprised there are some hard to interpret rules.
It’s interesting that LLMs struggle with the board game rules like we do. I think game designers should get the llm to teach them from their rulebook. If an LLM can’t understand the rules good chance people will also be confused.
> Dynamically typed languages generally have a lower LLM token cost than traditional statically typed languages because omitting explicit type declarations makes the code more compact.
If this was true, the programming languages that are very much on the left side of
should be very ideal for LLMs, in particular if they are dynamically typed.
What I can tell you is: I experimented with AI prompts for generating Wolfram (Mathematica) code using some LLMs, and I can tell you that the results were very disappointing: in my experience LLMs have difficulties with programming languages that are
- very concise, and
- for which there is less code publicly available.
Wolfram (Mathematica) is a good example of such a programming language.
I’ve actually found Sol delivers great results with Odin, despite there not being much Odin code available. I think it is able to work well with it because:
- It is a rather simple language
- It has a lot of very useful libraries already built in.
With just a single main.odin file you can do a heck of a lot stuff, which LLMs seem to like.
It’s interesting I’ve been surprised by how well Claude sonnet can write code in a language I’m developing that probably has no code in the training set. It seems like anything with syntax like python/ruby/elixir is pretty LLM friendly, and layering on a HM type system seems to help catch most errors.
> I think it is able to work well with it because:
> - It is a rather simple language - It has a lot of very useful libraries already built in.
> With just a single main.odin file you can do a heck of a lot stuff, which LLMs seem to like.
Also Wolfram/Mathematica has an insane amount of useful libraries already built in (there even exists the saying "Python is 'batteries included', Wolfram is 'spaceship included'"), and also there in a single file you can do a heck of a lot stuff.
On the other hand:
- LLMs tend to hallucinate non-existing function when you ask an LLM to code something in Wolfram that is not commonly done (concerning this point, nevertheless keep in mind that Wolfram is often used for "one-of-a-kind programs", i.e. for writing very specialized programs that have possibly never been done before).
- Wolfram code tends to be quite dense.
- If there is a small mistake in Wolfram code, the code typically simply won't work.
JS gives you wrong answers silently when thigns go sideways since for the original browser use case, they didn't want scripts to ever stop execution. It's the opposite of Python in this respect.
You are going to have to give more support for those assertions. I write Python every day, and never would I call it a good candidate for the clankers. Pretty much any dynamic language would be ruled out, as there is too much implicit logic which makes it harder to understand what is happening.
Python with a strict linter and type checker (eg ruff with the right lints on and ty with its stricter settings, or strict pyright if performance isn't too bad) works very well. Most of Python strengths (concise, large ecosystem, well represented in the LLM training data) while having good static analysis.
You don't need that, gets in the way more than it helps. Even Typescript isn't really needed, but at least it's decent devex unlike the Python typing stuff. What really helps is testing.
I try to capture as much as possible in types/schemas/constraints (then lint rules) and then only as a last resort write tests. And as few and complementary as possible. And I want a functional core with unit tests and imperative shell and not a bunch of complex mocks. ¯\_(ツ)_/¯
What's better for this, Go? That's the least verbose static one, and it's still a lot more verbose without helping you understand any better what it's doing. It's just faster. That's the real benefit of static types.
I'd like to see the results for Ada on these same measures. On the theory that the Ada type system covers more classes of errors than other languages, and so AI can self correct better.
Unfortunately for static type weenies like me (and you, presumably), types don't seem to matter at all, or Python and Javascript wouldn't be on top. There's no reason to believe that Ada's type system is so unique that it alone can help AI self-correct, and Rust, Haskell, ML, Typescript, etc. can't.
Well the reason I'm interested in Ada is because I saw a study that showed AI did worse at functional programming. So that might explain the problems with Haskell et al. But Ada has a strong type system while still having procedural code. So it would be an interesting comparison with Haskell etc. if the problem was that Haskell is functional or if the problem was that these are not so popular.
The study didn't really show that. What it did show, is that in this particular study there was no strong effect either way and there is really no way to generalize from the results. See in particular the details on what made the agent stumble, it was things like "cargo repeatedly gets invoked with the wrong arguments" - nothing to do with functional or static, just ecosystem idiosyncrasies and trivial differences.
Well, it did show you probably shouldn't use assembly, but that's about all it showed very strongly.
And of course it also showed very strongly that you should not base any choice-of-language decisions on single studies.
Any good LLM service (not just coding-focused ones) will write and run ad hoc code without being asked if your prompt involves lots of data. Gemini and Claude tend to pick Python with maybe some SQLite. Some of that must be due to portability alone, but it also means they'll make sure the model and tooling are good at those.
Correctness here is not about just writing bug free code, but the code that actually gets stuff done with correct results. JS might be better for this, at least up to some scale.
Is there a relationship between how good a programming language is for coding agents and how popular it is among humans? If so, wouldn't Python be the best language for agents, since it's is the most popular (and hence has the most context available for models)?
Pick something slightly esoteric (eg Haskell) and the quality of public code is very high, because you only have enthusiasts writing it. Choose something taught in schools (Python) and you are going to find 10,000 traveling salesmen homework problems and Django todo applications.
Not sure how you thread the needle on the quality vs quantity dynamic.
there is a relationship there, but there is also a relationship to the safety of the language and the guard rails in place.
it's a lot harder to experience an agent telling you with certainty that something incomplete is totally finished if there is a comprehensive test suite, a hard failing compiler, a strict type system, etc.
LLMs like to produce a lot of JS and python that silently fails in a graceful way -- why is that? because those languages support that kind of a failure.
when using something like go/rust the LLMs are more likely to re-iterate rather than declaring a victory when they get a strict compiler barking in their face, refusing to output.
There is definitely a relationship. But I personally believe that once the training corpus reaches a certain scale, the returns exhibit diminishing marginal effects, to the point that multiplying the data volume cannot surpass something essential inherent in language design. (Asked an LLM to help me with the translation, so forgive my expression)
Great post. If it wasn't clear by now, considering a language's token efficiency is almost certainly incorrect, since it's only a local optima for input/output of the code.
Most session tokens are spent elsewhere, so an LLM that handles a token-efficient language more poorly can be worse overall.
If anyone remembers TOON from a few months ago, it was an attempt to replace JSON with a more token-efficient representation. TOON was much more compact, but when researchers examined whole-session effects, it was a wash, because harnesses wasted more tokens than it saved dealing with it. (TBF, it's possible TOON use has gotten better if later models have it in their data set.)
I love this take because I had exactly the opposite idea. I thought the combo of remarkably simple text (not even wrapped) with incredible, full width visualizations was chef's kiss. I really like the balance there personally, but I hear you. Does your browser have Reader Mode or something like that? I don't use those tools personally, but I believe they will recast the text parts into something that renders optimally for reading (ideal font size, number of characters per line, etc....).
Is doing something for yourself really working for free? That's an interesting take. But I can understand why you don't want this for yourself, so enjoy the page in all its splendour as it is already!
A closer analogy is going to a gas station where a microwave is offered to heat up any food you purchased. If you don't want to heat up the food, cool. If you expect the food to come hot you're in the wrong place.
Except in this case it's a gas station that only exists for the benefit of its owners and there isn't any food for sale. The owners have graciously said you could still use the microwave if you'd like, though.
CSS is explicitly designed for you to apply your own user stylesheet. That is exactly what it envisions you doing. If you don't like web technologies you might want to question what you are doing on the web. However, the web also encourages sharing, so no, theoretically once one user has created a user stylesheet they would share it with others so there would be no need for everyone to create their own, unless they had alternative tastes.
I mean, if it really bothers you you could fairly trivially apply picocss or whatever to it using a user stylesheet. That is so little effort that calling it working for free would be disingenuous to say the least.
For people in the future who want to apply the centered viewport classless version of Pico CSS, just apply the following in Firefox's Style Editor (F12 to open DevTools, then click on Style Editor; click on the + sign to add a new style):
Browser vendors have really dropped the ball in supporting CSS, which is no doubt how we get comments like the above. Firefox is likely the least-worst offender, but as you point out still needlessly complicated for what CSS considers to be a foundational feature.
First, How fast is the Zstd decoder in python at runtime? If rust and python are essentially the same cost, then chose rust.
Second, I am surprised that python scored slightly better than rust. My own experience is that, when programming python, Claude would spend so much more time dealing with the code not working at runtime, while for any given rust problem, rust would likely fail at compile time, iterating faster and taking less tokens. Some tasks in python it just completely failed at, writing awful garbage. I suspect that is because there is much more awful garbage written in python. (I was trying to write an AI harness. Python seemed like the obvious choice. It was decidedly not).
But in this article, python took slightly less time and tokens than rust for both experiments.
I asked Claude: could you write a decoder, from memory, in python (dont do it, just tell me if you could)
> Honestly: I could write something that's structurally right and would not decode a real .zst file.
> The control flow I'm confident about from memory — frame/block parsing, the literals section dispatch, Huffman weight reconstruction, the backward bitstream reader, the interleaved three-state FSE loop, sequence execution with the repeat-offset rules and the overlapping-copy hazard. I'd expect to get that architecture right, and it would be readable.
So perhaps asking it to do things that are in its memory is not a good benchmark. It was trained with the C "educational decoder, and every third-party port in Rust, Go, Java, JS." and offered a working link [1] to the former.
Why would this be the case when the text that produces binaries (code) is usually both more token efficient and vastly more effectively organized for modification/extension?
Unless by bits you just mean text in general, or any data since it’s all bits, in which case what you’re saying is trivially already true.
It seems like you’re saying that long term LLMs will output pure machine code as the most effective way to use them.
We studied this question pretty systematically in the MirrorCode paper [1], comparing Python, C, Rust, Go, OCaml, and Ada across 19 very long-horizon tasks, for Claude Opus 4.7 and GPT-5.5.
> In our results, there was little sign of inter-language differences in solve rates, for any model (Figure 5b). This suggests that AI models have learned generalized programming skills, rather than pattern-matching syntax. This does not mean that implementation language is irrelevant. Conditional on solving a target, we found a small effect on token usage: successful Python solutions tended to use fewer tokens than average, while successful Ada solutions tended to use more (Appendix C). We consider these to be small differences, given that these six programming languages vary widely in how concise they are, and in how much functionality is provided by their standard library (recall that agents cannot download dependencies in MirrorCode, they must solve the task using only the standard library).
In Appendix C, Ada tended to use only about 25% more tokens than the average language. Ada is a language used mainly in safety-critical aerospace and defense systems, which has ~200x less pre-training data available than C or Python.
We're also comparing more recent language models (on just Go vs Ada, for cost reasons), on our leaderboard [2].
[1] https://arxiv.org/pdf/2606.30182
[2] https://epoch.ai/MirrorCode#leaderboard
Reading your quote really gets me wondering who the people making those kinds of analysis are...
It's like they haven't maintained any actual software, because the criteria they choose is... Completely irrelevant?
The things that matter are tooling, orchestration and ecosystem - as well as how the LLM will actually implement the solution for a task
LLMs constantly do idiotic things. If you have good libraries to utilize, the likelihood of the solution actually working goes up because they no longer need to implement the hard part.
If you have orchestration for dependency injection, code generation, meta analysis etc
Tooling like the way otel tracing is integrated, openapi generation etc is also invaluable because every time the LLM does something the likelihood of it being hallucinated/wrong increases etc
You'd need to implement a nontrivial project in different languages, then add nontrivial features across them and only then start by rating eg correctness and incident occurrence after the final output
But token use on a one shot? Completely irrelevant as far as I see it.
I'm not sure I trust a source that says "just 70 tokens average, nearly half of Clojure (109 tokens)".
There's no reason to add the phrase "nearly half of", and there's especially no reason to add it when it's significantly far away from half.
But on the main topic, I still feel that Go is an excellent choice for LLMs. There is pretty much just one way of doing most things, and the available training data is pretty consistent. This is very different from Python, where training data is polluted (I presume) with tons of code written by non-software engineers and demonstrating many different ways of doing the same thing.
Also a big plus for Go is the tooling. Fast compiles and good linting shortens the iteration cycle time, resulting in less need for me to tell the LLM to correct mistakes.
For some reason, most LLMs I've used default to wanting to write Python. I have to repeatedly teach them to use Go unless there is a very compelling reason to choose otherwise.
I would personally rather see and use Clojure, but I don't feel its ecosystem would provide the same benefits as Go, including obviously the easy single binary distribution.
Go is absolutely one of the best programming languages for LLMs for the reason you say, and Python is just what LLMs like to use to write short throwaway scripts. Frontier LLMs are generally pretty good at most programming languages and can pick up new ones pretty quickly. Training data seems to mostly just increase the speed which they write code, for example, GPTs tend to write Rust and Python faster than other programming languages.
For actual output quality, the main deciding factor is simply how much tooling it is there for the LLMs to check their own work, as LLMs seemed to avoid using a lot of libraries in general. That's why C# is underrated due to the tooling strength of the .NET ecosystem, as long as you tell LLMs to avoid using reflections unless absolutely necessary.
C++ is also surprisingly good, but you pretty much have to tell the LLMs to treat it like Go and don't use any of the dangerous features for normal code.
I think a lot of people are sleeping on the advantages of "batteries included" ecosystems.
The need to select an appropriate 3rd party library represents an entire dimension of the search space that can be eliminated. Imagine having to make this choice multiple times per day when your competition is just mindlessly using System.* types. The fact that the .NET ecosystem is curated by one entity should not be underestimated.
Even when we do need to import 3rd party nugets, the models seem to follow this highly structured pattern. They scan the xml docs, and failing that they will build a throwaway console app to reflect over all the unique types and build a report. The fact that we can easily do this with a simple powershell command makes a big difference. How many other ecosystems can even consider doing this? Reflection is a superpower, not something to be avoided.
Reflection is good for prototyping and get something setup quickly, but if you build your architecture around it, not only do you lose access to NativeAOT, the code becomes very hard to debug, and if you code with LLM a lot, you either have to spend time trying to debug reflections or just rewrite it with source generation to begin with, which is at least honest about the metaprogramming there.
Reflection is just such a dangerous feature that looks like ordinary code, which is why it is something to be avoided, and having an LLM write/analyze the code for eliminates the need to use a lot of reflective code to begin with.
> C++ is also surprisingly good, but you pretty much have to tell the LLMs to treat it like Go and don't use any of the dangerous features for normal code.
For existing codebases I made the experience, that LLMs are very good at replicating their style.
At work most of our C++ codebases use a fairly consistent style and subset of C++ features and to my initial surprise specifying style conventions etc explicitly turned out to be mostly superfluous.
Of course, we also have some legacy projects originally, written in ANSI C, which only received a few changes in the last 15 years to compile with a C++ compiler. Here a style guide is helpful, bit I consider it more like a temporary instruction for refactoring.
If you want the opposite experience, try LLMs with an older Oracle Database. Even with the official Oracle SqlDeveloper MCP Server they struggle a lot.
However, I struggle as well and the error messages of Oracle are some of the worst I've ever experienced.
> Go is absolutely one of the best programming languages for LLMs for the reason you say, and Python is just what LLMs like to use to write short throwaway scripts.
And yet this article has pretty strong empirical data to show that your intuition here is incorrect. You should back up your statement with something more than vibes.
Best to read the comments before replying. The article is about correctness, while the parent is talking about output quality.
What is output quality without correctness? That seems like a distinction without a difference.
Is the claim that LLMs produce Go code that is superficially nice looking but in fact fail to solve the stated problem? Because that's an anti-Go position I'd say.
Correctness is binary, while quality is not.
Correctness is a suitable property to act as a multiplier in your formula, where incorrect is 0 and correct is 1, but you also need other facets to find a quality gradient.
I like Go with agents too but:
> This is very different from Python, where training data is polluted (I presume) with tons of code written by non-software engineers and demonstrating many different ways of doing the same thing.
Counter-example: agents with Django-related stuff. Excellent output.
I think you will find that is a supporting example. Django pushes for a particular style and structure, which is a similar property found in the Go community.
LLMs seem to fall apart where human written projects of the same nature had no particular way about them. It is especially apparent when treading into waters where beginners are found. Like the earlier comment suggests, this is presumably because the LLMs struggle to find any kind of pattern to latch onto. Django offers a pattern, but one not shared by rest of the Python ecosystem. Whereas virtually all Go codebases look the same.
Agree that go is the best due to its main design goal: A language that's simple for any programmer fitting that definition https://news.ycombinator.com/item?id=30688969.
> "They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt."
This makes it a great language not just for young Googlers programmers, but also for LLM Agents!
IMO, the next big language will be similar philosophy, but without garbage collection. (is Zig the closest to filling that niche?)
> is Zig the closest to filling that niche?
Given what you said about Go, presumably that is Solod (https://solod.dev)
> This is very different from Python, where training data is polluted (I presume) with tons of code written by non-software engineers and demonstrating many different ways of doing the same thing.
Python's philosophy is there is one way to do it, as opposed to Perl's TIMTOWTDI.
Your statement also assumes that 'software engineers' write the best code, and from my experience, this is definitely not true
I believe the training data should simply be limited to only code written by someone like Fabrice Ballard, or whoever you think writes the best code.
But that's the trouble, Python has the slogan about only one way, but it's really not true in practice. Or maybe there's the one way that "should" be done, and then the half dozen other ways you'll encounter in the wild, as gp alluded.
Python is the language where in practice this is most untrue - maybe outside C++.
As an example, there are dicts, tuples, classes, NamedTuples, dataclasses, attr.s, Pydantic, and pretty much all of those solve similar problem (hold my data) but have slightly different properties and use cases.
Aren't LLMs a way to somehow extract the one way it should be done (or to be more precise, the more common way), over the half other ways? That correct way might be more difficult to extract from another languages that encourage multiple valid ways.
Also, if you trust the benchmarks, it seems that Python is, at the very least, decent enough for LLMs. There seems to be "no trouble" in practice, unless you show us better proof than "I feel like it must be bad for this and that".
You can rather easily ship Clojure apps as single binaries, eg with this: https://github.com/avelino/jbundle
Contrary to what most comments seem to indicate, my takeaway from this is that it doesn't really matter all that much for the agents what language you pick. If humans are still to be involved in the process at some point, then its imperative that the language can be read by them, so the preference or skills of the developer(s) are of primary concern, not the agent.
Ive been amazed at how well LLMs are at writing Gleam[1] and Lustre[2]. Compared to a mainstream language, there is basically zero gleam code in the training data.
I have no evidence to back this up, but I suspect that languages that are good for humans[3] will be good for LLMs. Compiled, strongly typed, statically typed, immutable, pure functions, pattern matched, memory safe, etc.
[1] https://gleam.run [2] https://lustre.hexdocs.pm [3] Yes I realize that languages features that are "good for humans" is a hotly debated topic. That's just my personal list for what I like in a language.
I used to hold this opinion but since changing to Rust on the server and Typescript on the client, I can confidently tell you agents are so much better at Rust, especially at producing idiomatic code, than they are at Gleam.
There are reasons to love Gleam and Lustre (I like Gleam a lot), but LLMs just aren't one of them. I made the switch to Rust around May this year. Also the community is super anti-AI, arguably with good reason (how it impacts open source), and I'd recommend keeping your AI code to yourself.
I’ve been developing a language for a few years, and even with incomplete semantics and a simple one page example LLMs don’t have much trouble writing it.
I think the language/syntax has an impact, but the tooling around it will be most important for LLMs, in the same way it is for humans.
Gleam has been stable for over two years, so maybe it's been long enough that LLMs have internalised the documentation.
Given it's a language that doesn't really contain any groundbreaking ideas[1] (the closest is 'use' IMO), it's possible LLMs can reuse patterns from other functional language.
[1] This isn't criticism. I love how Gleam turned out.
That's not a take I was expecting to find here. I've found most LLMs absolutely dreadful when it comes to Gleam, to the point that I most often disable even inline autocomplete when working in Gleam codebases.
Too often I find them getting pulled into larger ruts in the training data and trying to insert language features that don't exist (ifs, loops, and syntactic constructs) from more popular languages like TypeScript and Rust. Do you not experience other languages getting partially substituted in when you have LLMs write Gleam?
I suspect it depends a lot on the llm/harness being used. But when I use Opus/cc or sol/codex, at the end of the turn everything compiles, passes tests, and passes lint. I never even look at code that can't compile. Maybe the LLM is generating weird stuff in-between, but I don't see it.
What you're describing feels like my experience back in 2024/25. Back then I was using a llm auto complete or the chat interface, and I would get weird stuff all the time. (not just gleam but any language).
For an even more niche language, Roc basically became usable in the new syntax about two months ago (still has compiler crashes, there's a good reason it hasn't had a real release) but Opus writes it just fine after a couple corrections to handle the language's quirks.
I don't care so much about token efficiency and cost, within reason. I care about whether the quality of the code is maintainable over time and through many iterations. My gut feeling is that very strict languages with good types, a standardized style, and very strong static analysis tools, is what helps make that happen. Of course it also has to be well-represented in the training data.
That leaves Go, Rust, Python with type annotations, and Typescript. And, I choose them in roughly that order unless there's a reason to choose otherwise. Rapid iterations on scripty tasks get Python. Most CLI, system services, and web apps are Go. Desktop apps and games are Rust. Typescript if I don't have a choice (i.e. it runs in a browser).
I wonder if there's correlations between tasks and languages, e.g. maybe R is better for bioinformatics tasks, python for webdev, C for CLIs, etc. I'd expect to see it because some languages are used more often in some tasks than others, but on the other hand LLMs can learn across languages and it's not clear if task-language use patterns are just historical or if the language is genuinely better at the task.
It's not clear to me how useful of a signal replicating existing pieces of well-known software is for this kind of evaluation, given what we know about how effectively LLMs can retrieve data from their training corpus and style-transfer it across different settings (programming languages here). That would explain their convergence in ability across different languages on the tasks in this post. I'd be far more interested in people's real-world experiences.
I tried writing an AI harness in Python. Seemed the obvious way to go. Tons of libraries. Libraries for talking to model APIs. Libraries for context and conversation management. Libraries for talking to MCPs. It is the language for LLMs!
It was a shit show and just couldn't write anything that would not crash. Super confident it had done a good job. Full of random bugs. A UI needs interactivity, interruption, handling exceptions. It produced some of the worst code I've ever seen. And looking at the libraries' code: also some of the worst code I've ever seen.
I switched to rust + tauri. In about three person weeks of work I have UI with forking conversations, tool use with built in grepping, tons of quality tools. It's more productive (for me) than Claude Code (CLI or desktop).
I was recently working on an AI harness too, but I wasn't using AI to code it. It's really easy and requires little code. I wasn't even using langchain - that would require even less code.
UI is harder for sure, but it's not that bad. You need to think though where to catch which exceptions.
LLMs might opt for langchain which has had multiple breaking changes after the knowledge cutoff, making it hard for the LLM to work with it. This is probably going to lead to the LLM having to make many changes to it's code, making it messy and leading to further code being less maintainable.
Yeah, I've had similar experiences, also starting out with dynamic languages and migrating to Rust. If the LLM will write a lot of the code for me, why not choose something (1) super fast, and (2) which has types I can use to understand and specify the code I want without having to read all the output?
I've been trying out Lean for related reasons, to good effect. It's really interesting there since it can crank out proofs that would've been completely infeasible for a dedicated team of PhDs before, whereas I haven't seen any LLM projects written in Python that I couldn't have slung out in a few months myself. I personally think it's a lot more interesting to focus on the new things you can now do with LLMs that weren't possible before, as opposed to doing the same old stuff at moderately higher velocity.
I can highly recommend TypeScript/JavaScript with AI SDK:
- https://ai-sdk.dev/
I have used it, and I can say it is really nicely written.
Matt Pocock has created a good tutorial on it:
- https://www.youtube.com/watch?v=mojZpktAiYQ
In my experience, Dart/Flutter has been so much better than React. Go has been really good for the backend. Basically if the framework/language gives you structure and one way to do things, agents tend to create less mess with less guardrails from you.
For me c wins here. It is compact, there are all language parts one needs and available and it well fitted to transport knowledge without much syntax hussle
Related:
Which programming languages are most token-efficient? - https://news.ycombinator.com/item?id=46582728 - Jan 2026 (91 comments)
What's optimal for LLMs and for people is probably not going to be the same. People are a bit lazy.
Coding agents do much more than generating code though. Much of what they do relates to validating that what was generated is a valid solution. That includes everything from type checking, running tests, static code analysis, linting, running code in a headless browser, etc. The more tools agents have at their disposal, the better the feedback loop gets. But of course some of these tools are costly to run.
Statically compiled languages have a head start here as they simply exclude entire categories of bugs that a dynamically typed language might have. And with things like type inference, their token overhead can be pretty minimal. Modern languages like Kotlin or Swift are pretty compact and don't really add a lot of bloat relative to say typescript/javascript. Go is a bit more verbose but tends to work well. Rust seems pretty popular with LLM users as well. The main challenge with languages like this is the performance hit you take running their build tools. Doing that a lot slows you down and it burns a lot of tokens as well.
Zstd gets rather easier from dotnet 11, it becomes a near one-limer since it's getting added into System.IO.comoression.
I know this because my agent already knew this the other day when I was evaluating compression, but that's because it has access to search.
That's a key part of what makes agents good coders too, mine is often looking up and downloading the source for how libraries are implemented.
It seems unnatural to air-gap them for evaluation.
I guess they didn't want them just finding an existing library to copy, but it's not very "real-world" to deny the ability to search quickly.
That said, the best language is still just the one you know. No amount of token saving is worth getting a bunch of code back you can't easily understand and review.
Cool line of questioning, but one piece of information is pivotal and critically not-yet-included: equivalent accomplishments in each language. For example, if I want to write standard things: web server, memoized fibonnaci, recipe search engine, what's the length-and-density of these outputs for each language? I think that would add in some ~normalization.
Strongly agree- this is how I “evaluated” languages pre-agents. though I suspect this would bias results in favor of whatever has best signal to noise for boilerplate from stackoverflow/reddit , rather than what LLM’s “””reason””” best with. (Presuming those aren’t quite one-and-the-same)
This is a great discussion: I wonder though if we are asking the right question. Yes, absolutely language choice can play a large role in the efficiency of coding agents. The point about Rust is right: static typing provides a fast verification loop at compile time. I would argue though that the way the codebase is composed could actually generalize the concept of "easy verifiability" past the actual coding language.
For instance, if an application can be broken down into components that have a verifiable contract in how they are to be used, then an LLM can load only the relevant modules into its context and fully understand how to use them and fix them if needed. It is also easier for the LLM to verify the functionality of a component rather than the entire system.
Additionally, in an application composed of functioning components, issues are more likely to occur at the boundaries between them, which the LLM can focus on rather than having to always consider the entire application that it most likely can't load fully into its context.
A well designed componentized Python application will likely be far more efficient for modification by an LLM than a large Rust monolith.
C and C++ do well because there is most literature and code out there to help them reason about it. C is helpful because it has little hidden runtime for them to trip over.
that being said, those languages obviously have limits in applicability looking at the entire spectrum of software. JS, python and others still have useful domains.
i dont think newer languages as rust are better for LLMs as they might be for new programmers. for new programmers they offer extra features but for an LLM this is added potential to make mistakes. Also a lot of newer languages are less stable so you can realise their current implementations might not be fully trained on by the models or even be after their cutoff date..
One thing worth noting is that syntactic density doesn't necessarily mean cheaper because because symbols don't chunk/tokenize as well as plain English
What I see from results like this is that the delta between languages is small enough now that it's hard to justify not not using something like Rust for the performance and correctness benefits if you're using LLMs and it fits the domain
How good or bad are LLMs on languages that have evolved over the years and aren't popular enough to get hand tuned?
Asking because for non programming, if you use them instead of a wiki for a topic that has had yearly changes for like 10 years they get confused and mix releases like crazy.
I discovered last week that Fable 5 can write perfect xTensa LX7 assembler code without tools or references. Mind blown.
But, when working on a creative graphics task, the results were best in Lua, middling in integer-only C, and underwhelming in ASM in terms of creative depth.
Python has a less known advantage because it had no curly braces, so LLMs can focus its attention to logic instead of syntax.
https://blog.est.im/2026/stdin-11
It's unclear that this is an advantage, certainly not in terms of "logic vs syntax".
First of all programs written in curly-brace languages still also have indentation to indicate statement grouping / blocks / scope, even if it's not required, so for a correct program (and that's not deliberately obfuscated), and one that's in the process of being written by an LLM, any advantage there disappears. Furthermore, having both indentation and explicit block markers provides redundancy which could be a significant advantage for an LLM (it being a probabilistic text / program generator). And for an incorrect program that redundancy is a big advantage for the LLM because it should be very easy for it to notice a mismatch of indentation and braces.
The only downside would be a very slightly higher token cost for the redundancy. I realize that Python comes out on or near the top in most of the comparisons in the linked article, but I doubt that's the reason.
The thing with LLM is they don't automatically pair parenesis/curly braces like we do with editors/IDEs.
The closing } ) ] token has to be generated to match exactly the beginning { ( [ many lines before. You can challenge yourself writing Lisp by hand without cursor moving backwards, and try close correctly by counting ))))))) you'd have a big headache.
A long, nested sub-routine with many () {} will cost LLM's context and makes it underperform, because the attention head have to track the state. On the other hand the indentation level can be infered as a single token[1] and saves reasoning effort.
Note these discussion is about "code generation", not parsing.
1: https://platform.openai.com/tokenizer Try input many spaces.
Cool seeing Guards of Atlantis 2 here.
One thing that often happens with board games is rule issues in translations. Specifics that are clear in one language get lost in translation. Wolff Designa is out of Latvia. So not surprised there are some hard to interpret rules.
It’s interesting that LLMs struggle with the board game rules like we do. I think game designers should get the llm to teach them from their rulebook. If an LLM can’t understand the rules good chance people will also be confused.
> Dynamically typed languages generally have a lower LLM token cost than traditional statically typed languages because omitting explicit type declarations makes the code more compact.
If this was true, the programming languages that are very much on the left side of
> https://danuker.go.ro/programming-languages.html#non-math-ma...
> https://danuker.go.ro/programming-languages.html#overall-map
should be very ideal for LLMs, in particular if they are dynamically typed.
What I can tell you is: I experimented with AI prompts for generating Wolfram (Mathematica) code using some LLMs, and I can tell you that the results were very disappointing: in my experience LLMs have difficulties with programming languages that are
- very concise, and
- for which there is less code publicly available.
Wolfram (Mathematica) is a good example of such a programming language.
I’ve actually found Sol delivers great results with Odin, despite there not being much Odin code available. I think it is able to work well with it because:
- It is a rather simple language - It has a lot of very useful libraries already built in.
With just a single main.odin file you can do a heck of a lot stuff, which LLMs seem to like.
It’s interesting I’ve been surprised by how well Claude sonnet can write code in a language I’m developing that probably has no code in the training set. It seems like anything with syntax like python/ruby/elixir is pretty LLM friendly, and layering on a HM type system seems to help catch most errors.
> I think it is able to work well with it because:
> - It is a rather simple language - It has a lot of very useful libraries already built in.
> With just a single main.odin file you can do a heck of a lot stuff, which LLMs seem to like.
Also Wolfram/Mathematica has an insane amount of useful libraries already built in (there even exists the saying "Python is 'batteries included', Wolfram is 'spaceship included'"), and also there in a single file you can do a heck of a lot stuff.
On the other hand:
- LLMs tend to hallucinate non-existing function when you ask an LLM to code something in Wolfram that is not commonly done (concerning this point, nevertheless keep in mind that Wolfram is often used for "one-of-a-kind programs", i.e. for writing very specialized programs that have possibly never been done before).
- Wolfram code tends to be quite dense.
- If there is a small mistake in Wolfram code, the code typically simply won't work.
Is there a way in Wolfram to check whether all function names exist ? And than give it as feedback to the llm?
> Is there a way in Wolfram to check whether all function names exist ?
There is a way to check whether a symbol has been defined:
See https://reference.wolfram.com/language/ref/ValueQ.htmlReplace FunctionName by the function name that you want to check.
> omitting explicit type declarations makes the code more compact.
I guess this ignores languages with type inference? Hindley-Milner and others
Training data is a factor too
What is the best language for the user of the LLM?
What is the best language to have high quality correctness oracles so that the user doesn't have to babysit the LLM and do lots of manual testing?
JS is the best tradeoff between succinct and easy to understand. Python is next but has some rough edges that they avoided in JS.
JS gives you wrong answers silently when thigns go sideways since for the original browser use case, they didn't want scripts to ever stop execution. It's the opposite of Python in this respect.
You are going to have to give more support for those assertions. I write Python every day, and never would I call it a good candidate for the clankers. Pretty much any dynamic language would be ruled out, as there is too much implicit logic which makes it harder to understand what is happening.
Python with a strict linter and type checker (eg ruff with the right lints on and ty with its stricter settings, or strict pyright if performance isn't too bad) works very well. Most of Python strengths (concise, large ecosystem, well represented in the LLM training data) while having good static analysis.
You don't need that, gets in the way more than it helps. Even Typescript isn't really needed, but at least it's decent devex unlike the Python typing stuff. What really helps is testing.
I trust the type system more than vibe tests.
I try to capture as much as possible in types/schemas/constraints (then lint rules) and then only as a last resort write tests. And as few and complementary as possible. And I want a functional core with unit tests and imperative shell and not a bunch of complex mocks. ¯\_(ツ)_/¯
What's better for this, Go? That's the least verbose static one, and it's still a lot more verbose without helping you understand any better what it's doing. It's just faster. That's the real benefit of static types.
i dont see enough love for Ruby. I've been using it since last year and it feels like php's more robust brother
I'd like to see the results for Ada on these same measures. On the theory that the Ada type system covers more classes of errors than other languages, and so AI can self correct better.
Unfortunately for static type weenies like me (and you, presumably), types don't seem to matter at all, or Python and Javascript wouldn't be on top. There's no reason to believe that Ada's type system is so unique that it alone can help AI self-correct, and Rust, Haskell, ML, Typescript, etc. can't.
Well the reason I'm interested in Ada is because I saw a study that showed AI did worse at functional programming. So that might explain the problems with Haskell et al. But Ada has a strong type system while still having procedural code. So it would be an interesting comparison with Haskell etc. if the problem was that Haskell is functional or if the problem was that these are not so popular.
The study didn't really show that. What it did show, is that in this particular study there was no strong effect either way and there is really no way to generalize from the results. See in particular the details on what made the agent stumble, it was things like "cargo repeatedly gets invoked with the wrong arguments" - nothing to do with functional or static, just ecosystem idiosyncrasies and trivial differences.
Well, it did show you probably shouldn't use assembly, but that's about all it showed very strongly.
And of course it also showed very strongly that you should not base any choice-of-language decisions on single studies.
You're in luck! https://news.ycombinator.com/item?id=49254706
Any good LLM service (not just coding-focused ones) will write and run ad hoc code without being asked if your prompt involves lots of data. Gemini and Claude tend to pick Python with maybe some SQLite. Some of that must be due to portability alone, but it also means they'll make sure the model and tooling are good at those.
So Javascript beats typescript in correctness???
Correctness here is not about just writing bug free code, but the code that actually gets stuff done with correct results. JS might be better for this, at least up to some scale.
Is there a relationship between how good a programming language is for coding agents and how popular it is among humans? If so, wouldn't Python be the best language for agents, since it's is the most popular (and hence has the most context available for models)?
Pick something slightly esoteric (eg Haskell) and the quality of public code is very high, because you only have enthusiasts writing it. Choose something taught in schools (Python) and you are going to find 10,000 traveling salesmen homework problems and Django todo applications.
Not sure how you thread the needle on the quality vs quantity dynamic.
>Pick something slightly esoteric (eg Haskell) and the quality of public code is very high, because you only have enthusiasts writing it.
that and the language supports (enforces) good decision making; static typing w/ inference and a functional style as a first class concept.
which then rolls into the same result : higher quality code available.
Quantity, it seems is a quality in itself.
As much as I love Python, JavaScript (including TypeScript) is probably more popular.
That and JS code is more readily available in the source of tons of webpages, not hidden away in some backend
Wouldn't most frontend JS in Web page sources be minified?
The logic is still there, it's not meant as obfuscation. Also plenty of sites don't minify cause that involves a whole toolchain.
there is a relationship there, but there is also a relationship to the safety of the language and the guard rails in place.
it's a lot harder to experience an agent telling you with certainty that something incomplete is totally finished if there is a comprehensive test suite, a hard failing compiler, a strict type system, etc.
LLMs like to produce a lot of JS and python that silently fails in a graceful way -- why is that? because those languages support that kind of a failure.
when using something like go/rust the LLMs are more likely to re-iterate rather than declaring a victory when they get a strict compiler barking in their face, refusing to output.
There is definitely a relationship. But I personally believe that once the training corpus reaches a certain scale, the returns exhibit diminishing marginal effects, to the point that multiplying the data volume cannot surpass something essential inherent in language design. (Asked an LLM to help me with the translation, so forgive my expression)
BASIC.
Great post. If it wasn't clear by now, considering a language's token efficiency is almost certainly incorrect, since it's only a local optima for input/output of the code.
Most session tokens are spent elsewhere, so an LLM that handles a token-efficient language more poorly can be worse overall.
If anyone remembers TOON from a few months ago, it was an attempt to replace JSON with a more token-efficient representation. TOON was much more compact, but when researchers examined whole-session effects, it was a wash, because harnesses wasted more tokens than it saved dealing with it. (TBF, it's possible TOON use has gotten better if later models have it in their data set.)
I love Dan's writing. I really do. But I don't understand why he doesn't have some basic styling on his blog so that it's easier to read.
Enabling 'reader mode' in supporting browsers usually takes care of this.
I love this take because I had exactly the opposite idea. I thought the combo of remarkably simple text (not even wrapped) with incredible, full width visualizations was chef's kiss. I really like the balance there personally, but I hear you. Does your browser have Reader Mode or something like that? I don't use those tools personally, but I believe they will recast the text parts into something that renders optimally for reading (ideal font size, number of characters per line, etc....).
It may be an artistic/engineering choice to demonstrate what minimizing bloat to an extreme degree looks like.
https://danluu.com/web-bloat/
body { margin: 5%; }
That's all it needs, responsive enough for all devices. He can keep his styleless design but margin is always needed.
Users being able to supply their own stylesheet is a core tenant of CSS. Go nuts and make it look however your heart desires!
Supply my own stylesheet? No thank you, I'm not here to do work for free.
Is doing something for yourself really working for free? That's an interesting take. But I can understand why you don't want this for yourself, so enjoy the page in all its splendour as it is already!
> Go to restaurant
> Order food
> Food comes out as raw, unprepared ingredients
> Complain to chef
> Tells me to go cook it myself
> wtf, I'm not here to do work for free
> "Is doing something for yourself really working for free?"
A closer analogy is going to a gas station where a microwave is offered to heat up any food you purchased. If you don't want to heat up the food, cool. If you expect the food to come hot you're in the wrong place.
Except in this case it's a gas station that only exists for the benefit of its owners and there isn't any food for sale. The owners have graciously said you could still use the microwave if you'd like, though.
So every person who reads Dan's blog and finds the layout too dense, they should write and maintain a stylesheet for his site?
And every person globally should do this as well for any other website that doesn't have a good default reading experience?
If most readers of danluu don’t find that, then yes!
CSS is explicitly designed for you to apply your own user stylesheet. That is exactly what it envisions you doing. If you don't like web technologies you might want to question what you are doing on the web. However, the web also encourages sharing, so no, theoretically once one user has created a user stylesheet they would share it with others so there would be no need for everyone to create their own, unless they had alternative tastes.
Actions speak louder than words.
I mean, if it really bothers you you could fairly trivially apply picocss or whatever to it using a user stylesheet. That is so little effort that calling it working for free would be disingenuous to say the least.
For people in the future who want to apply the centered viewport classless version of Pico CSS, just apply the following in Firefox's Style Editor (F12 to open DevTools, then click on Style Editor; click on the + sign to add a new style):
I honestly did not know that until I took about 5-10 minutes looking up how to apply arbitrary styles in Firefox.Browser vendors have really dropped the ball in supporting CSS, which is no doubt how we get comments like the above. Firefox is likely the least-worst offender, but as you point out still needlessly complicated for what CSS considers to be a foundational feature.
Multiple people, me being the third or fourth, are not feeling the default layout and you all read that as a signal it's working as intended?
No, just that it’s easy to change for those that don’t like it. Clearly some people do like it (including Dan, presumably).
reader mode helps.
First, How fast is the Zstd decoder in python at runtime? If rust and python are essentially the same cost, then chose rust.
Second, I am surprised that python scored slightly better than rust. My own experience is that, when programming python, Claude would spend so much more time dealing with the code not working at runtime, while for any given rust problem, rust would likely fail at compile time, iterating faster and taking less tokens. Some tasks in python it just completely failed at, writing awful garbage. I suspect that is because there is much more awful garbage written in python. (I was trying to write an AI harness. Python seemed like the obvious choice. It was decidedly not).
But in this article, python took slightly less time and tokens than rust for both experiments.
I asked Claude: could you write a decoder, from memory, in python (dont do it, just tell me if you could)
> Honestly: I could write something that's structurally right and would not decode a real .zst file.
> The control flow I'm confident about from memory — frame/block parsing, the literals section dispatch, Huffman weight reconstruction, the backward bitstream reader, the interleaved three-state FSE loop, sequence execution with the repeat-offset rules and the overlapping-copy hazard. I'd expect to get that architecture right, and it would be readable.
So perhaps asking it to do things that are in its memory is not a good benchmark. It was trained with the C "educational decoder, and every third-party port in Rust, Go, Java, JS." and offered a working link [1] to the former.
I've long suspected that LLMs will just output pure bits eventually
Or just replace all of the apps? Why would a user use apps if everything (eventually) can be achieved with an AI?
Why would this be the case when the text that produces binaries (code) is usually both more token efficient and vastly more effectively organized for modification/extension?
Unless by bits you just mean text in general, or any data since it’s all bits, in which case what you’re saying is trivially already true.
It seems like you’re saying that long term LLMs will output pure machine code as the most effective way to use them.
well they can natively converse in base64
are you implying that text is impure bits?