What I feel a bit annoyed by, and what I feel obviously LLM-run ablations like this fail to capture, is any kind of reflection around previous research or any kind of proof that this is the best you can do. You don't know, you pulled the lever and you got something, is the best? Can you do better? What is the constraint?
As an individual researcher, you do not have 22M$ to run a massive brute-force search for your problem. You are constrained to your little subscription and you will barely dip your toe in the sea of possible solutions to a problem. So letting Claude run an autoresearch loop on your problem and then having it summarize it for you brings 0 value because you dont know what the downsides and trade-offs of LRU caches were, and how you would possible solve it.
I don't think the problem is a lack of previous research, the problem is it looks like the target metrics were selected by an LLM, and it's unclear what the LLM was told to optimize or if it was just told to try and make a better KV-cache.
What I've noticed with Claude is that regardless of how I prompt, it will find a few metrics to optimize. Often the metrics it chooses to optimize have zero relation to the actual metrics I want to optimize, which are ones that cannot be measured without more work than Claude can do in a single 1M token context window. It's really hard to stop Claude from optimizing whatever metrics it can find when the actual metrics I want to optimize are not computable.
Sometimes it's good enough to acknowledge a correct answer, and slowly build intuition towards it. Academic problem solving is rarely a straight path, anyway.
> What I feel a bit annoyed by, and what I feel obviously LLM-run ablations like this fail to capture, is any kind of reflection around previous research or any kind of proof that this is the best you can do.
At least the former is easy: Claude is very happy to do a prior art search and pretty good at it, too.
It'll just be telling you what the data it ingested claims. Not what is necessarily true. It's as subject to garbage in, garbage out as anything, and you don't know what it's actually trained on.
> It didn't work, and why it didn't work turned out to be more interesting than the policy would have been.
Spoken like a true Claude.
Snarking aside, I am glad that our AI agents make it cheap enough to do these experiments and publish these write-ups that people finally bother to publish null findings. Very useful!
The idea of publishing null findings is very attractive, but the problem here is that they didn't record what the actual problem they're trying to solve is. It seems a bit like someone prompted Claude to try and make tool calls more efficient and Claude responded with a detailed writeup on why LRU can't be beat for making a KV cache. Which might be interesting, if that were the question I had asked.
I always have a tinge of...perhaps 'sadness' when reading such articles because they are a lot like those large fishing nets that drag along the bottom of the ocean catching and destroying in quantities unneeded. From this list:
- Quietly/Delve/Tapestry/Landscape: they're all lovely words. I love(d) using them.
- "Serves as": I use this all the time too.
- "The result? Devastating". Perhaps it is a bit stilted, but "Outcome? Predictable." could be a nice turn of phrase.
- "It's worth noting": All academics everywhere use this in papers. See also: "Notably", "Essentially", "In general" and other such fillers that we reach for.
- "Think of it as": This is such a useful phrase to explain something. Analogy is how we learn after all.
- The fucking em dashes. I love them, and screw everyone who suggests one should stop using them. One big reason for using linux is the compose key, so the em dash is never far away.
- Same with the unicode decoration. Nah, screw you. I use these symbols all the time: °, ×, →, ⇒ etc. in my own notes. Again, compose key ftw.
Etc. (the list is endless)
I know what people will say, that everything is contextual, one has to study case-by-case etc. But I can't help feeling that someone is pointing a finger at me and accusing me before-the-fact. Just, AI has polluted the joy of writing.
I wouldn't fret too much: AI is still advancing so quickly that these specific tells will only last a short time. And then you are free to use em-dashes again, or at least no one will care.
I've seen Douglas Adams use almost exactly this in Dirk Gently's Holistic Detective Agency. A professor (IIRC) is talking to a former student. Quoting from memory
"Did you ever actually finish any papers?"
"No. But the reasons why not were always absolutely fascinating."
So, yeah, for me personally it's not an AI tell. I've seen it in the wild before AI.
LLMs write like some people cook, adding bespoke artisanal Belgian sea salt, truffle oil and weirdly specific cheese without any thought given to how the result will taste.
I suspect it’s more like Claude etc. optimizing for a tone that gets the green checkmark from RLHF reviewers, probably over and over in many contexts where style is otherwise an incidental concern.
If you're being given an A-B choice between cinnamon-flavored shit and lutfisk-flavored shit, you might convince the experimenter that cinnamon-flavored shit ranks highly in user preferences.
I would expect a preference for lutefisk flavored shit, the problem is then you then deduce from that that people prefer lutefisk flavor over cinnamon, rather than that you should stop feeding people shit (and also stop feeding them lutefisk.)
I'm guessing he means overall the kind of circling and concluding that Claude does, like "the actual truth is more interesting than it seems" or "the real smoking gun is not the x, it's the y that was under our noses all this time" type of writing.
"A block with resident children isn't evictable. So the baseline is LRU over radix leaves, which is what SGLang and vLLM actually implement. Beating naive flat LRU would be a strawman."
"The two numbers measure different things in different regimes, and I initially framed this as a contradiction. It isn't."
"That's not a result, that's a broken harness, and it's worth publishing because I expect it to be common"
While AI was clearly used to generate and structure a first draft, I say it has been extensively rewritten by the author. In my experience, AI tends to avoid the first person, instead writing from a passive observer or even using sentences without verbs. The length of the sentences is also more natural. AI had trouble with bind-words in an argument. It just states separate facts and assumes because they are stated in order you have a reasoning. I'm not seeing that here.
Of course, idioms are preserved, and not everything is rewritten.
I quite ok with this style, it isn't grating. I expect this will be a style we'll see more of in science.
LRU seems like the ideal strategy for most things LLM-related. Everything in this realm is about recency bias. I think it is a feature in this context, not a problem.
When I give an agent a piece of corrected information regarding a long running task, the last thing I want it to do is try and statistically compensate for the fact that it is new information. I want this new information to dominate the old information.
I work on inference at a neocloud, but opinions are my own .
The economics and thus tools you can throw at inference change at various scales .
As a “blunt” contrived example , on a gb300 the GPUs communicate super fast over nvlink, and the cards can offload kv cache to dram and then disk, “fast enough “ for these tool heavy agentic workloads.
Which come together to mean that at high enough scale and in the right scenario, we can work
with wild ttls on the kv cache and still comfortably hit SLAs and tokenomics.
I've been building latency-sensitive LLM systems for a while, and I've come to rely heavily on pre-fill-considerate mechanics like ping-pong overlapped async context construction. For interactive mechanics, the worst case, even if rare, is problematic.
Consideration of mutation rate (a sort of temporal Shannon-ish coding/ordering) lives in there (with some RoPE-friendly structuring). Note: That was a vacation project, not the day job, but similar principles apply even with larger models.
Based on the tables in part 4 it seems like some of the policies would have started overtaking LRU at larger KV-cache sizes.
And while the range of cache sizes chosen is reasonable based on the table about Moonshot's number at the start, the finding that all tested KV-cache sizes are so small that the 5-minute eviction never triggers should have been reason enough to reevaluate that choice
This is almost unreadable. The "papers" are never referenced anywhere, so the claims being refuted cannot be evaluated. The whole fact of the TTL doesn't seem relevant at all. There are many, many well-researched admission and eviction policies that this readme doesn't mention. I just don't get why we are reading this.
>Cross-request KV prefix caching is the largest practical lever in agentic LLM serving. It's why your coding agent's fiftieth turn costs a fraction of its first.
This is incorrect. If you are generating the 51th turn, then prefix caching makes the turns 1 to 50 cost a fraction of what they normally cost, to generate the 51th turn. Generating the 51th turn is more costly because it is not in the cache yet.
Unfortunately there is no way to steelman the statement, since quadratic scaling of attention means that the cost goes up even if you assume perfect caching of past inputs.
What the author actually means is something more mundane. The initial prompt is massive in a standard coding harness and this means prompt processing takes much longer than expected, creating the misleading impression that costs go down.
Edit: I noticed the word cross request too late. In that case the harness prompt is expected to be cached from other users, in which case the first message actually has a massive unfair advantage.
There's a growing literature arguing LRU is the wrong eviction policy for agentic LLM serving, because agent sessions idle and LRU can't distinguish a paused session from a dead one. I found the argument convincing and built a simulator to exploit it. Three separate mechanisms, all lost to plain radix-leaf LRU.
The reason turned out to be more useful than the policy. When I measured — policy-independently — where recompute actually comes from on 393 real Claude Code sessions, requests arriving after a gap longer than the 5-minute provider TTL account for 17.5% of it. Requests arriving within 10 seconds account for 33.1%. The dominant waste is tight tool loops whose 88k-token working sets exceed cache capacity, not sessions idling past a TTL. That's a capacity problem, and liveness prediction can't touch it.
it's just a bit disheartening to read Claude output for such a small comment like this.. it'd be great to read your own writings even if it's not as "perfect"
One side effect of extremely-accessible, high-quality English translation and, more importantly, English-grammar-and-idiosyncrasy-obeying AI, is that there will be more and more text that looks like this which comes from foreign, non-native-English countries.
Just FYI.
This will be especially true from non-English cultures where "avoiding shame" is high on the list of motivations.
Consider the upside, though: A much broader range of written perspectives written in high-quality, if slightly annoying, English.
I'm already seeing the benefit of this on X thanks to its autotranslation btw: I follow a few Chinese-language accounts now that I would have never been able to digest otherwise.
Interesting! I admit the AI-written text is rough to read, it could have used a pass or two from an actual human. E.g. "Publishing it unresolved rather than tuning until it matches." -- thanks for not lying, I guess?
Fun:
> In my first run, Belady — an offline oracle — lost to LRU. That's not a result, that's a broken harness, and it's worth publishing because I expect it to be common.
> The cause: inserting a long chain into a near-full cache lets a policy evict the very prefix it is currently building. LRU is accidentally immune because just-inserted blocks have the newest timestamp.
What I feel a bit annoyed by, and what I feel obviously LLM-run ablations like this fail to capture, is any kind of reflection around previous research or any kind of proof that this is the best you can do. You don't know, you pulled the lever and you got something, is the best? Can you do better? What is the constraint?
As an individual researcher, you do not have 22M$ to run a massive brute-force search for your problem. You are constrained to your little subscription and you will barely dip your toe in the sea of possible solutions to a problem. So letting Claude run an autoresearch loop on your problem and then having it summarize it for you brings 0 value because you dont know what the downsides and trade-offs of LRU caches were, and how you would possible solve it.
I don't think the problem is a lack of previous research, the problem is it looks like the target metrics were selected by an LLM, and it's unclear what the LLM was told to optimize or if it was just told to try and make a better KV-cache.
What I've noticed with Claude is that regardless of how I prompt, it will find a few metrics to optimize. Often the metrics it chooses to optimize have zero relation to the actual metrics I want to optimize, which are ones that cannot be measured without more work than Claude can do in a single 1M token context window. It's really hard to stop Claude from optimizing whatever metrics it can find when the actual metrics I want to optimize are not computable.
Sometimes it's good enough to acknowledge a correct answer, and slowly build intuition towards it. Academic problem solving is rarely a straight path, anyway.
> What I feel a bit annoyed by, and what I feel obviously LLM-run ablations like this fail to capture, is any kind of reflection around previous research or any kind of proof that this is the best you can do.
At least the former is easy: Claude is very happy to do a prior art search and pretty good at it, too.
you can just ask it to explain those things!
Sure, and it will explain it to you from the same embedding space that yielded the solution that LRU cache is unbeatable.
It'll just be telling you what the data it ingested claims. Not what is necessarily true. It's as subject to garbage in, garbage out as anything, and you don't know what it's actually trained on.
> It didn't work, and why it didn't work turned out to be more interesting than the policy would have been.
Spoken like a true Claude.
Snarking aside, I am glad that our AI agents make it cheap enough to do these experiments and publish these write-ups that people finally bother to publish null findings. Very useful!
The idea of publishing null findings is very attractive, but the problem here is that they didn't record what the actual problem they're trying to solve is. It seems a bit like someone prompted Claude to try and make tool calls more efficient and Claude responded with a detailed writeup on why LRU can't be beat for making a KV cache. Which might be interesting, if that were the question I had asked.
That line doesn’t strike me as overtly AI written, overall yes though
It's using a couple textbook [1] LLM tropes, 'negative parallelism' with a 'here's the kicker' tone.
[1] https://gist.github.com/ossa-ma/f3baa9d25154c33095e22272c631...
I always have a tinge of...perhaps 'sadness' when reading such articles because they are a lot like those large fishing nets that drag along the bottom of the ocean catching and destroying in quantities unneeded. From this list:
- Quietly/Delve/Tapestry/Landscape: they're all lovely words. I love(d) using them.
- "Serves as": I use this all the time too.
- "The result? Devastating". Perhaps it is a bit stilted, but "Outcome? Predictable." could be a nice turn of phrase.
- "It's worth noting": All academics everywhere use this in papers. See also: "Notably", "Essentially", "In general" and other such fillers that we reach for.
- "Think of it as": This is such a useful phrase to explain something. Analogy is how we learn after all.
- The fucking em dashes. I love them, and screw everyone who suggests one should stop using them. One big reason for using linux is the compose key, so the em dash is never far away.
- Same with the unicode decoration. Nah, screw you. I use these symbols all the time: °, ×, →, ⇒ etc. in my own notes. Again, compose key ftw.
Etc. (the list is endless)
I know what people will say, that everything is contextual, one has to study case-by-case etc. But I can't help feeling that someone is pointing a finger at me and accusing me before-the-fact. Just, AI has polluted the joy of writing.
I wouldn't fret too much: AI is still advancing so quickly that these specific tells will only last a short time. And then you are free to use em-dashes again, or at least no one will care.
I've seen Douglas Adams use almost exactly this in Dirk Gently's Holistic Detective Agency. A professor (IIRC) is talking to a former student. Quoting from memory
"Did you ever actually finish any papers?"
"No. But the reasons why not were always absolutely fascinating."
So, yeah, for me personally it's not an AI tell. I've seen it in the wild before AI.
Yeah, but that doesn't sound like AI at all. Not even close.
In that quote, Adams doesn't repeat himself, before making an assertion. He realises the audience has more of a context window than three words.
The same is not true, of the token generation that... Mostly Claude models... Have taken to.
LLMs glom onto pre-existing tropes, idioms, and verbal constructs.
And then subtly misuse them.
LLMs write like some people cook, adding bespoke artisanal Belgian sea salt, truffle oil and weirdly specific cheese without any thought given to how the result will taste.
> And then subtly misuse them.
For example?
https://x.com/badlogicgames/status/2098868942685012405
But it is specifically one of Claude's ticks
It is. And the reason for that is* surprising: the base knowledge corpus is clickbait internet articles.
*not at all
I suspect it’s more like Claude etc. optimizing for a tone that gets the green checkmark from RLHF reviewers, probably over and over in many contexts where style is otherwise an incidental concern.
It is more surprising than that: it's because the humans in the loop like it.
If you're being given an A-B choice between cinnamon-flavored shit and lutfisk-flavored shit, you might convince the experimenter that cinnamon-flavored shit ranks highly in user preferences.
I would expect a preference for lutefisk flavored shit, the problem is then you then deduce from that that people prefer lutefisk flavor over cinnamon, rather than that you should stop feeding people shit (and also stop feeding them lutefisk.)
The LLMs are going to turn us into stochastic parrots.
I'm guessing he means overall the kind of circling and concluding that Claude does, like "the actual truth is more interesting than it seems" or "the real smoking gun is not the x, it's the y that was under our noses all this time" type of writing.
"That's not a result, that's a broken harness, and it's worth publishing because I expect it to be common."
That's Claude. But I agree and am also glad the author, uh, clauded this up. :)
More examples from this article:
"A block with resident children isn't evictable. So the baseline is LRU over radix leaves, which is what SGLang and vLLM actually implement. Beating naive flat LRU would be a strawman."
"The two numbers measure different things in different regimes, and I initially framed this as a contradiction. It isn't."
"That's not a result, that's a broken harness, and it's worth publishing because I expect it to be common"
While AI was clearly used to generate and structure a first draft, I say it has been extensively rewritten by the author. In my experience, AI tends to avoid the first person, instead writing from a passive observer or even using sentences without verbs. The length of the sentences is also more natural. AI had trouble with bind-words in an argument. It just states separate facts and assumes because they are stated in order you have a reasoning. I'm not seeing that here.
Of course, idioms are preserved, and not everything is rewritten.
I quite ok with this style, it isn't grating. I expect this will be a style we'll see more of in science.
> In my experience, AI tends to avoid the first person, instead writing from a passive observer or even using sentences without verbs.
Not my experience with Claude at all.
Which AIs are you talking about?
This phrasing is a very specific "Hello, this is Claude"
The whole thing is clearly written by Claude.
Yes I covered that after the comma if you didn’t make it that far.
This sentence quoted just doesn’t seem like the AI smoking gun to choose given the rest of the content.
It was just the first example that stuck out to me. I didn't do an exhaustive check to look for the worst offender.
Sorry I assumed you read the whole thing, forgot where I was for a moment
Yeah, we have a habit of not bothering to read LLM-generated prose.
Yeah that's definitely the source of the didn't RTFA epidemic, surely.
LRU seems like the ideal strategy for most things LLM-related. Everything in this realm is about recency bias. I think it is a feature in this context, not a problem.
When I give an agent a piece of corrected information regarding a long running task, the last thing I want it to do is try and statistically compensate for the fact that it is new information. I want this new information to dominate the old information.
It makes them yes-men.
I work on inference at a neocloud, but opinions are my own .
The economics and thus tools you can throw at inference change at various scales . As a “blunt” contrived example , on a gb300 the GPUs communicate super fast over nvlink, and the cards can offload kv cache to dram and then disk, “fast enough “ for these tool heavy agentic workloads.
Which come together to mean that at high enough scale and in the right scenario, we can work with wild ttls on the kv cache and still comfortably hit SLAs and tokenomics.
The settings may change, but the two major problems in CS remain the same: cache invalidation, naming things, and off by one errors.
I've been building latency-sensitive LLM systems for a while, and I've come to rely heavily on pre-fill-considerate mechanics like ping-pong overlapped async context construction. For interactive mechanics, the worst case, even if rare, is problematic.
A toy/simplified version lives here: https://github.com/chaboud/goulash
Consideration of mutation rate (a sort of temporal Shannon-ish coding/ordering) lives in there (with some RoPE-friendly structuring). Note: That was a vacation project, not the day job, but similar principles apply even with larger models.
Based on the tables in part 4 it seems like some of the policies would have started overtaking LRU at larger KV-cache sizes.
And while the range of cache sizes chosen is reasonable based on the table about Moonshot's number at the start, the finding that all tested KV-cache sizes are so small that the 5-minute eviction never triggers should have been reason enough to reevaluate that choice
This is almost unreadable. The "papers" are never referenced anywhere, so the claims being refuted cannot be evaluated. The whole fact of the TTL doesn't seem relevant at all. There are many, many well-researched admission and eviction policies that this readme doesn't mention. I just don't get why we are reading this.
Here is the repo link: https://github.com/gauravapiscean/agentic-kv-cache
>Cross-request KV prefix caching is the largest practical lever in agentic LLM serving. It's why your coding agent's fiftieth turn costs a fraction of its first.
This is incorrect. If you are generating the 51th turn, then prefix caching makes the turns 1 to 50 cost a fraction of what they normally cost, to generate the 51th turn. Generating the 51th turn is more costly because it is not in the cache yet.
Unfortunately there is no way to steelman the statement, since quadratic scaling of attention means that the cost goes up even if you assume perfect caching of past inputs.
What the author actually means is something more mundane. The initial prompt is massive in a standard coding harness and this means prompt processing takes much longer than expected, creating the misleading impression that costs go down.
Edit: I noticed the word cross request too late. In that case the harness prompt is expected to be cached from other users, in which case the first message actually has a massive unfair advantage.
Author here. Context for why I did this:
There's a growing literature arguing LRU is the wrong eviction policy for agentic LLM serving, because agent sessions idle and LRU can't distinguish a paused session from a dead one. I found the argument convincing and built a simulator to exploit it. Three separate mechanisms, all lost to plain radix-leaf LRU.
The reason turned out to be more useful than the policy. When I measured — policy-independently — where recompute actually comes from on 393 real Claude Code sessions, requests arriving after a gap longer than the 5-minute provider TTL account for 17.5% of it. Requests arriving within 10 seconds account for 33.1%. The dominant waste is tight tool loops whose 88k-token working sets exceed cache capacity, not sessions idling past a TTL. That's a capacity problem, and liveness prediction can't touch it.
just curious, why the LLM writing even here?
it's just a bit disheartening to read Claude output for such a small comment like this.. it'd be great to read your own writings even if it's not as "perfect"
One side effect of extremely-accessible, high-quality English translation and, more importantly, English-grammar-and-idiosyncrasy-obeying AI, is that there will be more and more text that looks like this which comes from foreign, non-native-English countries.
Just FYI.
This will be especially true from non-English cultures where "avoiding shame" is high on the list of motivations.
Consider the upside, though: A much broader range of written perspectives written in high-quality, if slightly annoying, English.
I'm already seeing the benefit of this on X thanks to its autotranslation btw: I follow a few Chinese-language accounts now that I would have never been able to digest otherwise.
Its only high quality of this their thoughts. Autotranslation is different from choosing to have AI shape the expression permanently.
Yes, but without more evidence, how can you distinguish?
> Author here.
The best kind of correct.
Why/when would people expect agents to be idling?
I'd have thought they'd be busy (using cached prompt-prefixes) until they were finished.
Maybe waiting on a slow tool call or subagent or something?
Interesting! I admit the AI-written text is rough to read, it could have used a pass or two from an actual human. E.g. "Publishing it unresolved rather than tuning until it matches." -- thanks for not lying, I guess?
Fun:
> In my first run, Belady — an offline oracle — lost to LRU. That's not a result, that's a broken harness, and it's worth publishing because I expect it to be common.
> The cause: inserting a long chain into a near-full cache lets a policy evict the very prefix it is currently building. LRU is accidentally immune because just-inserted blocks have the newest timestamp.