I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately.
Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine?
The lack of long context is the main reason that I still end up using Anthropic.
The worst is when you need it to hold for example a number of papers in its head, or large and complex materials that it needs full resolution on and your context window ends up being perennially at 16%. You have about five minutes of conversation and it compacts and then you have to wait for it to read that again, get to 16%... and repeat.
372 was not perfect, but it was so much better and a godsend. It turned that 12 to 20% into more like 40%.
The fact there is no way to disable auto-compaction and no way to go back in the conversation history to before a compact makes codex a no-go for me on any codebase > 5kloc. Even worse it seems to fire randomly at 10-20% of context left. So really you have 80% of 272k as real usable context. Compaction kills my sessions, it hallucinates and is worse than starting fresh. I've had enough times screaming at my computer when it burns tokens on a large codebase, gets to 15%, auto-compacts, and hallucinates so bad it has to read the entire codebase agin, gets to 15%, auto-compacts....
Have you tried using another harness like Pi? You can easily turn off auto compaction and if you leave it on you can use /tree to go back before the compaction if you like.
> you can use /tree to go back before the compaction
This is really a killer feature in my opinion. I'm currently working on pi brains extension that is designed to solve the compaction issue I have, which is, I don't know what the AI knows after compaction.
The screenshot shot above contains what I call a compaction capsule which contains all the messages/events at the point of compaction. The "Files" section in the right side panel shows me how many files were read/modified since compaction.
After compacting, I can easily tell what Pi use to know and use `/tree` to go back in time.
I still wouldn't compact more than 5 times before starting a new session since compaction messages persists in Pi.
So the practical workaround is to set model_auto_compact_token_limit very near, but still below, the model’s context window. This postpones compaction rather than disabling it.
> The fact there is no way to disable auto-compaction and no way to go back in the conversation history to before a compact makes codex a no-go for me
Same. I've told @tibo many times about these issues but apparently OpenAI's focus is on features that buy them the most users ASAP instead of increasing harness quality.
That seems quite different from my design process. I write a plan.md that goes through multiple revisions. The plan is the memory. Restarting and reading the plan again to do another review is a good way to get a different perspective.
Start with the big idea. Pin down the product manager-level description, and maybe some details about which features are in or out, how the phased design/roadmap will work, maybe what the basic tech stack will be.
That document gets saved, and has the big picture context.
Next, we go a little deeper, flesh out schemas, APIs, a little more about what the code should look like, the overall testing strategy. This document gets saved, and the AI can reference it for context.
The next level is a very specific implementation plan. Think individual small JIRA tickets in an epic. The AI writes this based on previous context. I make sure each individual step has very explicit instructions about branch names, which branch is branched from where, the testing and documentation that will be done for that step. It should be obvious from the higher level documents, but that helps keep the AI to only worrying about the local document in its context window.
Then have a DAG of stacked pull requests designed for subagents to work through.
All of this keeps the context window down (as well as costs down, and accuracy up) and works super well, and I can get a tremendous amount of code that is easy to review through stacked PRs. Invariably I will want to fix earlier ones then percolate that down the stack, so I have some skills for that.
I try to do similar. I'll even break down parts of the plan into separate docs and have it solely refer to them one by one as needed with /clear between. It's a more manual process but not the worst. I could probably even whip up some kind of skill to automate it somewhat with subagents or similar.
You can just tell your main agents "I have these implementation docs in <DIRECTORY>, please spawn parallel subagents to implement all of them" and I generally find that works, and what's great is it's in parallel too so takes less wall clock time (even if there is slight token duplication from having the main agent and the subagent both read the prompt/doc, but overall I think it is less usage of tokens unless you go crazy with /clear).
So far I have three different folders in my project for three different types of md documents that I keep creating and revising together with the agent: reports, plans, and reviews
Every large feature has at least one of each file created for it in that order. I start by analysing stuff and making reports that might also contain design thinking and decisions. Once I'm satisfied we have properly analysed and thought though whatever it is we're working on then I might make a plan or just jump into implementation depending on how complex the issue is. And then I might have zero or multiple reviews of different kinds by different agents depending on how much verification I think the issue warrants.
This is my super simple worse-is-better development process. I make up what is needed on the spot.
I have these plan files as well, but it depends on the scope and scale of the things you're executing on, I think. However much detail gets put into the plan, it still doesn't help if part of what the model needs to understand is the fine-grained / perfect detail of a large surface area.
I agree. Compaction sucks, so I made tools that let the LLM selectively delete (and recall if needed) chunks of its context. You might want to try context bonsai if you're routinely hitting the auto-compaction wall.
I did calculations on the prefix cache effect on costs of sessions where I used it and found that the removal of tokens from context had a much bigger effect on reducing costs than cache busting had on increasing them. I should re-do that and publish it.
Yes the idea is cool but this could really hammer usage, especially just leaving it up to the agent to decide when to do it. I'm not surprised though, considering the github account is named "Vibecodelicious" and became active in December.
Poking around in the repo the whole implementation is an unsupervised LLM fever-dream.
It depends on what you mean by "unsupervised" - I've been strictly working through the agent, through specs the entire time, but it's been very supervised, I just leave the mistakes in-place and have it work from there.
However at this point it can completely maintain itself. When a new version of Claude or OpenCode is released, it updates itself to work on the latest version. It can also add new implementations for harnesses pretty reliably. It's actually pretty fun to watch it at this point. "Make this work on Hermes agent and message me when you're done" and an hour later or so, I can go play with it in Hermes.
When I've asked Codex agents about things that were in their context window, they've never – to my experience – been able to actually retrieve something from before compaction when using the proprietary compaction endpoint. Instead, they've had to consult their actual transcript.
So... at least as of a week ago or so, I don't believe so.
Instead of compaction, I very aggressively do everything with teams of subagents. The main agent keeps the high level context, and all of the nitty gritty detail and code details not needed by the main agent never even make it into the context.
This used to be really effective and cheap, now it lights my quota on fire, even after I fix the config and restore smaller subagents instead of like-agents
Multiple (slow) conversations seems to be the efficient path for me the past week or so
That is how the majority of developers use AI. Aim small miss small and all that. What's great about AI is that you can ask many many small questions about big data dumps (like logs and stack traces), for relatively little usage.
How does that work out in terms of usage? I imagine that, unless you're on the highest tier subscription or have a no expense spared approach to agentic coding, then you're going to hit a wall pretty fast.
All depends on context coherence. E.g. in Claude Code with the "1M context" models I am reluctant to push past 30%. So if that Codex 272K is 100% coherent than it is less of a difference. Still, even then the "hard fail" boundary has moved. Longer running agentic processes will surely run into this more often.
The solution isn’t to stuff everything it needs to know into the context, it is to have it write out what it needs to know for every part. It should write out plans, infrastructure decisions, etc, and then use those in future work as needed.
I suffered this same problem with Anthropic, ironically. Mainly with reverse engineering things. That would usually end up requiring a ton of context it had to remember, and then compaction would kick in and it would just forget half of it (that half being really important too), and even switching to Claude Code and having it document everything wouldn't really fix the problem. It was honestly quite sad because I've usually gotten better results with it, but it falls a part the moment you try doing something like that that's really complicated and requires a ton of different attempts to get it right.
The best way around this for me is planning a project well ahead of time and using subagents to execute individual tasks while the primary agent acts as a PM.
That isn't really a way around it for the kinds of things I'm talking about. The whole point is that getting all of the relevant context to even orchestrate it takes up most of the model's 272K.
For those kind of tasks, I do hierarchical planning, and then parts of the plan can focus one agent on keeping a smaller amount of pre-determined context when orchestrating.
Think: A VP of engineering has some super high level goal: "build a new product to beat our competitor in this new market". It then filters down and various levels of reports build more and more detailed yet more narrow pieces of planning/orchestration. They don't make it work by keeping all of the context in their head and micromanaging.
Yes, sometimes planning spans sessions too but I have even that broken down into stages where we start with the architecture and research, then save that, then define the milestones, then break down into the individual issues that the less sophisticated models can implement in a single session.
Pay attention to what it forgets, and start telling it to proactively note down those things into nominated files while it works, e.g. indexing topics covered by each chapter/section/page of each paper/material. GPT-5.6 is really good at finding context again, if given the smallest hint where to start looking.
You can also ask it to read your past sessions, find places where it was wrong after compaction, and figure out a strategy to persist the most important summary information.
I have several Hermes threads that have each had >10 compactions with a 200k context limit, and with the right instructions on note-taking, they require at most a "double-check that against our past decision records" to be put back on track after a slip-up.
I think I've seen the opposite end of this as well: I get the most value from any LLM when I ask it one very targeted question with a followup. I don't fall outside of any sort of context window sizing when I am doing that. That isn't to say that I don't have a few long engineering discussions that are helpful, but most are very to the point. It helps that I have decades of experience at this point and effectively know what to ask to narrow the scope quickly.
What. Anthropic's models have million token context windows! I'm switching to OpenAI next month, I can't believe they're still stuck on ~300k... Better start adapting to the new reality I guess.
Depends on what you are using. I have been experimenting with various compaction techniques in piclaw (https://github.com/rcarmo/piclaw/blob/main/docs/pipelined-co...) and Codex has native "opaque" server-side compaction at a dedicated OpenAI endpoint.
sometimes compaction is a death knell and other times it's totally fine. it really just depends on whether the thing you were doing at the very end has high correlation with stuff from early on in the session, or if there was an obvious ~break s.t. you were just on some specific subtask. the situation has clearly gotten a lot better than even 6 months ago-- i think openai handles compaction better in my own experience. but it's exactly the kind of thing that yegge was trying to get at with gastown-- i used to run every session s.t. i could re-animate it every time the compaction boundary came... now, when the compaction boundary comes i am so pot committed that i just hold my breath.
it's hard to modularize a monolith without keeping large chunks in context at first pre-delegation; the orchestrator -- however you implement it -- needs to carry as much of the original thing as possible into context and big monoliths make this task heavier.
so, tl;dr : I use a lot of tokens re-writing legacy codebases that were constructed with very little CS training in some huge spaghetti fashion by random people around the world.
A typical suggestion is to make the agent create or update a .md file once in a while, so the agent would remember the new important things that have shown up since the last update. The problem is, how would the agent know which things are really important. If it knows that well, then /compact should work well too...
If you’re using the pi.dev harness, I found this tool to be quite good as a compaction alternative: https://pi.dev/packages/pi-blackhole — it keeps a memory so every prior conversation piece is recallable, not lost.
If you want stuff to be referenced regularly, why are you doing it in the conversation instead of just putting the documents in the directory and referring to them in AGENTS.md or whatever?
You can mad customize it if you're willing to set up like, oh my pi or whatever.
I didn't quite get it tuned up enough to be a daily driver but I'm basically sure it can be hotrodded however you want. It comes out of the box with like four different vendor hostile compact strategies, including compacting into images at the smallest size Claude can read, which the "coding is basically solved" geniuses conveniently leak the resolution heuristic out of their website along with all the other side channels they print for the MSS.
>I use GLM5.2 and gpt-5.5 and 5.6 and never noticed any quality degradation near the limits of the context or due to compaction.
that's because you and your workloads probably fall into a bucket of users where compaction is tuned well.
but the reality is that those processes need to pick what to carry over, and that's a tough thing to get right for everyone, so for me compaction is a signal to restart the session or to tell the LLM to reference existing design docs lest we go off the rails.
large context limits is one of the things that make the open model competitors very attractive to me.
> The lack of long context is the main reason that I still end up using Anthropic.
Anthropic's "long context" is mostly marketing bullshit. It semi-reliably holds about the same amount of info, and then starts suffering the same issues.
I don't know if this is why they did it, but I think it is generally a mistake to go beyond this context size anyway.
I don't think people realize how much dumber the models get at larger contexts and how much more the token cost is. I never let claude get about 300k and in general I don't compact, I just divide up the work in chunks where I can fit it into 300k and I try to keep really clean and slim docs and modular code bases.
It's still kind of annoying, sometimes the larger context is useful for one time tasks, but I think if you are regularly going beyond 300k you are losing a lot and probably have a poorly designed code base.
Yeah I also try to compact/restart at 250k. I noticed though that it tracks with how big the project is so I guess the people needing more are just working on bigger projects
Exactly my experience. I would even draw the border at 100-150k. Long contexts would be nice, but they just don't work well, even if the models support them.
>> I don't think people realize how much dumber the models get at larger contexts and how much more the token cost is.
It does not match my experience that the model gets significantly dumber. It does get slower and more expensive, yes, but that's a sacrifice that needs to be made when working on anything complex.
My process involves having the main agent use subagents to explore what is needed for the given task. Then it writes a plan. Then it has the plan adversarially reviewed by more subagents and hardens it. After all is said and done, the 1M token window is 30-40% full. This flow would never work with 272k context, and in fact I've had to tone it down significantly for 5.6 Sol. Which, now that I think about it, probably explains why the results I get with it are inferior.
Am I dumb or does this chart make no sense? Or why does the line only go up even with compaction? Or maybe "overall trajectory size" is hiding some meaning I don't understand?
The chart makes sense and is describing the cumulative cost of a trajectory. Cache tokens are created when a trajectory’s prefix is used more than once. A larger pre-compaction context window means that a greater number of cache tokens are used per turn, and a larger number of turns are completed before compaction runs. So you get a cumulative cost that grows quadratically until the compaction event.
How can the overall trajectory length be the same across reasoning efforts? I don't see how this is possible even if reasoning is not included in the trajectory length calculation.
Not a fan of their context compaction and I feel like 1M-token context should minimum today. Each day I see how GPT 5.5 and 5.6 struggle a bit after each compaction before they get to the full speed, sometimes focusing too much on some older steering message that made it into the compacted context.
Note that GPT-5.6-Sol is ~2x as token efficient as Opus/Fable, so a 258K max is equivalent to ~516K on Claude models. You also may just be fooling yourself, as context rot is still a problem[1][2] and there's evidence compaction can be equivalent or better than longer context windows for agentic tasks[3]. Obviously it would be best if the models could reason over 1M context like they do over 256K context, but they can't yet.
The improved token efficiency would only account for the output and reasoning tokens occupying context, right? What about the user messages and other context like file contents? I’m not sure how these measure up in a long conversation
The fact there is no way to disable auto-compaction like every other coding harness makes me scream. It fires randomly at 10-20%, so really you now only have guaranteed context of 272k * 80%. So many times I've had it work on a large codebase, it is almost done with a task, it needs maybe a 2000 token response, but it drops below 20%, it churns and churns and churns and then I see the dreaded ``Context compacted`` and scream. You can't go back in the history to the state of the conversation before the compaction. So then it has to investigate the codebase again gets to 20% and then auto-compacts. So it then has to read more, and then auto-compacts again, and then I'm out of tokens.
Claude Code is much nicer than Pi, especially for long running autonomous work. Pi might be OSS but it's just not nearly as featureful or agentic as Claude Code.
Pi can do all the same that CC can. OSS is nice but the reason to use it is to control what is happening between me and the LLM. See: OP and many others for many reasons, bloated system prompts, the harness changing day to day, Anthropic shipping terrible and untested [features](https://news.ycombinator.com/item?id=48947776) that don't appear in the changelog, Codex ultra subagents nuking usage, etc. You can tell which harnesses are vibe coded.
I agree, but when it comes to the Pi defaults. I’ve spent a good deal of time (admittedly via vibe coding) writing extensions that completely alter the look/feel as well as functionality of Pi.
I view Pi as similar to NeoVim, both programs have kinda crappy defaults that nobody should really use but are incredibly extensible and customizable for power users willing to invest their time into them.
Interesting comparision with NeoVim, given that it was basically created to provide a better default (config) experience than classic vim. We might be close to PostNeoVim (or is helix [0] already that?).
Full agreement. I sincerely hope that reducing token is mostly a cost saving measure and not some roundabout way to get more token use. It still does not help. In my corp, bean counters rendered initially useful internal llm largely pointless as they severely limited context.
I swear there is a executive convention out there sharing worst practices.
Just use md files for working memory. No need for large context. LLMs get dumber as you put more in context, as it stretches their attention. Keeping context small is better for quality.
Big context windows made everyone lazy about deciding what actually deserves to be in there... compaction makes it worse, it's lossy compression applied to everything at once so the details you actually needed probably get lost. I always thought that retransmitting the full chat every turn is a foundamental problem... in my experience with conversational agents I clear the context every turn and reinject all the relevant info with a memory/context plugin I built myself, so small context stopped being a problem cos the model reads a few k of curated state instead of 200k of scrollback. Coding agents are harder and I haven't cracked that part, but in my opinion the real fix to the context problem is in this direction: a policy that keeps what's needed to finish the job or to do the next job and drops the rest. This can be achieved with a separate LLM specialized for exactly that.
> I always thought that retransmitting the full chat every turn is a foundamental problem
So did many NLP experts who worked on LSTM, GRU, etc. But transformers empirically won out and here we are.
It will be interesting to see if future model architectures need to take this into account again.
If humans are any indication, one thing we're still missing is efficient movement of information from short term to long term memory. Fine tuning in principle does something akin to this, but not efficiently.
I primarily use Opus for my day to day and I /clear often. 1M context sound great but it quickly degrades once you start hitting the 50% mark. I usually just /clear once I hit between 30 and 40 percent context and get much better results from the model. Compaction doesn’t really help much in my experience, starting fresh and having the model front load its context from scratch works much better IME. I have multiple skills with a pile of markdown docs organized by feature to assist in that first load by telling it where to find relevant information for the task. Works great.
I never felt that context size was any issue at all in codex. I don't know how their compaction works, but it just keeps on going as if it has no context limit at all. At least in my experience.
> I never felt that context size was any issue at all in codex.
I'm guessing you just started using codex, it suffered greatly from "model context size exceeded" errors early on, where compaction couldn't even recover from it, these errors just stopped happening maybe some month(s) ago.
Nowadays it's a lot better though and I've don't get stuck in those anymore. Although I don't like the idea that they don't surface what goes into the "concise summary" afterwards, make it really hard to know if it actually got everything important or not.
In general, codex seems to be moving into the way of hiding as much from the user as possible, and it wouldn't surprise me if they eventually start encrypting the entire session logs just like they did with agent>sub-agent prompts recently. Sad though as it's easily the best harness+model combo available today out of the ones I've tried.
That has nothing to do with context window or the models at all; it's the compute capacity.
Pretty much every single AI firm had the same issue when the new model came out, it's the result of everyone jumping onboard and consuming all available compute capacity to break it for days.
Claude had the same issues when they released their newer models.
> That has nothing to do with context window or the models at all; it's the compute capacity.
Quite literally a harness feature/bug though, that it's unable to compact when reaching the max context (which seemingly both codex and Claude Code struggled with in the beginning), not something that will happen at every model release forever.
Hasn't happen for at least the two last model releases in Codex, not sure why you'd imply that.
Claude also had issues with this kind of thing for months early on, too, where it simply couldn't compact its own emissions.
But definitely earlier GPT models suffered greatly as context got large, and the compaction itself in Codex was really crappy. That changed around January.
Most problems should be divide-and-conquerable into chunks that 300 vs 400k context is basically never an issue. A coding agent isn't an infinite chat.
No matter how good compaction is, on some big projects it needs to read a lot of files. In my experience the first 200.000 tokens go FAST, but after that it slows down. Most of my Fable sessions don't go over 500.000 tokens, I don't need to compact once. But when I use Codex a single session has to compact over and over again.
In my experience the only reason it needs to read a lot of files is if your agents.md is deficient. It should only need to read the file it's actually working on, maybe a few others. Everything else should be explained in documentation
Nobody does caching as well as DeepSeek, so I guess it's a big enough difference in the implementation to make it difficult.
If you use Reasonix with DeepSeek it gets silly, as it is append-only to work with how caching works. It gets something like 97-98% cached tokens in a long session. It makes an already cheap model even cheaper.
using open models and local AI via llamacpp, I'm compacting (via agent directed) between 55k-85k; going up to 120k is rare unless the agent is actually following a non trivial log tracing or some other thing that absolutely requires keeping large context.
I've also tailored an agent system prompt with llamacpp reasoning budget and message that asks the agent to spawn subagents then compress to reduce context bloat. It's in opencode using the dynamic context pruning, so it's mostly keeping a vector&direction without having to bloat up.
It mostly works to allow iteration into multiple sub components.
On long sessions even with workflows and sub-agents I typically hit around 500-750k tokens in context (say 4-16 hours of work in a session, across multiple 5-hour limits) with Claude Code.
I was going to explore paying for Codex since OpenAI seems to be a bit more generous with rate limits, but I'm now not sure - for the stuff I do limited context size would be a dealbreaker - basically lots and lots of documentation and guideline references, code review loops, documentation references and web searches, multi-repo exploration, lots of tool calls and so on.
I might just upgrade to Anthropic's more expensive Max subscription or something to get more subsidized tokens. Even with plan files and the plan mode, it's like a slot machine after compacting the context, sometimes steps or other details just evaporate in thin air, the less of that I need to do the better. It's not that it can't work, it just doesn't work reliably enough not to be annoying.
Luckily DeepSeek V4 Pro, GLM 5.2 and Kimi K3 don't seem to have those limits either - though DS is around Sonnet, GLM 5.2 feels a bit above Sonnet and only K3 only really is in the proper Opus ballpark that's good enough for me to work undisturbed, even if it seems to be slower.
Good move given some experienced issues and compaction across the 5.6 range is closer to 5.4 than 5.5, i.e solid and reliable.
Will say that 5.6-Sol is a minor bump in my benchmarks in most areas vs 5.5 but a severe regression in a few specific task focused on rearranging trees, addressing merge conflicts, etc. where the model to accomplish the task does not properly adhere to prompts in a way GPT-5 originally managed, not retaining parts of history in the way prompted despite specific instructions not to as that made the final completion easier…
I am of the conservative and cautious opinion that no model should be able to run destructive tasks at all, I have seen every model do things that make me concerned enough to maintain that opinion and know my evals can’t catch everything. But for 5.6-Sol specifically, I’d caution everyone to reevaluate how you run the model, maybe take a few more precautions you tend to forgo.
It is extremely capable as a reviewer and for extensive tasks, though for the later, the safety net I feel is required to be comfortable limits the utility. The code 5.6-Sol provides also still is a bit harder to parse in reviews.
Release strategy wise, feel it’s have been smarter to release only Luna and Sol now, then Terra a few weeks of posttraining later, I simply cannot see a purpose for it in the current form given how well both Luna and Sol scale up and down respectively with reasoning. Two models from a lab at a time is also the limit I feel one can properly assess at a time.
With purely quadratic attention, the cost of the token at 372K is 87% more than the token at 272K.
There are attention mechanisms that help to mitigate this, but you can clearly see in the chart that Tibo posted that the attention mechanisms they are using are still scaling quadratically.
It's not quadratic attention, you get that curve from the input tokens going up linearly, since the graph is measuring cumulative cost at each token count. Basically for y=5 it's 5+4+3+2+1, or f(x) = x(x+1)/2
I didn't try long coding sessions for a while. I had really noticeable performance degradation with all models above 100-150k and always try to stay below those numbers.
The other day I was trying to find out exactly how much context size 5.6 has in codex CLI (via subscription) and I didn’t find that documented in any of the obvious places. Sort of ridiculous that we have to find this out in X or Reddit, e.g:
I forget whether it's documented or not, and it is kind of annoying that that's how you find it, but it does tell you. Maybe useful for the future if nothing else.
Not sure if this is still up to date (2023), but https://arxiv.org/abs/2307.03172 shows that performance degrades mostly in the middle of the context.
Anecdotally I've been stuck in that situation of being at 400-500k tokens and "just one more prompt bro" will get the task done, and I appreciate not having to wait through a compaction. If anything, keeping the bloated context helps with accuracy at the expense of speed in these cases.
This is a shame, and hopefully it reverts soon. I switched from Claude to Codex just before the Fable release because Claude doesn't follow instructions and goes off the ranch too frequently. I really don't want to have to switch back to what I consider to be a fundamentally worse model.
I have over the last two months switched to OpenAI from Claude since it’s been vastly better for my cases. Wonder if I see any difference in the quality of the output with this change
I haven't used Codex in some time, but in Claude Code I force my context size down to 280k on purpose as in my experience your best work is done within the 30-250k~ range regardless of a models maximum context size.
I have rule files that guides the agent towards my coding standards, code style, house rules etc. They alone cost 60-80k tokens, and they are the backbone of my system that prevents slop. Pre 1M context, I had to build complicated tooling to re-include the relevant docs to the context upon compaction, which relied on unstable transcription file format, which was a pain to maintain. With 1M context I deleted all of those. Nowadays most of my sessions uses 300-450k context.
Another thing that's preventing me from trying Codex. (the other is @ referencing files not auto including them to the context)
1M should be table stakes for frontier models at this point for programming.
Try running your rule files through an LLM for optimization. 60k-80k tokens is massive.
Funnily enough, most anti-slop skills I found are both way too verbose and miss some common slop constructs.
I also reduced many rules from “When doing X, don’t do Y, but do Z.” Instead, the rule is “When doing X, do Z.” Fewer tokens and often works better.
I had one critical rule I was maintaining about searching the codebase using a structural index/graph and not grep. Every time the agent missed it, I asked it how to improve the rules. Eventually, I asked the AI to review that rule file and it rewrote it to be 30% smaller, but, crucially, structured to be more understandable by the LLM.
Another helpful thing was to ask AI to review my rules for things it can load on-demand when it works in that area.
AI cannot tell you why it did something. If you ask it why it didn't do X even though your prompt said so, it can come up with some plausible sounding explanation, which might even be correct, but there's a fundamental impedance mismatch between giving you the most plausible sounding answer and actually grounding that in knowledge of how it generated that answer.
TBF having a 60k personal style guide is also a bad idea, people get borderline AI-psychosed about how their special prompts are really steering the AI when in reality there is no way to evaluate this stuff. You invariably end up trying to define your style in terms of what it is and it isn't, it's like trying to define "red". "follow the style of my 5 most recent PRs" works pretty well.
Another interesting change in that commit is the addition of this section to the system prompt:
Before taking a destructive action:
- Make sure the action is clearly within
the user's request.
- Resolve the exact targets with
read-only checks when necessary.
- Do not use `$HOME`, `~`, `/`, a
workspace root, or another broad
directory as the target of a
recursive or destructive command
I've overridden my rm with this, which I threw together for fast-deletes of things like Rust target/ directories, and after seeing the GPT horror story, I taught it to flatly reject deletions directly under `/` and under home directories, with a message printing the path that it's trying to delete.
Not exactly a perfect mitigation, but given that the stated risk was the model mistakenly using the wrong $HOME, it seems like a reasonable safety. I should probably make it use an even scarier rejection notice, though.
This happened to me one time. On windows though, deleting C:/ lmao. The direction I'm heading now is a better harness, i.e. isolating codex at a container level with dedicated workspaces / mounts etc. I'm building something at the moment that serves my needs.
Codex is open source. You can build it yourself. You don't have to use upstream's parameter choices, compaction strategies, subagent heuristics, or whatever. Strange how people act like tweaks to these parameters are mandates when they're really just suggestions relayed to you via git.
For example: upstream recently changed the effort level hotkeys so M-. would stop at xhigh, not max. If you want max, upstream, you go to the /model menu. I didn't like this change, so I undid it locally. Easy peasy.
The models themselves support up to 1M. You are just charged more for all context over 400k. The 272k limit is just a client limitation to ensure you can never go over 400k since the maximum output size of the model is 128k.
This was done because they were charging the 100%/50% premium for long-context over 272k to Codex users, even though they set 372k as the default. It cost people a LOT of money.
If ChatGPT 5.6 is so good why do they continue to have the same motherfucking cache and usage issues that they were struggling with when 5.2 is new? Could not pay me to use this shit. Enjoy your reset bingo
That is a good point actually (this is how we at Hugston measure tokens, in bytes). 270k tokens should be around ~1100kb or 1 MB, so not really enough for a serious project.
Context size cuts like this are usually a cost/latency tradeoff rather than a capability one - serving a smaller window is cheaper and keeps latency in check, and most sessions probably don't need anywhere near 372k tokens anyway. Curious if this affects people who were actually relying on the larger window for big codebases.
You just need good compaction and a memory system with decent project layout and documentation. If you don't have that 1M tokens won't really save you either.
I know a lot of people like to say that compaction makes this moot, but the level of detail you lose across compaction is wildly too much for most things that I do, unfortunately.
Perhaps if your plans don't have as much detail, or if you're not, for example, having a discussion with a lot of nitty-gritty then it's fine?
The lack of long context is the main reason that I still end up using Anthropic.
The worst is when you need it to hold for example a number of papers in its head, or large and complex materials that it needs full resolution on and your context window ends up being perennially at 16%. You have about five minutes of conversation and it compacts and then you have to wait for it to read that again, get to 16%... and repeat.
372 was not perfect, but it was so much better and a godsend. It turned that 12 to 20% into more like 40%.
The fact there is no way to disable auto-compaction and no way to go back in the conversation history to before a compact makes codex a no-go for me on any codebase > 5kloc. Even worse it seems to fire randomly at 10-20% of context left. So really you have 80% of 272k as real usable context. Compaction kills my sessions, it hallucinates and is worse than starting fresh. I've had enough times screaming at my computer when it burns tokens on a large codebase, gets to 15%, auto-compacts, and hallucinates so bad it has to read the entire codebase agin, gets to 15%, auto-compacts....
Have you tried using another harness like Pi? You can easily turn off auto compaction and if you leave it on you can use /tree to go back before the compaction if you like.
> you can use /tree to go back before the compaction
This is really a killer feature in my opinion. I'm currently working on pi brains extension that is designed to solve the compaction issue I have, which is, I don't know what the AI knows after compaction.
https://gitsense.com/screenshots/pi-inspect-overview.png
The screenshot shot above contains what I call a compaction capsule which contains all the messages/events at the point of compaction. The "Files" section in the right side panel shows me how many files were read/modified since compaction.
After compacting, I can easily tell what Pi use to know and use `/tree` to go back in time.
I still wouldn't compact more than 5 times before starting a new session since compaction messages persists in Pi.
So the practical workaround is to set model_auto_compact_token_limit very near, but still below, the model’s context window. This postpones compaction rather than disabling it.
config.toml
> The fact there is no way to disable auto-compaction and no way to go back in the conversation history to before a compact makes codex a no-go for me
Same. I've told @tibo many times about these issues but apparently OpenAI's focus is on features that buy them the most users ASAP instead of increasing harness quality.
That seems quite different from my design process. I write a plan.md that goes through multiple revisions. The plan is the memory. Restarting and reading the plan again to do another review is a good way to get a different perspective.
I like to actually have hierarchical plans.
Start with the big idea. Pin down the product manager-level description, and maybe some details about which features are in or out, how the phased design/roadmap will work, maybe what the basic tech stack will be.
That document gets saved, and has the big picture context.
Next, we go a little deeper, flesh out schemas, APIs, a little more about what the code should look like, the overall testing strategy. This document gets saved, and the AI can reference it for context.
The next level is a very specific implementation plan. Think individual small JIRA tickets in an epic. The AI writes this based on previous context. I make sure each individual step has very explicit instructions about branch names, which branch is branched from where, the testing and documentation that will be done for that step. It should be obvious from the higher level documents, but that helps keep the AI to only worrying about the local document in its context window.
Then have a DAG of stacked pull requests designed for subagents to work through.
All of this keeps the context window down (as well as costs down, and accuracy up) and works super well, and I can get a tremendous amount of code that is easy to review through stacked PRs. Invariably I will want to fix earlier ones then percolate that down the stack, so I have some skills for that.
I try to do similar. I'll even break down parts of the plan into separate docs and have it solely refer to them one by one as needed with /clear between. It's a more manual process but not the worst. I could probably even whip up some kind of skill to automate it somewhat with subagents or similar.
You can just tell your main agents "I have these implementation docs in <DIRECTORY>, please spawn parallel subagents to implement all of them" and I generally find that works, and what's great is it's in parallel too so takes less wall clock time (even if there is slight token duplication from having the main agent and the subagent both read the prompt/doc, but overall I think it is less usage of tokens unless you go crazy with /clear).
So far I have three different folders in my project for three different types of md documents that I keep creating and revising together with the agent: reports, plans, and reviews
Every large feature has at least one of each file created for it in that order. I start by analysing stuff and making reports that might also contain design thinking and decisions. Once I'm satisfied we have properly analysed and thought though whatever it is we're working on then I might make a plan or just jump into implementation depending on how complex the issue is. And then I might have zero or multiple reviews of different kinds by different agents depending on how much verification I think the issue warrants.
This is my super simple worse-is-better development process. I make up what is needed on the spot.
I have these plan files as well, but it depends on the scope and scale of the things you're executing on, I think. However much detail gets put into the plan, it still doesn't help if part of what the model needs to understand is the fine-grained / perfect detail of a large surface area.
That sounds more monolithic than modular, but I guess some projects don’t decompose easily? Or maybe they do, but getting there is the goal.
I agree. Compaction sucks, so I made tools that let the LLM selectively delete (and recall if needed) chunks of its context. You might want to try context bonsai if you're routinely hitting the auto-compaction wall.
https://github.com/Vibecodelicious/context-bonsai-agents
I've implemented a similar approach – although I'm surprised not to see mention of cache prefix busting in there!
I did calculations on the prefix cache effect on costs of sessions where I used it and found that the removal of tokens from context had a much bigger effect on reducing costs than cache busting had on increasing them. I should re-do that and publish it.
Yes the idea is cool but this could really hammer usage, especially just leaving it up to the agent to decide when to do it. I'm not surprised though, considering the github account is named "Vibecodelicious" and became active in December.
Poking around in the repo the whole implementation is an unsupervised LLM fever-dream.
It depends on what you mean by "unsupervised" - I've been strictly working through the agent, through specs the entire time, but it's been very supervised, I just leave the mistakes in-place and have it work from there.
However at this point it can completely maintain itself. When a new version of Claude or OpenCode is released, it updates itself to work on the latest version. It can also add new implementations for harnesses pretty reliably. It's actually pretty fun to watch it at this point. "Make this work on Hermes agent and message me when you're done" and an hour later or so, I can go play with it in Hermes.
This is actually brilliant. I've seen other approaches where the context is selectively pruned, but never in a recoverable way.
P.S. Love the "bonsai" name btw!
ive built a variation that spins the chunks out as if they had been done by a subagent, and so the summary is like what the subagent returns
i couldnt figure out a nice enough way to reload the conversation, and didnt want to have to worry about the cache rebuilds so its a bit languishing
I have a hunch that OpenAIs proprietary compaction endpoint actually does something similar on the backend.
When I've asked Codex agents about things that were in their context window, they've never – to my experience – been able to actually retrieve something from before compaction when using the proprietary compaction endpoint. Instead, they've had to consult their actual transcript.
So... at least as of a week ago or so, I don't believe so.
Instead of compaction, I very aggressively do everything with teams of subagents. The main agent keeps the high level context, and all of the nitty gritty detail and code details not needed by the main agent never even make it into the context.
This used to be really effective and cheap, now it lights my quota on fire, even after I fix the config and restore smaller subagents instead of like-agents
Multiple (slow) conversations seems to be the efficient path for me the past week or so
That is how the majority of developers use AI. Aim small miss small and all that. What's great about AI is that you can ask many many small questions about big data dumps (like logs and stack traces), for relatively little usage.
How does that work out in terms of usage? I imagine that, unless you're on the highest tier subscription or have a no expense spared approach to agentic coding, then you're going to hit a wall pretty fast.
It's actually less token usage overall, since the gigantic context that builds up in the main agent is a huge driver of costs.
All depends on context coherence. E.g. in Claude Code with the "1M context" models I am reluctant to push past 30%. So if that Codex 272K is 100% coherent than it is less of a difference. Still, even then the "hard fail" boundary has moved. Longer running agentic processes will surely run into this more often.
The solution isn’t to stuff everything it needs to know into the context, it is to have it write out what it needs to know for every part. It should write out plans, infrastructure decisions, etc, and then use those in future work as needed.
Yea by the time you get to 300k context its IQ is a solid standard deviation lower anyway, why would you really want to beyond that?
Exactly. Just about every prompt I issue these days generates or edits an .md file.
I suffered this same problem with Anthropic, ironically. Mainly with reverse engineering things. That would usually end up requiring a ton of context it had to remember, and then compaction would kick in and it would just forget half of it (that half being really important too), and even switching to Claude Code and having it document everything wouldn't really fix the problem. It was honestly quite sad because I've usually gotten better results with it, but it falls a part the moment you try doing something like that that's really complicated and requires a ton of different attempts to get it right.
The best way around this for me is planning a project well ahead of time and using subagents to execute individual tasks while the primary agent acts as a PM.
That isn't really a way around it for the kinds of things I'm talking about. The whole point is that getting all of the relevant context to even orchestrate it takes up most of the model's 272K.
For those kind of tasks, I do hierarchical planning, and then parts of the plan can focus one agent on keeping a smaller amount of pre-determined context when orchestrating.
Think: A VP of engineering has some super high level goal: "build a new product to beat our competitor in this new market". It then filters down and various levels of reports build more and more detailed yet more narrow pieces of planning/orchestration. They don't make it work by keeping all of the context in their head and micromanaging.
Yes, sometimes planning spans sessions too but I have even that broken down into stages where we start with the architecture and research, then save that, then define the milestones, then break down into the individual issues that the less sophisticated models can implement in a single session.
I haven’t found I need to plan way too far ahead of time as long as I’ve defined the interface between the tasks
Pay attention to what it forgets, and start telling it to proactively note down those things into nominated files while it works, e.g. indexing topics covered by each chapter/section/page of each paper/material. GPT-5.6 is really good at finding context again, if given the smallest hint where to start looking.
You can also ask it to read your past sessions, find places where it was wrong after compaction, and figure out a strategy to persist the most important summary information.
I have several Hermes threads that have each had >10 compactions with a 200k context limit, and with the right instructions on note-taking, they require at most a "double-check that against our past decision records" to be put back on track after a slip-up.
I think I've seen the opposite end of this as well: I get the most value from any LLM when I ask it one very targeted question with a followup. I don't fall outside of any sort of context window sizing when I am doing that. That isn't to say that I don't have a few long engineering discussions that are helpful, but most are very to the point. It helps that I have decades of experience at this point and effectively know what to ask to narrow the scope quickly.
What. Anthropic's models have million token context windows! I'm switching to OpenAI next month, I can't believe they're still stuck on ~300k... Better start adapting to the new reality I guess.
Depends on what you are using. I have been experimenting with various compaction techniques in piclaw (https://github.com/rcarmo/piclaw/blob/main/docs/pipelined-co...) and Codex has native "opaque" server-side compaction at a dedicated OpenAI endpoint.
sometimes compaction is a death knell and other times it's totally fine. it really just depends on whether the thing you were doing at the very end has high correlation with stuff from early on in the session, or if there was an obvious ~break s.t. you were just on some specific subtask. the situation has clearly gotten a lot better than even 6 months ago-- i think openai handles compaction better in my own experience. but it's exactly the kind of thing that yegge was trying to get at with gastown-- i used to run every session s.t. i could re-animate it every time the compaction boundary came... now, when the compaction boundary comes i am so pot committed that i just hold my breath.
Compaction is a feature like swapping out a car for a tricycle is a feature for greater economy.
Curious, what kinds of tasks do you do that require such a large context window? Anything specific?
for me it's often codebase decomposition.
it's hard to modularize a monolith without keeping large chunks in context at first pre-delegation; the orchestrator -- however you implement it -- needs to carry as much of the original thing as possible into context and big monoliths make this task heavier.
so, tl;dr : I use a lot of tokens re-writing legacy codebases that were constructed with very little CS training in some huge spaghetti fashion by random people around the world.
What plan are you on?
With Anthropic I run out of quota very fast on a Pro plan with long contexts.
Is long context primarily for Max tier or for API usage? Or are there special ways of working with long context on a Pro plan?
Yep, Max. I would definitely struggle to code with Anthropic's plans on Pro.
A typical suggestion is to make the agent create or update a .md file once in a while, so the agent would remember the new important things that have shown up since the last update. The problem is, how would the agent know which things are really important. If it knows that well, then /compact should work well too...
If you’re using the pi.dev harness, I found this tool to be quite good as a compaction alternative: https://pi.dev/packages/pi-blackhole — it keeps a memory so every prior conversation piece is recallable, not lost.
If you want stuff to be referenced regularly, why are you doing it in the conversation instead of just putting the documents in the directory and referring to them in AGENTS.md or whatever?
You can mad customize it if you're willing to set up like, oh my pi or whatever.
I didn't quite get it tuned up enough to be a daily driver but I'm basically sure it can be hotrodded however you want. It comes out of the box with like four different vendor hostile compact strategies, including compacting into images at the smallest size Claude can read, which the "coding is basically solved" geniuses conveniently leak the resolution heuristic out of their website along with all the other side channels they print for the MSS.
OpenAI compaction is very different than Anthropic compaction, especially in 5.6
I use GLM5.2 and gpt-5.5 and 5.6 and never noticed any quality degradation near the limits of the context or due to compaction.
>I use GLM5.2 and gpt-5.5 and 5.6 and never noticed any quality degradation near the limits of the context or due to compaction.
that's because you and your workloads probably fall into a bucket of users where compaction is tuned well.
but the reality is that those processes need to pick what to carry over, and that's a tough thing to get right for everyone, so for me compaction is a signal to restart the session or to tell the LLM to reference existing design docs lest we go off the rails.
large context limits is one of the things that make the open model competitors very attractive to me.
Why arent you using /goal?
> The lack of long context is the main reason that I still end up using Anthropic.
Anthropic's "long context" is mostly marketing bullshit. It semi-reliably holds about the same amount of info, and then starts suffering the same issues.
I don't know if this is why they did it, but I think it is generally a mistake to go beyond this context size anyway.
I don't think people realize how much dumber the models get at larger contexts and how much more the token cost is. I never let claude get about 300k and in general I don't compact, I just divide up the work in chunks where I can fit it into 300k and I try to keep really clean and slim docs and modular code bases.
It's still kind of annoying, sometimes the larger context is useful for one time tasks, but I think if you are regularly going beyond 300k you are losing a lot and probably have a poorly designed code base.
Yeah I also try to compact/restart at 250k. I noticed though that it tracks with how big the project is so I guess the people needing more are just working on bigger projects
Exactly my experience. I would even draw the border at 100-150k. Long contexts would be nice, but they just don't work well, even if the models support them.
>> I don't think people realize how much dumber the models get at larger contexts and how much more the token cost is.
It does not match my experience that the model gets significantly dumber. It does get slower and more expensive, yes, but that's a sacrifice that needs to be made when working on anything complex.
My process involves having the main agent use subagents to explore what is needed for the given task. Then it writes a plan. Then it has the plan adversarially reviewed by more subagents and hardens it. After all is said and done, the 1M token window is 30-40% full. This flow would never work with 272k context, and in fact I've had to tone it down significantly for 5.6 Sol. Which, now that I think about it, probably explains why the results I get with it are inferior.
This was tweeted about when it happened, with some explanation from Tibo here: https://x.com/thsottiaux/status/2076543065045795309
To see replies: https://xcancel.com/thsottiaux/status/2076543065045795309
The linked tweet is an unofficial reply to Tibo's official info and Tibo makes a correction in a reply.
Am I dumb or does this chart make no sense? Or why does the line only go up even with compaction? Or maybe "overall trajectory size" is hiding some meaning I don't understand?
The chart makes sense and is describing the cumulative cost of a trajectory. Cache tokens are created when a trajectory’s prefix is used more than once. A larger pre-compaction context window means that a greater number of cache tokens are used per turn, and a larger number of turns are completed before compaction runs. So you get a cumulative cost that grows quadratically until the compaction event.
The Y axis is total cost in USD. For it to go down would mean OpenAI refunding you money.
The blue line (200K context) is lower than the red line (300K context). Indicating that it's cheaper to run blue rather than red over longer cycles.
How can the overall trajectory length be the same across reasoning efforts? I don't see how this is possible even if reasoning is not included in the trajectory length calculation.
I think Tibo was just keeping all else fixed and it’s an illustrative example rather than a perfect real-world trajectory.
Not a fan of their context compaction and I feel like 1M-token context should minimum today. Each day I see how GPT 5.5 and 5.6 struggle a bit after each compaction before they get to the full speed, sometimes focusing too much on some older steering message that made it into the compacted context.
Note that GPT-5.6-Sol is ~2x as token efficient as Opus/Fable, so a 258K max is equivalent to ~516K on Claude models. You also may just be fooling yourself, as context rot is still a problem[1][2] and there's evidence compaction can be equivalent or better than longer context windows for agentic tasks[3]. Obviously it would be best if the models could reason over 1M context like they do over 256K context, but they can't yet.
[1] https://arxiv.org/abs/2605.12366 [2] see F1 of GraphWalks 256K vs 1M in Opus 4.8 System Card: https://www-cdn.anthropic.com/0b4915911bb0d19eca5b5ee635c80f... [3] https://context-folding.github.io/
The improved token efficiency would only account for the output and reasoning tokens occupying context, right? What about the user messages and other context like file contents? I’m not sure how these measure up in a long conversation
For agentic use it impacts input tokens because the output is fed back into the model as input on subsequent turns.
The fact there is no way to disable auto-compaction like every other coding harness makes me scream. It fires randomly at 10-20%, so really you now only have guaranteed context of 272k * 80%. So many times I've had it work on a large codebase, it is almost done with a task, it needs maybe a 2000 token response, but it drops below 20%, it churns and churns and churns and then I see the dreaded ``Context compacted`` and scream. You can't go back in the history to the state of the conversation before the compaction. So then it has to investigate the codebase again gets to 20% and then auto-compacts. So it then has to read more, and then auto-compacts again, and then I'm out of tokens.
Use pi. Can't trust OAI or Anthropic to make good tools.
Claude Code is much nicer than Pi, especially for long running autonomous work. Pi might be OSS but it's just not nearly as featureful or agentic as Claude Code.
Pi can do all the same that CC can. OSS is nice but the reason to use it is to control what is happening between me and the LLM. See: OP and many others for many reasons, bloated system prompts, the harness changing day to day, Anthropic shipping terrible and untested [features](https://news.ycombinator.com/item?id=48947776) that don't appear in the changelog, Codex ultra subagents nuking usage, etc. You can tell which harnesses are vibe coded.
I agree, but when it comes to the Pi defaults. I’ve spent a good deal of time (admittedly via vibe coding) writing extensions that completely alter the look/feel as well as functionality of Pi.
I view Pi as similar to NeoVim, both programs have kinda crappy defaults that nobody should really use but are incredibly extensible and customizable for power users willing to invest their time into them.
Interesting comparision with NeoVim, given that it was basically created to provide a better default (config) experience than classic vim. We might be close to PostNeoVim (or is helix [0] already that?).
[0]: helix: A post-modern text editor. https://helix-editor.com/
Pardon the plug, but I have built a tool that ingests the state of conversations to a local DB in realtime. It has both macOS and Linux clients.
When context compaction introduces a gap, I use the /total-recall skill to pull prior turns back into context and off it goes.
The tool is free for personal use and has a source available local cloud option to sync convo histories across multiple machines.
http://contextify.sh/docs
Full agreement. I sincerely hope that reducing token is mostly a cost saving measure and not some roundabout way to get more token use. It still does not help. In my corp, bean counters rendered initially useful internal llm largely pointless as they severely limited context.
I swear there is a executive convention out there sharing worst practices.
Just use md files for working memory. No need for large context. LLMs get dumber as you put more in context, as it stretches their attention. Keeping context small is better for quality.
Big context windows made everyone lazy about deciding what actually deserves to be in there... compaction makes it worse, it's lossy compression applied to everything at once so the details you actually needed probably get lost. I always thought that retransmitting the full chat every turn is a foundamental problem... in my experience with conversational agents I clear the context every turn and reinject all the relevant info with a memory/context plugin I built myself, so small context stopped being a problem cos the model reads a few k of curated state instead of 200k of scrollback. Coding agents are harder and I haven't cracked that part, but in my opinion the real fix to the context problem is in this direction: a policy that keeps what's needed to finish the job or to do the next job and drops the rest. This can be achieved with a separate LLM specialized for exactly that.
> I always thought that retransmitting the full chat every turn is a foundamental problem
So did many NLP experts who worked on LSTM, GRU, etc. But transformers empirically won out and here we are.
It will be interesting to see if future model architectures need to take this into account again.
If humans are any indication, one thing we're still missing is efficient movement of information from short term to long term memory. Fine tuning in principle does something akin to this, but not efficiently.
I primarily use Opus for my day to day and I /clear often. 1M context sound great but it quickly degrades once you start hitting the 50% mark. I usually just /clear once I hit between 30 and 40 percent context and get much better results from the model. Compaction doesn’t really help much in my experience, starting fresh and having the model front load its context from scratch works much better IME. I have multiple skills with a pile of markdown docs organized by feature to assist in that first load by telling it where to find relevant information for the task. Works great.
I never felt that context size was any issue at all in codex. I don't know how their compaction works, but it just keeps on going as if it has no context limit at all. At least in my experience.
> I never felt that context size was any issue at all in codex.
I'm guessing you just started using codex, it suffered greatly from "model context size exceeded" errors early on, where compaction couldn't even recover from it, these errors just stopped happening maybe some month(s) ago.
Nowadays it's a lot better though and I've don't get stuck in those anymore. Although I don't like the idea that they don't surface what goes into the "concise summary" afterwards, make it really hard to know if it actually got everything important or not.
In general, codex seems to be moving into the way of hiding as much from the user as possible, and it wouldn't surprise me if they eventually start encrypting the entire session logs just like they did with agent>sub-agent prompts recently. Sad though as it's easily the best harness+model combo available today out of the ones I've tried.
That has nothing to do with context window or the models at all; it's the compute capacity.
Pretty much every single AI firm had the same issue when the new model came out, it's the result of everyone jumping onboard and consuming all available compute capacity to break it for days.
Claude had the same issues when they released their newer models.
> That has nothing to do with context window or the models at all; it's the compute capacity.
Quite literally a harness feature/bug though, that it's unable to compact when reaching the max context (which seemingly both codex and Claude Code struggled with in the beginning), not something that will happen at every model release forever.
Hasn't happen for at least the two last model releases in Codex, not sure why you'd imply that.
Claude also had issues with this kind of thing for months early on, too, where it simply couldn't compact its own emissions.
But definitely earlier GPT models suffered greatly as context got large, and the compaction itself in Codex was really crappy. That changed around January.
For me, Codex regularly forgets to complete its last task when compaction occurs, especially when the last message I sent was right before compaction.
It seems odd to me that Codex doesn't carry the plan file through context compaction? Claude does this, it re-reads it in full from disk
I’ve noticed it having weird message dropping and replay in general, but the compaction boundary has been pretty solid.
Most problems should be divide-and-conquerable into chunks that 300 vs 400k context is basically never an issue. A coding agent isn't an infinite chat.
No matter how good compaction is, on some big projects it needs to read a lot of files. In my experience the first 200.000 tokens go FAST, but after that it slows down. Most of my Fable sessions don't go over 500.000 tokens, I don't need to compact once. But when I use Codex a single session has to compact over and over again.
In my experience the only reason it needs to read a lot of files is if your agents.md is deficient. It should only need to read the file it's actually working on, maybe a few others. Everything else should be explained in documentation
That's quite small for my workloads.
I try to keep it under 200k but my DeepSeek and MiMo sessions can sometimes grow to 350k tokens when I try to squeeze one last iteration I compact.
Can't OpenAI copy DeepSeek K/V cache tech (from published papers) to make it super cheap?
Nobody does caching as well as DeepSeek, so I guess it's a big enough difference in the implementation to make it difficult.
If you use Reasonix with DeepSeek it gets silly, as it is append-only to work with how caching works. It gets something like 97-98% cached tokens in a long session. It makes an already cheap model even cheaper.
using open models and local AI via llamacpp, I'm compacting (via agent directed) between 55k-85k; going up to 120k is rare unless the agent is actually following a non trivial log tracing or some other thing that absolutely requires keeping large context.
I've also tailored an agent system prompt with llamacpp reasoning budget and message that asks the agent to spawn subagents then compress to reduce context bloat. It's in opencode using the dynamic context pruning, so it's mostly keeping a vector&direction without having to bloat up.
It mostly works to allow iteration into multiple sub components.
On long sessions even with workflows and sub-agents I typically hit around 500-750k tokens in context (say 4-16 hours of work in a session, across multiple 5-hour limits) with Claude Code.
I was going to explore paying for Codex since OpenAI seems to be a bit more generous with rate limits, but I'm now not sure - for the stuff I do limited context size would be a dealbreaker - basically lots and lots of documentation and guideline references, code review loops, documentation references and web searches, multi-repo exploration, lots of tool calls and so on.
I might just upgrade to Anthropic's more expensive Max subscription or something to get more subsidized tokens. Even with plan files and the plan mode, it's like a slot machine after compacting the context, sometimes steps or other details just evaporate in thin air, the less of that I need to do the better. It's not that it can't work, it just doesn't work reliably enough not to be annoying.
Luckily DeepSeek V4 Pro, GLM 5.2 and Kimi K3 don't seem to have those limits either - though DS is around Sonnet, GLM 5.2 feels a bit above Sonnet and only K3 only really is in the proper Opus ballpark that's good enough for me to work undisturbed, even if it seems to be slower.
Good move given some experienced issues and compaction across the 5.6 range is closer to 5.4 than 5.5, i.e solid and reliable.
Will say that 5.6-Sol is a minor bump in my benchmarks in most areas vs 5.5 but a severe regression in a few specific task focused on rearranging trees, addressing merge conflicts, etc. where the model to accomplish the task does not properly adhere to prompts in a way GPT-5 originally managed, not retaining parts of history in the way prompted despite specific instructions not to as that made the final completion easier…
I am of the conservative and cautious opinion that no model should be able to run destructive tasks at all, I have seen every model do things that make me concerned enough to maintain that opinion and know my evals can’t catch everything. But for 5.6-Sol specifically, I’d caution everyone to reevaluate how you run the model, maybe take a few more precautions you tend to forgo.
It is extremely capable as a reviewer and for extensive tasks, though for the later, the safety net I feel is required to be comfortable limits the utility. The code 5.6-Sol provides also still is a bit harder to parse in reviews.
Release strategy wise, feel it’s have been smarter to release only Luna and Sol now, then Terra a few weeks of posttraining later, I simply cannot see a purpose for it in the current form given how well both Luna and Sol scale up and down respectively with reasoning. Two models from a lab at a time is also the limit I feel one can properly assess at a time.
With purely quadratic attention, the cost of the token at 372K is 87% more than the token at 272K.
There are attention mechanisms that help to mitigate this, but you can clearly see in the chart that Tibo posted that the attention mechanisms they are using are still scaling quadratically.
This becomes an optimization problem for token t:
min f[cost, quality loss] = compaction cost[t] + compaction quality loss[t] + token cost[t]
It's not quadratic attention, you get that curve from the input tokens going up linearly, since the graph is measuring cumulative cost at each token count. Basically for y=5 it's 5+4+3+2+1, or f(x) = x(x+1)/2
https://pbs.twimg.com/media/HNFc4Dma8AA76FW.jpg?name=orig
On the one hand, compacting at 300k saves OpenAI 40%. That's great.
On the other hand, $10 for 1M tokens still seems really high? It's not too hard to blow through that in an hour or two.
I didn't try long coding sessions for a while. I had really noticeable performance degradation with all models above 100-150k and always try to stay below those numbers.
The other day I was trying to find out exactly how much context size 5.6 has in codex CLI (via subscription) and I didn’t find that documented in any of the obvious places. Sort of ridiculous that we have to find this out in X or Reddit, e.g:
https://www.reddit.com/r/codex/s/hVv29obfFD
You can run ’codex debug models' into jq!
I forget whether it's documented or not, and it is kind of annoying that that's how you find it, but it does tell you. Maybe useful for the future if nothing else.
it's in the GitHub repo
Does the context size really matter that much if a “dumb zone” starts around 120k-150k anyway?
Things change fast! For Fable 5 it definitely feels past at least 272k
hmm, what about other models?
Not sure if this is still up to date (2023), but https://arxiv.org/abs/2307.03172 shows that performance degrades mostly in the middle of the context.
Anecdotally I've been stuck in that situation of being at 400-500k tokens and "just one more prompt bro" will get the task done, and I appreciate not having to wait through a compaction. If anything, keeping the bloated context helps with accuracy at the expense of speed in these cases.
This is a shame, and hopefully it reverts soon. I switched from Claude to Codex just before the Fable release because Claude doesn't follow instructions and goes off the ranch too frequently. I really don't want to have to switch back to what I consider to be a fundamentally worse model.
Even Fable doesn't fully follow instructions.
I have over the last two months switched to OpenAI from Claude since it’s been vastly better for my cases. Wonder if I see any difference in the quality of the output with this change
I haven't used Codex in some time, but in Claude Code I force my context size down to 280k on purpose as in my experience your best work is done within the 30-250k~ range regardless of a models maximum context size.
claudes/codex performance suffers after 250k anyway and my workflow reset after every 200 so this is a non-issue
Looks like we will be waiting quite a while to see a 1M context window from OpenAI
HugstonOne increased coding context size, from 1 to 4 Million ctx
I have rule files that guides the agent towards my coding standards, code style, house rules etc. They alone cost 60-80k tokens, and they are the backbone of my system that prevents slop. Pre 1M context, I had to build complicated tooling to re-include the relevant docs to the context upon compaction, which relied on unstable transcription file format, which was a pain to maintain. With 1M context I deleted all of those. Nowadays most of my sessions uses 300-450k context.
Another thing that's preventing me from trying Codex. (the other is @ referencing files not auto including them to the context)
1M should be table stakes for frontier models at this point for programming.
Try running your rule files through an LLM for optimization. 60k-80k tokens is massive.
Funnily enough, most anti-slop skills I found are both way too verbose and miss some common slop constructs.
I also reduced many rules from “When doing X, don’t do Y, but do Z.” Instead, the rule is “When doing X, do Z.” Fewer tokens and often works better.
I had one critical rule I was maintaining about searching the codebase using a structural index/graph and not grep. Every time the agent missed it, I asked it how to improve the rules. Eventually, I asked the AI to review that rule file and it rewrote it to be 30% smaller, but, crucially, structured to be more understandable by the LLM.
Another helpful thing was to ask AI to review my rules for things it can load on-demand when it works in that area.
It’s optimized to death. Every word, every comma. I have a rule file for writing rule files.
Problem stems from using my own abstractions instead of common frameworks.
For a full stack session, backend + front end + docs about the part we are working on adds to that amount.
I’ve accepted it because it results in code exactly the style I would’ve written. It’s a good tradeoff to cut off the slop.
AI cannot tell you why it did something. If you ask it why it didn't do X even though your prompt said so, it can come up with some plausible sounding explanation, which might even be correct, but there's a fundamental impedance mismatch between giving you the most plausible sounding answer and actually grounding that in knowledge of how it generated that answer.
TBF having a 60k personal style guide is also a bad idea, people get borderline AI-psychosed about how their special prompts are really steering the AI when in reality there is no way to evaluate this stuff. You invariably end up trying to define your style in terms of what it is and it isn't, it's like trying to define "red". "follow the style of my 5 most recent PRs" works pretty well.
Another interesting change in that commit is the addition of this section to the system prompt:
Looks like the fix for this bug where Codex would occasionally accidentally delete your entire home directory: https://twitter.com/thsottiaux/status/2077630111499882637https://github.com/tekacs/fast-rm
I've overridden my rm with this, which I threw together for fast-deletes of things like Rust target/ directories, and after seeing the GPT horror story, I taught it to flatly reject deletions directly under `/` and under home directories, with a message printing the path that it's trying to delete.
Not exactly a perfect mitigation, but given that the stated risk was the model mistakenly using the wrong $HOME, it seems like a reasonable safety. I should probably make it use an even scarier rejection notice, though.
I also... have backups.
This happened to me one time. On windows though, deleting C:/ lmao. The direction I'm heading now is a better harness, i.e. isolating codex at a container level with dedicated workspaces / mounts etc. I'm building something at the moment that serves my needs.
That's ok, if AI deletes my home directory I can just ask AI to regenerate my home directory.
Codex is open source. You can build it yourself. You don't have to use upstream's parameter choices, compaction strategies, subagent heuristics, or whatever. Strange how people act like tweaks to these parameters are mandates when they're really just suggestions relayed to you via git.
For example: upstream recently changed the effort level hotkeys so M-. would stop at xhigh, not max. If you want max, upstream, you go to the /model menu. I didn't like this change, so I undid it locally. Easy peasy.
Not true. Try to increase the context window and you will be greeted with an error when you reach the real limit enforced server-side.
The models themselves support up to 1M. You are just charged more for all context over 400k. The 272k limit is just a client limitation to ensure you can never go over 400k since the maximum output size of the model is 128k.
Perhaps not this for this parameter, but for others? Client side.
I am wondering why the codex repo is open source ? Can they afford it to be open source in the longer run ?
Not much point making it closed source at this point, coding agent harnesses are trivial to reverse-engineer.
Being open source was also a useful community lever they could pull when they were trying to catch up with Claude Code.
Not much point making anything closed source at this point, if the client has access to the binary.
What do you mean “afford”?
This was done because they were charging the 100%/50% premium for long-context over 272k to Codex users, even though they set 372k as the default. It cost people a LOT of money.
Codex compaction is really quite good. Smaller context doesn't really harm me.
I had a /goal running last night for 9.5 hours straight while I slept. When I woke up in the morning it was fully on task and focused.
Write up a detailed design doc. Build a decent AGENTS.md, and write up a good prompt or /goal.
Long context can be more of a curse than a benefit sometimes anyways.
If ChatGPT 5.6 is so good why do they continue to have the same motherfucking cache and usage issues that they were struggling with when 5.2 is new? Could not pay me to use this shit. Enjoy your reset bingo
"272k ought to be enough for anybody"
Thats not even enough to read a simple codebase, how is that enough?
Almost certainly an homage to “640kB ought to be enough for anybody.”
https://quoteinvestigator.com/2011/09/08/640k-enough/
That is a good point actually (this is how we at Hugston measure tokens, in bytes). 270k tokens should be around ~1100kb or 1 MB, so not really enough for a serious project.
Context size cuts like this are usually a cost/latency tradeoff rather than a capability one - serving a smaller window is cheaper and keeps latency in check, and most sessions probably don't need anywhere near 372k tokens anyway. Curious if this affects people who were actually relying on the larger window for big codebases.
We'll have to wait because Tibo (OpenAI) said this was a temporary measure and it'd go back to 372k soon.
The issue was more specific to higher token burn rates, not latency.
5.6 is a mess
Please elaborate.
> Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.
Sincerely, HN Guidelines Police :-)
https://news.ycombinator.com/newsguidelines.html
You just need good compaction and a memory system with decent project layout and documentation. If you don't have that 1M tokens won't really save you either.