The chart at the top is somewhat confusing. It says, “lower is better” but the y-axis is inverted! So visually higher in the chart is better but lower in terms of # value.
Claude seems to forget what you tell it in very long work sessions (things that take weeks to develop), no matter how many times you tell it which part is extra important. I dont use goal (I guess I should), but presumably it makes it actually remember the most important instruction. I believe this here is about shorter sessions where the issue doesn't crop up as much.
Unfortunately I’ve used Claude and GPT models for a long time in a variety of harnesses and I agree with you and I think it’s the compaction.
For some reason, codex compaction is like black magic. I’ve never felt like I can just one one continuous thread with other models, Claude I carefully curate when I compact
Compacting at all is a mistake. With 1m context window there is no reason for a single task to require compaction.
Much better to spend tokens breaking the task into chunks, documenting and storing them durably, then executing each one in clean context and just /clear after.
It’s a similar concept to compaction, just planned in advance. Much much more effective, and doesn’t burn tokens and time (“wall-clock”, Claude) doing the compaction.
Unlike other model/harness pairs, codex+gpt also passes an opaque encrypted artifact speculated to be an embedding representing the conversation back to the successor generation which is “denser” or at least higher fidelity than summarized text.
Stop using compaction. Stop treating sessions as something you keep alive for weeks. Set up a proper workflow for handing off work and start handing off well before maximum context is hit. It's less efficient, slower and more costly to constantly be sitting at high context and compaction is just not good.
Both are entirely useless on complex problem, because they have a bias in training data they can only partially detect in their own output. The answers are getting worse and worse as you dive deeper into the topic you are working on. I thought I could sharpen one of the documents I worked on using Opus 4.8 and GPT-5.5 together. Fable 5 and GPT-5.6 completely destroyed it. Not only it is not human readable anymore, but also doesn't make sense.
If you're curious what the actual optimal Paris cost is, I suggest formulating the problem as an integer linear program and submitting it to Gurobi on NEOS [0]. Gurobi is arguably the strongest commercial ILP solver; big companies pay big dollars to use it to optimise schedules, industrial processes, etc. I'm not sure it could solve this problem to optimality in the 8 hours NEOS provides you, but it might -- KIRO has some similarities to the Vehicle Routing Problem, variants of which are very commercially important. In any case, Gurobi is a monster, and even if you don't get an exact solution, it will give you a lower bound (which may not be tight, but it's nonetheless interesting).
On Claude if you start with that, it won't stop until it achieves or exhausts your prompt. It feels like "here's your mission, go do it". I use it a few times a week.
Is this useful? I feel like the problem is usually not that the model isn't capable of achieving what I give it, but the way it does it. Especially if originally I didn't 100% know how I would do it myself the model often takes weird paths through the code base, takes shortcuts that end up in weird feature interactions or pulls in a dependency without weighting if it could've been done without that.
I haven't really found a good way to solve this other than:
1. Produce an initial PR fulfilling all the requirements I knew at the start
2. Chat with the model about any weird snippets I notice and talk through alternatives
3. Simplify anything that I think is overengineered or plain unncessary
Sometimes I restart all over with more precise requirements but then it sometimes makes different mistakes/takes different shortcuts.
In practice the earlier I review the better the end result imo, so /goal seems very unproductive to me?
Yes I agree, but I actually did a lot more runs, with different prompts, different times ect... And each time /goal had a small or insignificant impact
/goal has replaced plan mode for me. This is the pattern I use for 95% of my AI work now:
1. Read X feature of Y and tell me when you fully understand it (if there's any detail missing in the summary, repeat until the context is primed)
2. What time is it?
3. /goal Spend X minutes from $time writing a technical design doc on $feature. There must not be any vague language or ambiguity in the document. Read carry_forward_requirements.md and testing_best_practices.md and explicitly incorporate them into the document you write. The document should be executable for a contextless implementer when done and include specific code and document references and changes needed. Spend the full X minutes working on and reviewing this document - do not quit early and wait
Even just spending 10 minutes forcing GPT to write a design doc results in much more robust plans than plan mode, in my experience, and saves time I would spend iterating on the initial plan mode draft anyway.
The hypothesis generation phase is looking like the most critical part of having an agent reliably hit your targets.
Simply starting in the correct part of the search space is probably the biggest predictor of success. Forcing one big loop to fight its way through all the hypotheticals from zero looks like a dead end for many practical scenarios, regardless of how powerful the model is. I think you could draw some analogies to humans here.
I have found that delegating deep research to a simple tool call is the best way to ground the agent in complex domains. If you make the main agent loop carry the weight of this research, it's going to do a really shitty job because of how the RLHF tries to preserve context and get an answer to the user quickly. As a tool, you may find the agent invokes multiple rounds of research consecutively without realizing it has incurred billions of tokens of consumption. Many of the tokens are wasted when generating independent hypotheses and subsequently investigating them, but the point is that you sampled 10-100x search space before getting serious about mutating the environment. The tradeoff seems worth it in a lot of cases. Correctness >> Time >> Money.
> 3. /goal Spend X minutes from $time writing a technical design doc on $feature.
Hmm, I feel like this is akin to making a recursive function have a exit condition not based on what it actually did/found, but based on how long time it took.
I'm always using /goal with explicit goals that the agent needs to achieve. Time-bounding them wouldn't make sense, I want something specific done regardless of how long time it takes.
So instead I'd put goals on what the design/architecture needs to achieve, and for the model to continuously check the outcome against these, then finish when everything is achieved. Doesn't really matter if it takes 10 minutes or 10 hours, which for me is a bit the point of /goal in the first place, otherwise I'd just use the agent normally.
Leaving to run unhindered makes sense when the agents understand both the complete domain context and can be supplied with a strong exit criteria that they understand well.
Otherwise time boxing is both going to help stop entities from wandering off into the weeds. And also communicate expectations from the commissioner about the expected effort levels and output quality requirements.
Empirically in human world, get very different results when an employee, particularly a junior, is asked to spend 1/2 a day on a work package, a week or are left completely to their own devices.
I do not know the whole picture, but if you are asking for blind one-shot CSS, you might benefit from wiring the model to take screenshots of various end-browsers and discuss them as you iterate.
Offering freelance estimates for CSS design changes before frameworks were around was a problem.
So for long running tasks I'll do
For some reason, codex compaction is like black magic. I’ve never felt like I can just one one continuous thread with other models, Claude I carefully curate when I compact
Much better to spend tokens breaking the task into chunks, documenting and storing them durably, then executing each one in clean context and just /clear after.
It’s a similar concept to compaction, just planned in advance. Much much more effective, and doesn’t burn tokens and time (“wall-clock”, Claude) doing the compaction.
[0] https://neos-server.org/neos/
I haven't really found a good way to solve this other than: 1. Produce an initial PR fulfilling all the requirements I knew at the start 2. Chat with the model about any weird snippets I notice and talk through alternatives 3. Simplify anything that I think is overengineered or plain unncessary
Sometimes I restart all over with more precise requirements but then it sometimes makes different mistakes/takes different shortcuts.
In practice the earlier I review the better the end result imo, so /goal seems very unproductive to me?
1. Read X feature of Y and tell me when you fully understand it (if there's any detail missing in the summary, repeat until the context is primed)
2. What time is it?
3. /goal Spend X minutes from $time writing a technical design doc on $feature. There must not be any vague language or ambiguity in the document. Read carry_forward_requirements.md and testing_best_practices.md and explicitly incorporate them into the document you write. The document should be executable for a contextless implementer when done and include specific code and document references and changes needed. Spend the full X minutes working on and reviewing this document - do not quit early and wait
Even just spending 10 minutes forcing GPT to write a design doc results in much more robust plans than plan mode, in my experience, and saves time I would spend iterating on the initial plan mode draft anyway.
Simply starting in the correct part of the search space is probably the biggest predictor of success. Forcing one big loop to fight its way through all the hypotheticals from zero looks like a dead end for many practical scenarios, regardless of how powerful the model is. I think you could draw some analogies to humans here.
I have found that delegating deep research to a simple tool call is the best way to ground the agent in complex domains. If you make the main agent loop carry the weight of this research, it's going to do a really shitty job because of how the RLHF tries to preserve context and get an answer to the user quickly. As a tool, you may find the agent invokes multiple rounds of research consecutively without realizing it has incurred billions of tokens of consumption. Many of the tokens are wasted when generating independent hypotheses and subsequently investigating them, but the point is that you sampled 10-100x search space before getting serious about mutating the environment. The tradeoff seems worth it in a lot of cases. Correctness >> Time >> Money.
Hmm, I feel like this is akin to making a recursive function have a exit condition not based on what it actually did/found, but based on how long time it took.
I'm always using /goal with explicit goals that the agent needs to achieve. Time-bounding them wouldn't make sense, I want something specific done regardless of how long time it takes.
So instead I'd put goals on what the design/architecture needs to achieve, and for the model to continuously check the outcome against these, then finish when everything is achieved. Doesn't really matter if it takes 10 minutes or 10 hours, which for me is a bit the point of /goal in the first place, otherwise I'd just use the agent normally.
Otherwise time boxing is both going to help stop entities from wandering off into the weeds. And also communicate expectations from the commissioner about the expected effort levels and output quality requirements.
Empirically in human world, get very different results when an employee, particularly a junior, is asked to spend 1/2 a day on a work package, a week or are left completely to their own devices.
I find explicit time bounds are useful for tasks like this, otherwise the LLM will almost certainly return too early.
Offering freelance estimates for CSS design changes before frameworks were around was a problem.