So using this harness five qwen3.8-27b models roughly matched Fable 5 on a saturated coding benchmark from early 2025. No comparison to astra/6.1 sol/fable 5.1/opus 5.5. No comparison on a benchmark that reflects actual agentic work (and agentic behavior is what modern models optimize for anyway - probably makes qwen look worse than it is?). And also obviously using a very old benchmark like this:
> The updated release of the dataset with problems released between May 2023 and Apr 2025 containing 1055 problems
means essentially guarenteed contamination.
I have no idea if this harness is any good. Agressive orchestration could be interesting given how hard models are RLed on orchestration? But the methods used to evaluate this harness are not informative
I'm not surprised. I used Claude Code for the first time again since April because I had been hearing good things about Opus 5.5 and Fable 5.1, but I was shocked by how bad the harness is and how its only gotten worse since spring? I may be doing something wrong, but getting Opus/Fable to properly spin up non-blocking sub-agents takes like four or five attempts and it still cannot reliably poll tasks like test suites that take more than 30 seconds for proper completion.
I think even if you have the world's best model (not my opinion, but seems to be the popular consensus these days with Anthropic), if the harness around it sucks none of that intelligence is going to matter.
Every role but the verifier is the same
underlying model, called in a fresh context
and coordinating through a shared workspace on disk:
<ws>/task.md - the problem statement
<ws>/plan.md - the manager’s overarching plan
<ws>/tasks.json - a debug mirror of the task list
<ws>/notes.md - curated ideas / findings / partial proofs
<ws>/solution.py - the current solution
Then they describe how there is a ping-pong between manager and worker with fresh context for the worker, but i wonder, is that part or even the worker really necessary?
My use of Qwen 3.8 27B (which is the only model i use almost since it came out) is really doing something very similar to the above where first a task is described, a `PLAN.md` file is made with the (current) plan and tasks are either in the plan file or in a separate issue tracker the LLM has access to via a CLI tool, a `STATUS.md` that tracks the current status and a `GOTCHAS.md` file for writing down various notes/tidbits/gotchas (TBH this is largely because Qwen 3.8 27B does want to note down stuff anyway so it mainly acts as an attractor for not writing its notes down in the other two places :-P).
There isn't any real harness driving this, i just give it a task, tell it how it'll be handled and off it goes. I guess i'm playing the "role" of the harness there as i tell it to not go and do everything in one go :-P but i never bother with fresh contexts, Pi's "rolling" approach to compaction (i.e. summarizing the older messages to make space while keeping the recent messages as-is) make it almost a non-issue.
I have considered making it a planning tool, though as some tool it can use via the CLI (it is running in a container), not a harness - but at the same time it feels like it can do the same thing with the `PLAN.md`/`STATUS.md`/`GOTCHAS.md` triplet. The only thing the tool would help with is to provide a UI for me to check what it's currently doing, but even then i usually just stop it and ask it directly :-P then tell it to continue.
Perhaps at some point i'll make some custom benchmark with stuff that are vaguely similar to what i work on and try a bunch of different approaches. It is something i want to do anyway to see how and if Qwen4 27B (whenever that comes it) is an improvement over 3.8.
This is pretty close to a process I use (regardless of model size) and I would agree getting it into a workflow I can run with a one-liner is a good idea. There are a few ways of doing this and I have tended to fallback to markdown engineering. (custom agents and subs in opencode)
It's really interesting to see the progress in the 27-30B range, they have gotten quite good but still make noticeably more hallucinations or bad assumptions (ime).
it definitely feels like sooner than later there will be “Mythos level” cyber threats available on open source / uncensorable models. I think the only thing that can be done is continued work on hardening anything that is remotely close to “critical infrastructure” and software supply chain.
On one hand, self hosting will always be niche so you could pass a law like “if your serving inference at scale you need to have safeguards/ classifiers in place” since > 95% of inference will go through hosted services in the long run … but that could turn into thought policing dystopia real quick.
… on one benchmark, that has 2 modes of running. 1 shot and 1 shot with repair. Zero tools.
Then ignore the rules of the bench (which is a pretty obsolete one anyway), give it an agentic loop of sorts via delegation and python eyes and then win.
Harness changes dynamically depending on your LLM intelligence.
The smarter LLMs get, you apply less, as LLM takes care of pre-existing harnesses.
This is a shitty, unreliable, and unmaintainable way of using harness because the harness is built around for the sake of getting good benchmark scores (someone mentioned overfitting in comments, so that)
What are people using for a local offline harness? I've heard that OpenCode has lax approach to security and has some default dial home settings. Aider? Want to use Qwen3.8-27b locally for coding on 24gb VRAM. Best option?
It just barely fits in my RX 7900 XTX's 24GB of VRAM (TBH i'm not sure if dflash is much better than MTP - which is included as part of the GGUF - and MTP saves ~1GB of VRAM). I'm using Linux with Xorg/Window Maker as my DE so it uses no VRAM at all (Qt-based stuff - and i guess modern Gtk too - tend to use some VRAM for themselves, even opening Kate shows up in amdgpu_top :-P). The above gives me on average ~50 tokens/sec generation (drops to about 35 when the context is close to full) and ~700 tokens/sec prompt processing, which while not particularly speedy is fine for agentic stuff as i just give the AI a task and leave it in the background doing things. To save some VRAM i run the multimodal projector for vision on the CPU (the --no-mmproj-offload) and only one task is ever executed at the same time (--parallel 1). Also the kvcache is quantized to q8_0 - supposedly this makes the model a bit dumber, though the extra VRAM needed for the full f16 kvcache makes it more than ten times slower :-P (and AFAIK dense models like the 27B aren't as affected by it as MoE models are).
I'm using Pi[1] (a slightly older version that i've had remain offline and disabled the telemetry) for now (will replace it at some point since i'm not a fan of the ~500MB dependency tree, the almost-dark patterns for online access/telemetry and the increasingly worrying push for the devs' services, but i do like how it approaches compaction and that for the most part it doesn't need any handholding when some error happens - which is ideal for "let the AI do something and check it some time later").
It is running inside a container so it doesn't get any access to my real stuff (mainly to avoid the LLM making any mess - of which it sometimes does in the container, but well, it is contained :-P). In the container Pi gives the LLM some basic tools - read, write, edit and bash - and the last one is basically used to do most of the work (i've also made a bunch of tools the LLM can use via the CLI).
> The updated release of the dataset with problems released between May 2023 and Apr 2025 containing 1055 problems
means essentially guarenteed contamination.
I have no idea if this harness is any good. Agressive orchestration could be interesting given how hard models are RLed on orchestration? But the methods used to evaluate this harness are not informative
I think even if you have the world's best model (not my opinion, but seems to be the popular consensus these days with Anthropic), if the harness around it sucks none of that intelligence is going to matter.
My use of Qwen 3.8 27B (which is the only model i use almost since it came out) is really doing something very similar to the above where first a task is described, a `PLAN.md` file is made with the (current) plan and tasks are either in the plan file or in a separate issue tracker the LLM has access to via a CLI tool, a `STATUS.md` that tracks the current status and a `GOTCHAS.md` file for writing down various notes/tidbits/gotchas (TBH this is largely because Qwen 3.8 27B does want to note down stuff anyway so it mainly acts as an attractor for not writing its notes down in the other two places :-P).
There isn't any real harness driving this, i just give it a task, tell it how it'll be handled and off it goes. I guess i'm playing the "role" of the harness there as i tell it to not go and do everything in one go :-P but i never bother with fresh contexts, Pi's "rolling" approach to compaction (i.e. summarizing the older messages to make space while keeping the recent messages as-is) make it almost a non-issue.
I have considered making it a planning tool, though as some tool it can use via the CLI (it is running in a container), not a harness - but at the same time it feels like it can do the same thing with the `PLAN.md`/`STATUS.md`/`GOTCHAS.md` triplet. The only thing the tool would help with is to provide a UI for me to check what it's currently doing, but even then i usually just stop it and ask it directly :-P then tell it to continue.
Perhaps at some point i'll make some custom benchmark with stuff that are vaguely similar to what i work on and try a bunch of different approaches. It is something i want to do anyway to see how and if Qwen4 27B (whenever that comes it) is an improvement over 3.8.
It's really interesting to see the progress in the 27-30B range, they have gotten quite good but still make noticeably more hallucinations or bad assumptions (ime).
On one hand, self hosting will always be niche so you could pass a law like “if your serving inference at scale you need to have safeguards/ classifiers in place” since > 95% of inference will go through hosted services in the long run … but that could turn into thought policing dystopia real quick.
Then ignore the rules of the bench (which is a pretty obsolete one anyway), give it an agentic loop of sorts via delegation and python eyes and then win.
Sorry but this is not how benchmarking works
The smarter LLMs get, you apply less, as LLM takes care of pre-existing harnesses.
This is a shitty, unreliable, and unmaintainable way of using harness because the harness is built around for the sake of getting good benchmark scores (someone mentioned overfitting in comments, so that)
llama-server --model Qwen3.8-27B-UD-Q4_K_XL.gguf --ctx-size 131072 --parallel 1 --cache-type-k q8_0 --cache-type-v q8_0 --host 0.0.0.0 --port 8080 --spec-type draft-dflash --spec-draft-model dflash-Qwen3.8-27B-Q4_0.gguf --spec-draft-n-max 3 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --mmproj mmproj-BF16.gguf --no-mmproj-offload --image-min-tokens 2048 --reasoning-preserve
It just barely fits in my RX 7900 XTX's 24GB of VRAM (TBH i'm not sure if dflash is much better than MTP - which is included as part of the GGUF - and MTP saves ~1GB of VRAM). I'm using Linux with Xorg/Window Maker as my DE so it uses no VRAM at all (Qt-based stuff - and i guess modern Gtk too - tend to use some VRAM for themselves, even opening Kate shows up in amdgpu_top :-P). The above gives me on average ~50 tokens/sec generation (drops to about 35 when the context is close to full) and ~700 tokens/sec prompt processing, which while not particularly speedy is fine for agentic stuff as i just give the AI a task and leave it in the background doing things. To save some VRAM i run the multimodal projector for vision on the CPU (the --no-mmproj-offload) and only one task is ever executed at the same time (--parallel 1). Also the kvcache is quantized to q8_0 - supposedly this makes the model a bit dumber, though the extra VRAM needed for the full f16 kvcache makes it more than ten times slower :-P (and AFAIK dense models like the 27B aren't as affected by it as MoE models are).
I'm using Pi[1] (a slightly older version that i've had remain offline and disabled the telemetry) for now (will replace it at some point since i'm not a fan of the ~500MB dependency tree, the almost-dark patterns for online access/telemetry and the increasingly worrying push for the devs' services, but i do like how it approaches compaction and that for the most part it doesn't need any handholding when some error happens - which is ideal for "let the AI do something and check it some time later").
It is running inside a container so it doesn't get any access to my real stuff (mainly to avoid the LLM making any mess - of which it sometimes does in the container, but well, it is contained :-P). In the container Pi gives the LLM some basic tools - read, write, edit and bash - and the last one is basically used to do most of the work (i've also made a bunch of tools the LLM can use via the CLI).
[0] https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qw...
[1] https://pi.dev/
https://inference-docs.cerebras.ai/models/qwen-3.8-27b
That times 5 is steep!
https://spark-arena.com/leaderboard
(you'll have to filter model yourself, no query params)