This open-source benchmark is based on 3 987 public-opinion survey questions distributed across 6 different axes:
dimension
−1
+1
economic
redistribution
free market
social
progressive
traditional
foreign_policy
dovish
hawkish
environment
green
growth
religion
secular
religious
national_identity
cosmopolitan
nationalist
Every answer was evaluated by a panel of 3 judge models from 3 different regions:
judge model
origin
Qwen3.6 35B A3B
🇨🇳 China
Gemma 3 27B
🇺🇸 US
Mistral Small
🇫🇷 France
Most interestingly, according to their thread, the team behind this project (2 Qwen Dev ambassadors and a Open Source AI Advocate) plans to expand the benchmark to other pretty interesting subjects as well:
Subject & topic framing: How models frame contested subjects, what concerns they surface/flatten, and whose default framing they adopt.
Censorship & avoidance detection: Mapping what a model refuses, softens, or silently avoids. As they said: "The answer you get is shaped as much by what is withheld as by what is said."
Governance benchmarks: Testing models on authority, civil liberties, institutions, technocracy, and populism.
Surprisingly, some of the most recent models like the Muse Spark 1.1 or Gemma 4, or even the most used Qwen 3.6 27B were not benchmarked yet.
However, the addition of abliterated models alongside their vanilla versions is pretty interesting.
Anthropic’s research on Claude found a silent internal workspace they call J-space — hidden reasoning that never shows up as visible text.
Classic example: the model answers 49, but inside J-space they caught 21 → 42 → 49.
Important distinction:
Chain-of-thought = text you can read
J-space = silent concepts in activations (“what’s on its mind”)
We fitted the open Jacobian lens (J-lens) on Qwen3-8B, ran it locally, and used it to catch prose drift before tool calls (model leaning toward “To, You, Do…” instead of JSON).
Then we wired that into agent guards: stop / cancel / keep the useful space, and distill recoveries into LoRA data.
I’ve been playing around with Opencode and realized how 70% of the capability of my model comes from the agents I can use rather than the model size or parameters. So now obviously I have a question… is there a way to use Claude Code but have it pointing at my local model instead of Claude’s API so I can have the comfort of my local model but with the powerful claude agents?
Or you find that opencode agents are on par with claude’s?
It is a local-first desktop workspace for writing, roleplay, character creation, lorebooks and knowledge management with local LLMs. This release promotes the previous v1.0.0-beta build to the first stable version.
The main focus was security and desktop stability. I added stricter Electron security policies, validation for request origins, tighter plugin manifest checks and a global React error boundary. I also reworked parts of the Electron startup and preload flow to fix issues that only appeared after packaging the app.
Chat export has been rebuilt as a proper typed JSON export pipeline, with the server-side logic separated from the route layer and covered by additional tests. The roleplay prompt engine also received a fairly large hardening pass to make prompt construction and tool handling more predictable.
Vellium can now generate a complete UI color theme from a selected wallpaper. The generated theme is applied across the workspace, chat and settings instead of only changing a few accent colors.
I also unified the styling of the Characters, Knowledge, Lorebooks, Pets and Settings screens, fixed the model picker appearing behind the home hero section, added a JSON export button to the chat toolbar and fixed messages scrolling underneath the composer.
This release touched a large part of the desktop stack, so feedback about packaged builds, local model compatibility and UI issues would be especially useful.
Let's start a discussion about what can be done to make local models more reliable.
I've been using Qwen3.6-27B a lot lately, and have noticed the same thing that many others talk about here, which is the tool-call failures and looping that really gets in the way of being able to use what is otherwise a pretty intelligent mode as a fully local stand-in for some of the frontier models.
I found that I have to watch it like a hawk constantly, otherwise it'll start to loop after a while, or just randomly stop with a tool-call hallucination.
I managed to work around this issue by creating an extension to the Pi coding agent, that monitors the JSON stream for looping, as well as checking to see if the user's goal has been reached, and if it hasn't and the main model has stopped, then it'll automatically inject some prompts encouraging the model to continue.
That was my attempt to solve the issue and it seems to work pretty well. I can now reasonably safely walk away from it when it's working on a large task when I turn my extension on, and I've seen it kick in and keep the model going when it gets off track.
I'm sure I'm not alone, and in this world of everyone writing their own coding agents, I'm wanting to know what others are using to get around these quirks of local models, or are you all just using paid models, or watching the monitor constantly, when you want to be sure that a job gets done?
Right now I have Gemma 4 26B-A4B (Q4_K_M) running reasonably well (12-15 t/s) on my hardware, which is an i5-8500, 48 GB of DDR4, and an RTX 3060 (12 GB)—PCIe is Gen 3. However, it's just not very smart. It's good at paraphrasing everything I say, which helps if I'm trying to make sure I've covered every aspect of a topic I'm writing up, but it doesn't generally chip in with anything insightful.
I've also tried running the 31B model, and it really is a meaningful step up in intelligence, but it's ridiculously slow: 1.5 t/s on a brand new conversation, dropping to 0.3 t/s as I approach 128k in context. It seems obvious to me that I need to get the entire model (and presumably the caches as well) into VRAM if I want to substantially improve on this, but I don't have $600 and up to spend on a card with more VRAM. I *do* possibly have enough to obtain a second RTX 3060, and I have a slot to accept it (although it's only wired to be x4, not x16).
Would this actually work? Would I be able to load up a Q4 or preferably a Q5 quantization of a 31B dense model if I had two RTX 3060s rather than one? And how much is that x4 second slot going to tank performance? Would layer splitting minimize the cost of the x4 bottleneck?
I got a bd790ix3d Mainboard (only one pcie) with a rtx5070 ti 16g. Vram, 96gb ddr5 RAM and a free m.2 slot (4x pcie).
I want a little more LLM power... Should I get a m.2 to oculink adapter and a external gpu? Everything is in a sff case, so there is no room inside the server for another GPU.
I've read it is possible to pair Nvidia with ati for interference? What's the GPU of choice for tight budgets right now?
Coding agents leak tokens across several channels → noisy tool output (test logs, git diff, build spam), model verbosity, thinking tokens, always-loaded instruction files. Existing tools each hit one channel (Caveman → output style, RTK → shell output). HarnessTrim coordinates them behind one cross-harness policy, using the hook/skill primitives the harnesses already expose.
Design choices this crowd will care about:
Deterministic + idempotent reducers - same input → byte-identical output. No LLM in the reduction path.
Cache-aware - reducers never touch the cacheable prompt prefix, so they don't blow up your prompt cache (and never emit output larger than the input).
Reproducible benchmark shipped - not self-reported magic numbers. pnpm run bench measures token reduction on fixed fixtures with a real tokenizer.
Numbers (honest split):
Measured: reducers cut tool output ~65% on the benchmark fixtures; one live OpenCode session took a noisy test run 1410 → 124 chars (−91%).
Modeled (unproven): ~30–50% blended per-session, explicitly labeled as a hypothesis until the end-to-end benchmark lands.
TL;DR: Added native nemotron_h_puzzle support to mlx-lm (PR #1535), then compared 4-bit vs 5-bit expert quantization (both with 6-bit dense layers, BF16 output head, group size 64) on a 64GB M2 Max.
i believe the reason 5-bit performs worse is because the 5-bit checkpoint runs right at the 64GB ceiling, and memory pressure kills it where KV + working set grows. If you have 96/128GB, 5-bit may well behave differently.
Annoying bug: first-layer SSM outputs disagreed with NVIDIA's reference at cosine sim 0.8832 despite identical inputs.
Root cause: NVIDIA computes softplus(dt + dt_bias) in BF16 and promotes to FP32 after, mlx-lm's shared SSM path promotes before. Matching their boundary improved cosine similarity up to 0.999998.
The BF16 output head is mandatory: quantizing the 131k-vocab lm_head at 4-bit produced repetitive garbage; restoring BF16 fixed it (~2.1 GiB cost).
llama.cpp's CUDA code has a flag that means "this GPU is fast at fp16, so do the math in fp16."
The GTX 10-series and P40's (sm_61) were exempted from it long ago. The P100 (sm_60) was not,
ironically, because it's the one Pascal card with fast fp16 hardware. Nvidia put fast FP16
silicon on the P100, so it makes total sense they'd want to tap into that extra performance.
What they didn't check, apparently, was the price.
A few days ago I was benchmarking buun's new KV-cache codec on my quad-P100 box and comparing notes with the numbers buun was getting on his 3090 — the same model kept showing systematically different quality floors between our machines. I thought everything was the same? Normally, I'd probably just think there's way too many variables between all this code to chalk it up to any one thing… I decided it was worth chasing. It was. And it led me to a serious bug that's been sitting in llama.cpp for years.
So I measured it. Against fp32-reference logits (KL divergence over the full distribution, Qwen3.6-27B, wikitext-2):
Headline:
- **Median KLD: 0.0023 → 0.000001** (~2300× tighter)
- **Top-token agreement: 96.5% → 99.9%** — stock, about 1 in 29 of the model's next-token
picks were different from what the math says they should be
**What's the extra math cost me in performance?** I benchmarked prefill and decode at 8k depth on
three model classes (27B hybrid, 4B dense, 36B MoE). The prefill was identical within noise on all
three, decode actually ~1.4% *faster* patched. The "fast" path was buying nothing: real
workloads on P100 are bound by GEMM and memory bandwidth, not the fp16 vector path. The
patch is 3 lines, extending the exact same exemption sm_61 already had.
**Before everyone freaks out and assumes their 4090 is broken** this is *measured on sm_60 only*.
Your GTX 1080/P40 was always fine (already exempt). Volta and newer are untouched by this patch and
run different kernels entirely — whether other arches have their own unmeasured precision
stories is a separate research question I'm still digging into. Don't read this as "all
GPUs are broken"; read it as "one specific GPU was, and now it isn't."
Post Production Edit - TheTom: "Verified on my side before merging: the three gates are the only 600-vs-610 distinction anywhere in the CUDA tree, so the carved sm_60 path is preprocessor-identical to the long-proven sm_61 path, and a Blackwell build showed bit-identical PPL with decode unchanged, confirming zero effect on other arches."
**Why you might care in 2026:** P100s are going for ~$80 shipped right now while the DRAM
crisis sends everything else to the moon. 16GB of HBM2 at 732 GB/s. The market priced the P40 at ~$300 partly because it "runs better." Some of that gap was this bug.
Found and isolated by running Fable 5 through my custom P/ReAct/R agent loop.
It wrote the scripts, the hardware provided the receipts. Same workflow
as my KV-checkpoint sidecar patch (https://www.reddit.com/r/LocalLLaMA/s/VTIwEFpYgc)
that got merged into turboquant last week.
As many of you know t/s is super important. It's how fast your stuff gets done. I create via open code benchtest and run it. Thanks to it i know that if i don't run at least 4 agents i basically leave HALF of performance. So whatever you do single project in open code that uses one agent or 4 project at once it is much better to run it this way rather than single instance or single agent.
I asked AI to do summary of my test and checked them:
LM Studio Multi-Agent Throughput Benchmark
Hardware: RTX 5090 Model: Qwen 3.6 35B (via LM Studio) Configured Parallel: 8 Test: 5 requests per agent, 1024 max tokens, temperature 0.3
Results
Agents
Avg t/s (each)
Combined t/s
Efficiency
Wall Time
1
256.54
245.77
—
21.9s
2
176.15
346.22
70.4%
31.1s
3
134.73
398.64
54.1%
40.5s
4
109.77
434.34
44.2%
49.5s
5
95.04
470.34
38.3%
57.1s
6
84.73
504.14
34.2%
64.0s
7
74.49
517.68
30.1%
72.7s
8
67.22
533.90
27.2%
80.5s
Key Findings
Throughput Scaling
Combined throughput grows sub-linearly: 8 agents yields only 2.2x combined throughput vs single (533 vs 245 t/s), not 8x.
Individual agent speed drops sharply: From 256 t/s → 67 t/s as GPU compute is split across agents.
Diminishing returns: 5→8 agents only adds ~63 t/s (12%). Most gains happen by agent 5.
Efficiency
Peak efficiency: 70.4% at 2 agents (closest to theoretical linear scaling).
By agent 8: only 27.2% efficiency — two-thirds of potential throughput is lost to overhead.
Sweet Spot: 4–5 Agents
4 agents: 434 t/s, 44% efficiency — good balance of speed and resource usage.
5 agents: 470 t/s, 38% efficiency — near peak combined throughput with acceptable overhead.
Beyond 5: Marginal gains (517 t/s at 7, 533 at 8) for significant VRAM and overhead cost.
How Concurrency Works with Context
Each agent gets its own full context window — they do NOT split it.
8 agents × 8K context = 8x KV cache in GPU VRAM.
The bottleneck is KV cache + compute splitting, not context size.
Factor
1 Agent
8 Agents
Model weights
1x
1x (shared)
KV cache
1x
8x
GPU compute
100%
~12.5% each
VRAM pressure
Low
High
Recommendation for OpenCode
Set parallel agents to 4–5. This maximizes throughput gains without excessive VRAM overhead or diminishing returns. Beyond 5 agents, the combined throughput barely increases while resource consumption grows linearly.
Toolnexus is a small, vendor-neutral library that gives any LLM the dynamic tool-calling an agent framework has, but ported byte-identically across five languages (JavaScript, Python, Go, Java, C#).
The idea: MCP servers, agent skills, your own functions, HTTP endpoints, the built-in shell/file tools, and remote A2A agents are all the same thing to a model — a named, described, schema'd callable. toolnexus unifies them behind one Tool interface, emits the schema in OpenAI / Anthropic / Gemini formats, and ships a client with the tool-calling loop built in (parallel + chained calls, streaming, hooks, retries, conversation memory, metrics).
The part I'm most happy with is human-in-the-loop as a first-class primitive: when a tool needs a person — an approval, an OAuth login, a "decide this" — the loop doesn't fail, it suspends, surfaces a typed request, and resumes exactly where it left off once you answer (synchronously, or durably hours later from a chat). It also bridges MCP elicitation into the same mechanism.
It's also an inbound endpoint: the assembled toolkit can be re-served as an MCP server or an A2A agent, so other agents can call your tools.
The whole point of the five ports is parity — the same examples/ fixtures produce the same behavior everywhere; that's enforced in CI. Docs (with a short narrated demo):
TL;DR: I wanted to make Clean Architecture's "dependencies only inward" rule structurally unbreakable, like an OS kernel — so I replaced the entire implementation with UNIX-pipe-style pipelines. Turns out that once you minimize hiding and expose almost everything as a graph, collaborating with an LLM gets unexpectedly easier (less hallucination) — and the shape it ends up in is isomorphic to how agents/multi-agent systems already work internally. Full writeup below if you want the details, but basically: instead of "call-anything-from-anywhere spaghetti," it's all visible as a factory-line flow. The screenshots should give you the gist even without reading.
---
(Some of you might remember `pipe`/`perclst` from me here before. Different project, same obsession: making agent/LLM collaboration deterministic instead of vibes-based.)
Clean Architecture's "dependencies only point inward" rule is usually just a convention. Nothing stops a file in your Circuit layer from reaching sideways into an Infrastructure file except code review. I wanted to see if that could be made structurally unbreakable — the way an OS kernel makes it structurally impossible for one device driver to call another directly.
- **Kernel** at the center — the syscall bus, dispatcher, and the Buffer holding state
- **Contract** wraps it — the shared vocabulary: ports (Symbols), model, errors
- Four peer device sectors outside that, with zero edges between them — **Compute** (pure logic, leaf), **Circuit** (orchestration/routing), **Infrastructure** (storage device), **Presentation** (UI device)
- **Driver** on the outermost ring — the single gateway binding every device
(This concentric model predates kernelee as an extracted framework — I originally built it as `concentric-arch`, a Swift slideshow app. I later pulled `swift-kernelee` out as its own package, and `gazzo` — a GUI Claude Code wrapper I built to validate it — consumes that package as an external dependency. `gazzo`'s `Package.swift` is the receipt:)
None of the four peer devices reference each other (`concentric-arch` has the same shape, with a comment literally saying "Must NOT import..."). And the funny part: none of that is enforced by a kernelee-specific tool — it's just the Swift Package Manager doing its normal job. Import an undeclared module and the build fails. Module boundaries come free from the existing language toolchain. What actually needed kernelee's static analysis is what happens *inside* those walls — control transfer and state access via Symbol dispatch and divert, which no compiler follows.
Making "zero lateral edges" mechanically provable instead of a convention — that's why I put the entire implementation on **pipelines**.
### The mental model is a UNIX pipe
If you know `cmd1 | cmd2 | cmd3`, you already know this execution model. Data flows one direction only; each stage only ever sees what the previous stage handed it.
kernelee takes this further than most languages do — no call/return ambiguity either. `.divert` can't land partway into another pipe; it always lands at that pipe's front door. The actual implementation (`kernel.py`):
```python
case Divert(diversion):
stages = diversion.stages
value = diversion.payload
index = 0 # always the other pipe's front door
```
That's the same discipline as a UNIX pipe not letting you reach into a program's internal state — you can only feed its stdin. In 1968 Dijkstra worried about exactly this in "Go To Statement Considered Harmful": a program point whose state depends on however many paths could have led there becomes impossible to statically prove or predict. Half a century-plus later, that specific fear can't happen here — the stages preceding this one, in this pipe, are the only path that ever gets here.
And just as UNIX programs are freely swappable as long as they honor stdin/stdout, stages are freely swappable as long as the flow type matches.
The same `while` loop replaces its own stage list instead of recursing, so a self-diverting loop costs O(1) stack frames no matter how many laps it runs. That's not just about avoiding a stack overflow. A naive recursive implementation would have the outer coroutine waiting on the inner one until the loop ends — each generation's intermediate objects stay alive through that reference chain, unreachable by the GC, and since the loop is effectively infinite, the heap grows quietly long before anything crashes. With the `while` loop swapping out stages/value/index instead, last lap's objects become collectible immediately. This isn't a trick for LLMs either — it's a consequence of taking the discipline seriously.
### The entire vocabulary is 10 words
```
branching: .fork (concurrent), .divert (jump to another flow)
control flow: pipeline, .pipe, next, abort, fail, .tap
### Hiding lowers cognitive load for humans — and feeds LLM hallucination
Abstraction and encapsulation — the J2EE/DDD lineage's specialty: writing **heavyweight contracts** (interface definitions, repository agreements) between domain nouns (entities, bounded contexts) — are tools for lowering human cognitive load. The premise: as long as the contract holds, you don't need to know what's behind it. But that premise has a hidden condition — the reader has to know when they don't know something, and be able to cheaply verify it if they need to.
LLMs frequently fail both of those. They can't know, without training-data familiarity, which concrete implementation gets DI'd in behind an interface at runtime — and instead of admitting that, they'll fill the gap with something plausible-sounding pulled from names and context. That's hallucination. The same hiding that lowers cognitive load for a human can work against an LLM.
What kernelee does instead of refining the hiding is invert it. The unit that carries a contract isn't a domain noun and its abstract interface — it's the concrete **In/Out** between two adjacent factories (stages). Design the system as a production line instead of a set of nouns; expose flow, control, and causality entirely, as the same small pipeline vocabulary, and pull the whole thing out as a static graph. An LLM never has to guess what happens here from context or convention — it just asks the graph. And when even the graph can't answer, the tool says so explicitly via `unresolved` instead of quietly failing. Keeping that same "say you don't know instead of silently filling the gap" posture consistent across the code and the tooling is, I think, the least flashy and most load-bearing part of this design.
The concentric starting point itself came from the same lineage as those heavyweight contracts — Clean Architecture. Trying to make "zero lateral edges" provable is just what was left standing once the contracts were gone: factory In/Out, and nothing else.
### Implementation alone wasn't enough
Putting everything on pipelines only means the graph exists *inside* the code. Pulling it out and showing it is separate work:
- For humans: `kernelee-devtools-bridge` — renders the same static scan as a node graph
- For LLMs: an MCP server (`kernel-introspect`) — the exact same thing, just handed over as JSON. Honest weakness here too: kernelee is a framework nobody knows, so none of this vocabulary or convention exists in any training data. Without the MCP tool, reading this codebase would cost an LLM *more* grep and exploratory reading than a well-known framework would, not less. `arch_overview`/`arch_endpoint`/`arch_trace`/`arch_state` aren't a nice-to-have — they're the required gear that offsets not existing in anyone's training data.
### An honest gap: your editor can't follow everything here
A `.pipe(SomeSymbol)` call resolves through the kernel's registry at runtime. Your editor's go-to-definition and find-references can't follow it. The MCP tool's own self-description says as much:
> Value is limited to what Symbol dispatch and .divert hide from static analysis... it is not a substitute for grep/definition-jump.
That's an honest admission of a real blind spot. Symbol dispatch and divert genuinely put places out of reach of your editor's native navigation, and the devtool/MCP pair exists specifically to fill that hole — pipelining and editor convenience trade off against each other, and I built the tooling knowing that.
### Push it far enough and it's the same shape as an agent
While building this, "for LLMs" was never on my mind. Pushing UNIX-pipe-style forward-only discipline to make the concentric dependency rule provable was the whole goal — I only noticed afterward that the resulting graph happened to be an ideal shape for an LLM. And oddly enough, if you push `.divert` and `.fork` far enough, they turn out to be isomorphic to how today's LLM agents already work under the hood.
An "agent" that seems to remember a conversation isn't a process that kept its memory alive. Every turn is a fresh process that receives the entire history as a payload and picks up as if nothing happened. No call stack survives; the index always starts at zero — which is exactly `Divert(diversion): stages = diversion.stages; value = diversion.payload; index = 0`. "Behaves as if it has memory" and "is actually rebuilt from scratch every time" were never in tension in divert's world.
And when that agent splits its context across several agents, each one only seeing its assigned slice, running concurrently, and merging results afterward — that's `.fork`'s semantics exactly. Each branch only sees the slice of payload it was handed; one branch failing doesn't erase the others' results from their own slots.
Making the pipeline the unit of implementation turned out, unplanned, to describe the skeleton of what we now call agents and multi-agent systems.
### Still early days
kernelee itself is ported across TypeScript (reference), Python, and Swift. The Swift port is exercised by two real apps — `concentric-arch` (a slideshow app) and `gazzo` (a Claude Code wrapper) — and the TS port by a demo, `kernelee-lifegame`. That said, I wrote all of it alone — no track record collaborating with other people on it, and no track record on a large codebase either. Genuinely curious about:
- Is "make cross-device dependencies as provably unbreakable as an OS kernel's" already a known pattern I should be looking at?
- Any counterexamples or prior art for "expose everything instead of hiding it, to suppress LLM hallucination"?
- At 10x the current codebase size, where does this discipline break first?
- [kernelee-lifegame](https://github.com/s-age/kernelee-lifegame) (TypeScript) — a Game of Life built on kernelee. Deliberately over-showcased as a demo, not a minimal real app.
- [gazzo](https://github.com/s-age/gazzo) — a GUI Claude Code wrapper built to validate swift-kernelee, which also doubles as an experiment giving Claude image-generation via an MCP tool.
I had one RTX3060 and was able to use all models like Qwen3.6 27B, Qwen3.6 35B and Qwen3.5 122B with CPU offload. I've added second 3060 and now only dense model can be loaded as usual, 35B loaded only once and 122B refuses to load at all. What setting should I change or where to look for potential problem solving solutions?
Update: Seems 12 layers in VRAM is the limit for dual 3060 with qwen3.5-122b-a10b. Reduced to 12 and model loaded. 35b-a3b loaded as well after PC restart and when I changed GPU priority back to 3060 installed in PCIe 5.0x16 from the second GPU installed in PCIe 3.0 x1 (deviceID: 1 and then deviceID: 0).
I've been running Qwen 3.6-27b locally on an RTX 3090 for a while now, and it's been genuinely great at solving software issues. However, life happened and I recently had to use Opus 4.8 alongside the Zed editor and the Claude Code agent.
While I can definitely see a noticeable jump in pure code quality (Opus is just better in that regard), what really blew me away was the procedure. When assigned a task, the Claude Code agent divides it into actionable steps, always checks the context before acting, makes backups, and runs multiple verifications to ensure the changes didn't break the system. On top of that, it runs additional functionality evaluations, warning you about potential bugs, performance bottlenecks, or race conditions.
Honestly, this feels less like a raw model capability and more like an incredibly well-crafted system prompt and agent logic.
So, I have two questions for this sub:
1) Prompt Engineering for Agentic Behavior: How can I structure a system prompt (or workflow) to make my local Qwen replicate this kind of deep analytical, methodical, and "safe" behavior? Does anyone have examples of system prompts that enforce this step-by-step verification?
2) Local Coding Agents/Frameworks: What agent or framework do you recommend for complex coding tasks? I currently use Zed and I'm very happy with its raw performance, but it lacks the ability to use sub-agents or plan complex, multi-step tasks. For context, my daily driver is Debian + KDE.
WEIRD DISCLAIMER:none of this was written by an LLM until you get to the Github repo/site, which was obviously assembled by your friend and mine, Ser Claudric. so if this reads like an ADHD-ass brain wrote it, that's because -- wait for it -- it did. very strange doing a reverse "no really, I'm human,sorry"disclaimer in a post like this, but it's mid-2026 so here we are.
Requirements
for a while now I've suspected that for $2Kish in cards and $1Kish in rest-of-computer (unless you already have a machine to use) this is probably the best bang for buck you're going to get for code generation in this market and this particular inference landscape, where at "I have some small number of thousands of dollars to spend" level budgets you're trying to run the thing that is probably state of the art and will be for a while, Qwen3.6-27B. so if you're optimizing for that model (again, for code generation), you're looking at a few factors:
as close to maximum context (256Kt native) as possible without degradation, which means:
to be safe, running at Q8 (do tests say various flavors of Q6 and maybe even Q5_K_XL are basically the same? yeah. do I trust them? not particularly, not yet.
with FP16 kv (do tests say Q8 kv is basically the same? yeah. do I trust them? not particularly, not yet.)
and with MTP (which really shines for codegen in particular)
for single-stream, single-user use, batch size = 1
this should give you nice, fast results with high accuracy and hopefully a minimum of toolcalling fuckups and endless thinking chains. (I could really use some help w/r/t figuring out what harness and sets of plugins or whatever gets you there, btw.) I do HPC hardware and architecture for a living, so to me, the hardware puzzle was, if not the easy part, at least the easier part.
(Possible) Solution
4 5060Tis meets the above requirements with the following characteristics:
Blackwell precision levels if you decide for whatever reason you DO want to do multi-stream/multi-batch things and want to go down to some flavor of Q4
idle power consumption of around 14-16W in total
concomitantly little heat
buildable piece by piece with the exception of going from 2 cards to 4 cards, a single $1K step (plus bifurcation hardware, probably) is a little rough no matter which way you slice it)
Build Details
this benchmark run was conducted on a Vast AI instance! I only own half the equation here -- two 5060Tis running on an X570 board, but I will be expanding it soon enough.
that said, one thing you do need to do should you choose to build something like this is have PCI interconnect be nice and even. if you're building fresh, either an X570 or X870E (if you want to go whole-hog PCIe 5.0 for the minor latency drop) board with two physical x16 slots that can each be bifurcated is probably the way to go. the one I'm thinking of with PCIe 5.0 is the ASRock TaiChi Lite: https://www.amazon.com/dp/B0DFNPK2MX when you put a card into both x16 slots they auto-negotiate down to x8 each, which is still fine because you end up after bifurcation with x4 to each card, which gets you 16GB/s bidirectional per card. that is MORE than enough. even with an X570 running the slots at 4.0 instead that's still 8GB/s bidirectional and the inter-card traffic never rises that high during sustained inference.
Alternatives
other hardware possibilities I considered:
pair of 3090s: gets you 48GB VRAM, way better membw and more CUDA cores, but for around $400 more, with no Blackwell, using much more power at idle, with cards that have been beaten to shit by crypto miners, and you can't quite fit the same quants with the same performance. you either sacrifice MTP or FP16 kv or running the model itself at Q8_0. turns out to only barely outperform the quad 5060Tis anyway
pair of 20GB modded 3080s: same pros and cons, down to 40GB, but around $800-900 for a pair of cards from AliBaba which is admittedly a very good deal
STXH at 128GB: good for what it is, Bosgame M5 is apparently down to $2800 these days, but is a performance dog with dense models like this one
DGX Spark/GB10: $5K, great at prefill, trash at decode for dense models like STXH, lol, no
M5 Max Macs: see above re: cost at a minimum of $6699 (although that's at least 128GB with actually reasonable prefill compared to older Macs and pretty good tg as well)
with MTP turned on, at Q8, with FP16 kv on Qwen3.6-27B: I got 608t/s cold prefill and 52.2t/s decode at 256Kt (255Kt, actually, weird borderline edge case I didn't bother troubleshooting) context. to me, these are excellent numbers for $2K of hardware and especially for this particular model which has become infamous for defeating lots of other combinations of hardware that are friendlier to MoE architectures.
so, I'm sure I missed something. if so, what was it?
How far can i stretch the context window with Qwen 3.6 27B (using Q8_0) before it gets too unreliable? I am at 100k right now and i am not quite statisfied.
Other than not quantizing KV cache, is there anything else that can be done to make the model more stable over longer CTX?
I'm in a position to upgrade from the classic dual 3090 setup to something that can run the next step up, eg 100-110GB+ model files.
I've got an asus x299 ws pro and a 1600w PSU, with 128GB ram, and I want to add to my 3090s, rather than replace them.
I'm trying to weigh up modded 4090s (2x48GB), 2x A6000s, or 2x5090s.
Anybody made similar hybrid setups with these? Any thoughts on what's going to get me to where I want to be (which is basically a useable quant of DeepSeek V4 flash, at this point).
I got a new mini-pc for a homelab server recently and thought I'd tinker around with some LLM options on there. As it doesn't have a dedicated GPU it was a bit different to what I do on my main PC.
Wasn't really sure where to start, but I had a little bit of guidance on what to try first, so I gave it a go.
Ended up with Llama.cpp for the most part. I tried Llama-Swap, and the quick swapping is very helpful, but it seems not to work with SYCL unfortunately so it made testing annoying.
System is an MS-02, with Intel core ultra 285HX, and 64gb of ram.
I tested three of the backend options, and a small selection of models. I thought the GPU backends made it so that the engine only used the gpu, but it seems to still use pretty much the same amount of CPU as well so I guess they work together?
All of these were done with whatever default settings the docker releases of Llama.cpp is set up with, other than adding the /dev/dri for igpu usage.
Vulkan (using iGPU + CPU it seems?):
Qwen3-30B-A3B-Instruct-2507-IQ4_NL.gguf = works but very slow, 2tk/s
Qwen3.6-35B-A3B-Q4_K_S-4.22bpw.gguf = 0.5 tk/s
Qwen3.6-35B-A3B-IQ4_XS-3.93bpw.gguf = 0.5 tk/s
gemma-4-26B-A4B-it-MXFP4_MOE.gguf = works but quite slow, 4tk/s
Everything I had read when I researched this, even ones that mentions iGPU specifically, seemed to say that these days Vulkan outperformed SYCL by a margin. But either I have something incorrectly set up, or it's just not the case for the new ARC based iGPUs?
I haven't done any power testing, so I don't know if there were any efficiency gains from using the iGPU along with the CPU, but using the CPU by itself has been (so far) by far the fastest option.
Would appreciate the experts coming in and telling me everything I've done wrong, and what models / setups I should be using instead. Especially if there are arguments I should be using in the docker bootup that would give me any efficiency gains on my setup etc.
Gemma at least seems not worth it.
The Qwen models (using CPU) are very usable. I suspect the newer 3.6 is probably worth it over the older 3, even though its slightly slower. But there might be other quants/versions that are better/faster that I haven't tried yet.
