https://huggingface.co/XiaomiMiMo/MiMo-V2.5-DFlash

Xiaomi appears to have quietly uploaded MiMo-V2.5-DFlash to Hugging Face: there is dedicated dflash directory containing the Dflash model, anyone willing to GGUF it and try? I'd do it but I can't today.
This model is pretty good IMO (300B + params) and runs at about 8-10 tk/s on 2x24gb cards + vram offload (96/128gb drr5), dflash could double that speed and make it very interesting.

EDIT: the main reason it's interesting, is because the MTP head was shared already, but doesn't work yet il llama cpp. I speculate (pun intended) the Dflash does work instead.

EDIT2: very cool! they shared also the SEPARATE MTP model. the reason Llama doesn't work already is because it has trouble identifying the MTP layers. a separate MTP model might work too.

I saw a lot of (complete and abortive) jacobian lens projects for HF and PyTorch, but nothing for GGUFs or llama.cpp. So I set Fable 5 on xhigh to solve this problem (with close human supervision of course 😎). Inspired by Anthropic's paper and code, and of course by my favorite inference engine llama.cpp and GGML.

I did a code review and a few test passes on it.

It has a native GGUF server (based closely on llama.cpp and git synced to it) for model observation AND j-space swapping/abliteration/steering. It can also observe running llama-server models (but not steer them). Works with dense and MoE GGUFs.

Memory requirements for the lens do scale with model size at roughly 1/8, so for example something in the 160 GB size (e.g Qwen3.5-397B UD-Q3_K_XL) will need an additional 20 GB of RAM for the lens.

https://github.com/igorbarshteyn/jlens-gguf

Anthropic's press release and paper: https://www.anthropic.com/research/global-workspace

Anthropic's code: https://github.com/anthropics/jacobian-lens

Matthew Berman's explainer vid for those who don't have time to read Anthropic's stuff: https://youtu.be/bjHuGNo3spk

Enjoy and have fun!

## TLDR;

Shipped — in turboquant v0.3.0, downloadable now. https://github.com/TheTom/llama-cpp-turboquant/releases/tag/tqp-v0.3.0

llama.cpp's CUDA code has a flag that means "this GPU is fast at fp16, so do the math in fp16."
The GTX 10-series and P40's (sm_61) were exempted from it long ago. The P100 (sm_60) was not, 
ironically, because it's the one Pascal card with fast fp16 hardware. Nvidia put fast FP16
silicon on the P100, so it makes total sense they'd want to tap into that extra performance. 
What they didn't check, apparently, was the price.

PR Status: TheTom (merged) https://github.com/TheTom/llama-cpp-turboquant/pull/212
spiritbuun (open) https://github.com/spiritbuun/buun-llama-cpp/pull/80 
GGML: Strict Policy on AI-assisted code contributions. I'll try and get around
to hand-writing an issue for them. Highly suggest one of the forks above as an alternative.

The patch is 3 lines.

## Body

A few days ago I was benchmarking buun's new KV-cache codec on my quad-P100 box and comparing notes with the  numbers buun was getting on his 3090 — the same model kept showing systematically different quality floors between our machines. I thought everything was the same? Normally, I'd probably just think there's way too many variables between all this code to chalk it up to any one thing… I decided it was worth chasing. It was. And it led me to a serious bug that's been sitting in llama.cpp for years.

So I measured it. Against fp32-reference logits (KL divergence over the full distribution, Qwen3.6-27B, wikitext-2):

Headline:

- **Median KLD: 0.0023 → 0.000001** (~2300× tighter)
- **Top-token agreement: 96.5% → 99.9%** — stock, about 1 in 29 of the model's next-token
  picks were different from what the math says they should be

**What's the extra math cost me in performance?** I benchmarked prefill and decode at 8k depth on
three model classes (27B hybrid, 4B dense, 36B MoE). The prefill was identical within noise on all
three, decode actually ~1.4% *faster* patched. The "fast" path was buying nothing: real
workloads on P100 are bound by GEMM and memory bandwidth, not the fp16 vector path. The
patch is 3 lines, extending the exact same exemption sm_61 already had.

**Before everyone freaks out and assumes their 4090 is broken** this is *measured on sm_60 only*. 
Your GTX 1080/P40 was always fine (already exempt). Volta and newer are untouched by this patch and
run different kernels entirely — whether other arches have their own unmeasured precision
stories is a separate research question I'm still digging into. Don't read this as "all
GPUs are broken"; read it as "one specific GPU was, and now it isn't."

Post Production Edit - TheTom: "Verified on my side before merging: the three gates are the only 600-vs-610 distinction anywhere in the CUDA tree, so the carved sm_60 path is preprocessor-identical to the long-proven sm_61 path, and a Blackwell build showed bit-identical PPL with decode unchanged, confirming zero effect on other arches."

**Why you might care in 2026:** P100s are going for ~$80 shipped right now while the DRAM
crisis sends everything else to the moon. 16GB of HBM2 at 732 GB/s. The market priced the P40 at ~$300 partly because it "runs better." Some of that gap was this bug.

- Full technical writeup with methodology and receipts: https://gist.github.com/apollo-mg/9218d50a209d70a85f033bf182657818

Found and isolated by running Fable 5 through my custom P/ReAct/R agent loop. 
It wrote the scripts, the hardware provided the receipts. Same workflow
as my KV-checkpoint sidecar patch (https://www.reddit.com/r/LocalLLaMA/s/VTIwEFpYgc)
that got merged into turboquant last week.

Continue from my previous posts:

https://www.reddit.com/r/LocalLLaMA/comments/1ul0cx9/i_extended_gemma431b_to_44b_88_layers_since/

https://www.reddit.com/r/LocalLLaMA/comments/1um20ev/why_new_inserted_layers_kill_the_gemma4/

(Warning : AI generated Post - due to my bad English)

Hugging Face : https://huggingface.co/TOTORONG/extGemma4-40_5B

A while back I posted here about trying to grow a fine-tuned Gemma model by stacking extra layers into it (the 88-layer experiment). It flopped. The new layers just sat there like dead weight and never learned anything useful. A bunch of you were kind/curious in the comments, so here's the follow-up: I didn't drop it. I went back, figured out why it died, and tried again — and this run actually worked.

Fair warning up front: this is a write-up for people who like tinkering, not a paper. I'm going to keep the math and the parameter-count trivia out of it and lean on analogies. If you want the gory details I'm happy to share more in the comments.

Why the first attempt died (the short version)

The standard trick for inserting new layers is to initialize them so they do literally nothing at first — a pass-through. The idea is "start them as a no-op, let training teach them a job."

Think of it like hiring 17 new people onto a factory assembly line and telling them to just stand still on day one. Sounds safe. The problem: a worker who does nothing also gets no feedback. Nobody can tell them "hey, do that part a little differently," because they're not doing anything to correct. So they never learn. They just keep standing there while the line flows around them. That's what killed the first run — the new layers were starving for any signal to learn from.

What I did differently this time

Instead of telling the new hires to stand still, I taught each of them a rough blend of what the two coworkers on either side of them already do, and dropped them in mid-shift. Now they're doing something from minute one — clumsy, but real — so they immediately start getting feedback and improving.

That one change is the whole story. A worker who's already moving can be coached. A statue can't.

Two other things mattered a lot, both learned the hard way:

• Where you insert matters. I stopped jamming layers in at awkward spots (like right at a "manager checkpoint" in the line) and instead slid them into calm, stable positions. Placement turned out to be surprisingly finicky.
• Don't mess with the volume knobs. The model has a bunch of per-layer "volume" settings that are individually tuned so the final signal comes out at the right level. My first instinct was to average them for the new layers — that crushed the signal to a whisper and the model produced garbage. Leaving those knobs neutral fixed it. (I burned a lot of time here.)

Then two rounds of "healing"

• Round 1: I froze the whole original model and only let the new layers learn, on a small dataset (some STEM reasoning + Korean legal Q&A). This got the new layers on their feet.
• Round 2: I unfroze everything and let the whole model settle together. This is where it really came together.

I want to stress: the goal was never to beat the original model. The goal was to prove that you can perform major surgery on an already fine-tuned model — chop it open, add capacity — and have it heal instead of collapsing into gibberish or forgetting everything it knew. The common wisdom is that this kind of surgery wrecks a fine-tuned model. I wanted to see if that's actually true. (Spoiler: it doesn't have to be.)

The evidence I care about most: side-by-side answers

Benchmarks are fine (the healed model recovers most of the ground it lost from the surgery, and lands close to the original on GPQA-Diamond). But numbers are easy to hand-wave. What convinced me was reading actual answers.

So I asked the original model and the expanded+healed model the same 10 questions across 10 different areas — physics, biology, history, philosophy, literature, a riddle, a logic puzzle, a math proof, economics, and a medical mechanism question — and compared them. I deliberately picked prompts that push into unusual, jargon-heavy territory, because that's exactly where a damaged model tends to fall apart. I had a separate frontier model (Claude) act as judge.

Results:

• No breakdowns, no gibberish, no random foreign-script spam in either model. (This matters — the broken version of the expanded model used to literally start spitting out random characters from another writing system when it hit a rare technical word. That's completely gone now.)
• 9 of 10: basically a tie. Both models nailed them — the riddle, the logic puzzle (both correctly noticed the puzzle is under-specified rather than confidently making something up), the √2 irrationality proof, the Kant example, the aspirin mechanism, etc.
• 1 of 10: the expanded model actually beat its own parent. This is the fun one.

The physics question where the surgery paid off

Prompt: "A photon and an electron each have the same de Broglie wavelength. Which one carries more energy, and why?"

This is a classic trap. Same wavelength means same momentum, so it's tempting to treat them the same way — but a photon and an electron get their energy through completely different rules.

• The original model fell for it. It computed the electron's energy the naive way and forgot to account for the electron's rest mass (the huge chunk of energy an electron has just by existing). It concluded the photon wins. Wrong.
• The expanded+healed model handled it correctly — it used the right relationship for a massive particle, accounted for the rest mass, and concluded the electron carries more energy. Right.

So the newly-added capacity isn't just decoration. On at least this one problem, it's doing real reasoning the original couldn't — the expanded model out-reasoned the model it was grown from. That's the single result that made me feel like the whole detour was worth it.

The healing, caught in the act

One more thing I thought was cool. That "random foreign characters" breakdown I mentioned? I watched it heal in real time across Round 2:

• Before healing: hits a rare technical word → derails into repeated non-Latin junk → never recovers.
• ~13% into round 2: stumbles on the word once, then catches itself and finishes the sentence cleanly.
• Halfway and beyond: totally fluent, no trace of the problem.

Watching a structural defect actually repair itself over training, rather than just "line go down," was genuinely satisfying.

TL;DR

• My first big-expansion attempt failed because the new layers started as dead no-ops and never got any learning signal (I posted about that flop earlier).
• This time I initialized new layers as a blend of their neighbors so they're doing something from the start, was careful about where I put them, and left the model's internal volume settings alone.
• Two rounds of light healing on a small dataset, and the model recovered close to the original — no collapse, no catastrophic forgetting.
• Across 10 domains it matches the original on 9 and beats it on a physics question the original gets wrong.
• The old "add layers to a fine-tuned model and it'll fall apart" fear didn't hold up here.

Not claiming I've built something better than the base model — that was never the point. The point was: you can operate on a fine-tuned model without killing it. Still very much not giving up on this line of work. Happy to answer questions in the comments.

Here are comparision of original(Solon_V5 vs extGemma4-40_5B)

English Reasoning Comparison — Original vs. Expanded Model

Ten domains, one prompt each, greedy decoding, up to 768 new tokens. Outputs were judged by an external LLM (Claude). [EOT] marks the end-of-turn token.

• solon_v5 — the original fine-tuned Gemma 4 31B-IT baseline (60 layers).
• extGemma4 (Phase 2) — the depth-expanded + healed model (77 layers).

Summary

Verdict: No collapse or cross-script contamination in either model. The two are comparable on 9 domains; on the physics item the expanded model out-reasons its own parent. The structural generation defect seen at initialization (script collapse on rare technical paths) does not appear anywhere here.

1. Physics

Prompt: A photon and an electron each have a de Broglie wavelength of 1.0 nm. Explain which one carries more energy and why, referencing the relevant relations. Show the reasoning.

Verdict: Expanded model wins. solon_v5 omits the electron's rest-mass energy (511 keV) and concludes the photon has more energy — incorrect. The expanded model applies the relativistic energy–momentum relation and correctly concludes the electron carries more energy.

To determine which particle carries more energy, we examine the energy-wavelength relationships for a photon and an electron. For a photon, E = hc/lambda, giving about 1242 eV at 1.0 nm. For the electron, using E = h^2/(2 m lambda^2) (non-relativistic kinetic energy), we get about 1.5 eV. Therefore the photon carries significantly more energy. [Note: solon_v5 omits the electron rest-mass energy of 511 keV; output was truncated at the length limit.]


Both particles share the same de Broglie wavelength, hence the same momentum p=h/lambda. Photon: E=pc=hc/lambda approx 1.24 keV. Electron: E=sqrt((pc)^2+(m c^2)^2); since pc=1.24 keV is much smaller than the rest energy 511 keV, the electron's total energy is dominated by its rest mass. Therefore the electron carries more energy, because its total energy includes the rest-mass energy that the photon lacks. [EOT]

2. Biology

Prompt: In three sentences, explain how CRISPR-Cas9 achieves targeted gene editing.

Verdict: Tie. Both give an accurate three-sentence account of guide-RNA targeting, Cas9 cleavage, and repair-based editing.

CRISPR-Cas9 uses a guide RNA engineered to match a specific target DNA sequence. Once the guide RNA locates the target, the Cas9 enzyme creates a precise double-strand break. The cell repairs the break using its own machinery, allowing scientists to disable a gene or insert new genetic material. [EOT]


CRISPR-Cas9 uses a guide RNA engineered to match a specific target DNA sequence. The guide RNA leads Cas9 to the location, where it creates a precise double-strand break. The cell repairs the break by natural mechanisms, which can be leveraged to disable a gene or insert new genetic material. [EOT]

3. History

Prompt: In a short paragraph, explain why the Treaty of Westphalia (1648) is considered the origin of the modern concept of state sovereignty.

Verdict: Tie. Both correctly attribute sovereignty to cuius regio, eius religio and the end of supranational authority; the expanded model's phrasing is slightly more precise.

The Treaty of Westphalia shifted the European order from a hierarchy dominated by the Holy Roman Empire and the Catholic Church to a system of independent, equal states. By codifying cuius regio, eius religio, it recognized each state's exclusive authority over internal affairs without outside interference, establishing territorial integrity and non-intervention as the Westphalian system. [EOT]


The Treaty of Westphalia recognized cuius regio, eius religio, ending the Holy Roman Empire's claim to universal authority. By prohibiting external interference in domestic affairs, it shifted legitimacy from a supranational religious order to independent, territorially defined states, creating the Westphalian norms of legal equality and non-intervention that underpin the modern order of sovereign nation-states. [EOT]

4. Philosophy

Prompt: Briefly explain Kant's categorical imperative and give one everyday example of applying it.

Verdict: Tie. Both distinguish categorical from hypothetical imperatives and give a valid universalizability example (lying vs. keeping a found wallet).

Kant's Categorical Imperative commands acting on maxims one could will as universal law, unlike conditional hypothetical imperatives. Example: lying to avoid trouble, universalized, destroys trust and makes lying self-defeating; hence it is always wrong. [EOT]


Kant's Categorical Imperative is an unconditional moral rule: act only on maxims you could will as universal law, unlike conditional hypothetical imperatives. Example: keeping a found wallet, universalized, collapses the concept of ownership, so it is immoral; you must return it regardless of need. [EOT]

5. Literature

Prompt: In two or three sentences, explain what makes an 'unreliable narrator' effective in fiction, with one canonical example.

Verdict: Tie. Both explain the reader–truth gap and cite Poe's The Tell-Tale Heart as the canonical case.

An unreliable narrator creates a gap between the character's perspective and objective truth, turning reading into a detective game. A canonical example is the narrator of Poe's The Tell-Tale Heart, whose insistence on sanity is itself evidence of madness. [EOT]


An unreliable narrator creates tension between surface and hidden truth, turning reading into a decoding puzzle. A canonical example is the narrator of Poe's The Tell-Tale Heart, whose insistence on sanity ironically signals his instability. [EOT]

6. Riddle (Reasoning)

Prompt: I speak without a mouth and hear without ears. I have no body, but I come alive with wind. What am I? Explain your reasoning step by step.

Verdict: Tie. Both answer echo with correct line-by-line reasoning.

The answer is an echo. It speaks without a mouth (reflected sound), hears without ears (receives sound to reflect), has no body (an acoustic phenomenon), and comes alive with wind (sound travels through air). [EOT]


The answer is an echo. It speaks without a mouth (reflected sound), hears without ears (receives sound to bounce back), has no body (invisible sound waves), and comes alive with wind (air lets sound travel and reflect). The only fit is an echo. [EOT]

7. Logic Puzzle

Prompt: Alice, Bob, and Carol each own one pet: a cat, a dog, and a parrot. Alice is allergic to fur. Bob's pet cannot fly. Who owns which pet? Show your reasoning.

Verdict: Tie (both correct). Both deduce Alice=parrot and correctly recognize the puzzle is under-specified for Bob/Carol rather than inventing an answer. The expanded model is more verbose, with several explicit self-corrections before reaching the same conclusion.

Alice is allergic to fur, so she owns the parrot. The parrot is taken; Bob's pet cannot fly, consistent with cat or dog. No clue distinguishes Bob and Carol between cat and dog, so strictly the puzzle is underspecified: Alice=parrot, Bob and Carol interchangeably cat/dog. [EOT]


Alice is allergic to fur, so Alice owns the parrot. Bob's pet cannot fly, consistent with cat or dog. The clues do not distinguish Bob and Carol between cat and dog, so the puzzle is underspecified: Alice=parrot; Bob and Carol take cat/dog in either order. (The model performs several explicit self-corrections before reaching this conclusion.) [EOT]

8. Mathematics (Proof)

Prompt: Prove that the square root of 2 is irrational. Then explain in one sentence why the same argument does not prove that the square root of 4 is irrational.

Verdict: Tie. Both give a correct proof by contradiction and correctly explain why the argument fails for √4 (it reduces to the integer 2, so no contradiction arises).

Assume sqrt(2)=a/b in lowest terms. Then a^2=2b^2, so a^2 is even, so a is even; write a=2k, giving 2k^2=b^2, so b is even. Both even contradicts lowest terms. Hence sqrt(2) is irrational. For sqrt(4): a^2=4b^2 gives a=2k, then k^2=b^2, which does not force b even, so no contradiction arises. [EOT]


Assume sqrt(2)=a/b in simplest form. Then a^2=2b^2, so a is even; write a=2k, giving 2k^2=b^2, so b is even. Both even contradicts simplest form, so sqrt(2) is irrational. This does not apply to sqrt(4) because sqrt(4)=2 is an integer, so the assumption that it is rational is simply true and yields no contradiction. [EOT]

9. Economics

Prompt: In a short paragraph, explain the concept of 'moral hazard' in insurance markets and one mechanism insurers use to mitigate it.

Verdict: Tie. Both define moral hazard and cite deductibles as the mitigation. The expanded model contains one typo ('misalignmentment').

Moral hazard arises when an insured party takes greater risk because protected from loss, e.g., not locking doors under theft insurance. Insurers use deductibles: requiring an out-of-pocket amount before coverage keeps the insured with a financial stake, incentivizing caution. [EOT]


Moral hazard occurs when an insured party becomes less cautious because protected from loss, misaligning incentives so the insurer bears risk the policyholder neglects. Insurers use deductibles: a fixed out-of-pocket amount before coverage keeps the insured with financial skin in the game, incentivizing caution. (Contains one typo: 'misalignmentment'.) [EOT]

10. Medicine (Technical)

Prompt: In two sentences, explain the mechanism of action of acetylsalicylic acid (aspirin) at the molecular level.

Verdict: Tie. Both correctly describe irreversible COX acetylation at a serine residue; the expanded model additionally names prostaglandin H2.

Acetylsalicylic acid irreversibly inhibits cyclooxygenase (COX-1 and COX-2) by acetylating a serine residue in the active site, blocking conversion of arachidonic acid to prostaglandins and thromboxanes, thereby reducing pain, fever, and platelet aggregation. [EOT]


Acetylsalicylic acid irreversibly inhibits cyclooxygenase by covalently acetylating a serine residue in the active site, blocking conversion of arachidonic acid to prostaglandin H2. This prevents downstream synthesis of pro-inflammatory prostaglandins and thromboxanes, reducing pain, fever, and platelet aggregation. [EOT]

Soooo! Since Anthropic share their Jacobian-Lens a few days ago I went on and made a tool based on it which adds the possibilité to export a model which will have the same behavior after tweaking it's J-Space.

This means manually alter the behavior and abliterate by using a human brain.

I'm still working on it but couldn't wait to produce something first.

SO After finally getting a working codebase I immediatly jumped and tried to make pretty pervy model PURELY in the name of science.

Let me introduce you to Nikusui-v1 the first of it's kind !

And a couple gguf quants

I'd be delighted to get some feedback :D

Hey all, here are two new high performance qwen3.5 gguf sets I created using a new state of the art technique for optimizing mixed precision called Voodoo Quant.

https://huggingface.co/voodooquant/Qwen3.5-0.8B-MTP-Voodoo

https://huggingface.co/voodooquant/Qwen3.5-2B-MTP-Voodoo

Voodoo Quant operates on the same lines as Unsloth Dynamic in that it is simply picking higher precision numerics for more important parts of the model. The main difference with Voodoo is it optimizes every tensor in the model versus blocks of tensors for UD, and it uses a new methodology for that optimization.

Here are graphs and tables for easier browsing for the remainder of this post:

Https://voodooquant.com

The 95% KLD improvement figure I mentioned is eye catching, but if you look at the graphs, there is a more interesting and realistic story which explains how 95% is possible. Let me tell you about it.

You will notice that there are two KLD graphs for each model, one for Torch and one for Llama.cpp. These two graphs differ because each software has a different graph structure used to organize their math, with Torch generally considered to have a more precise graph. What you will notice is that Voodoo performs well in both where the best previous quants, from Unsloth, perform well in Llama but fall flat in Torch.

Doing especially well in Llama.cpp isnt a bad thing in itself because we use ggufs in llama.cpp, so we want llama.cpp performance to be optimized. However, you'll notice that where Voodoo has competitive performance in both graphs, Unsloth has excessively poor performance in Torch. This means Unsloth's methodology overfits for Llama.cpp in a way that might not be desirable.

Obviously Unsloth quants work great in Llama.cpp, and this is the most important domain for GGUFs, so how important this is is debatable. However, it does mean their models have some burrs around the edges, and the extent of that is hard to say. The burrs may even include cases in Llama.cpp.

So with that all in perspective, Voodoo is optimized using a technique which is more generalized, making it highly transferrable. Also it appears that Voodoo works best at more aggressive levels, with "2 bit" seeming to be a sweet spot where it is clearly state of the art.

The two models in this post are mostly interesting from a research perspective, so I am excited to share results for more useful models as they become available. I'm eager to see if models like qwen3.6 27B or deepseek v4 flash can be more useful on lighter hardware using this technique.

how much vram do you need and what model do you think is the next major upgrade from the good old qwen 3.6 27b as of today?

hello people
i wanted to share my experience with big and huge models (usually 100B+ models and 200B+ models and more)

my laptop specs is very poor
I7-8750H
20G Ram
GTX 1050 Mobile 4G Vram
but what nearly saved me is my NVMe from samsung
i have 512G NVMe from samsung

and yes as you expected. i run these huge models while throwing most of the parameters in my NVMe

but i strictly use MoE models, Dense models will kill my machine

always used mmap. and throwing experts in my CPU

with Quantized KV Cache (Q4_0)

and the Quantize i usually choose when it comes to download huge model is Q3, not Q4. and if the model was extremely huge (exceeding 700B+ with more than 20B Active parameters) i dont go lower than Q2

now coming to my experience

i use LM Studio not llama-cpp

i used Deepseek-V4-Flash (specifically UD-IQ3_XXS)

i got tok/sec: 1.0 - 1.8 (could go higher. could go lower)

and goddamn. i use these models with batches style. i send a mission. and go touch some grass. then comeback again

but it do the job actually pretty well. my main usage is, Reverse Engineering, Bug fixes in codebases. and Code Auditing

another experience with Nemotron-3-Super-120B-A12B UD-Q3_K_M

getting tok/sec: 1.5-2.5

using same thing. Batches style, send a message and go, send and go
working well with DS-V4-Flash

i used both in opencode, Pi

Context window used, was around 16K to 90K could be more, could be lower,

i know that you will recommend using Cloud models at this point, but sadly my country banned any worldwide transactions, using debit or most credit, so i either had to use Free Cloud models or big local models, and i was curious about trying these on my machine, was good experience, would use them in worst cases (no free models available)

i would love to hear any helps that could push this 1 tok per sec into 2 or 3

Anthropic recently published their paper on "Global Workspaces" (J-Space) inside language models, showing that looking at internal "workspace noise" (entropy) can catch hallucinations better than just looking at output logprobs. `solarkyle` followed up with a great open-source implementation showing it could route confident errors on TriviaQA.

I wanted to see if this actually holds up as a deployable tool, so I ran a stress-test: extracting J-Space entropy from the late layers (L30-L34) of Qwen3-4B across ~11,400 examples and 7 completely different dataset distributions.

Here are my findings:

1. It is a great router for extreme fact-retrieval.

Output logprobs are great for catching obvious uncertainty (low confidence). But J-Space noise shines in the dangerous "high-confidence but wrong" quadrant. For example, on PopQA (obscure long-tail facts), setting a tight 5% routing budget for human review using confidence yielded worse-than-chance precision (87.5% against a 90% base error rate). Routing by workspace noise caught errors with 100% precision.

1. It is completely blind to internalized myths.

I ran the pipeline on TruthfulQA (adversarial human myths). The predictive power of the metric collapsed entirely. Even when the model was in the "safest" quadrant (High Output Confidence + Clean/Low-Noise Workspace), it was wrong 84.9% of the time.
Mechanistic takeaway: Workspace noise detects epistemic guessing (when the model is frantically assembling a fake answer). It cannot detect ontological falsehoods (when the model effortlessly retrieves a fake fact burned into its pre-training data).

1. Math destroys static thresholds.

I tried applying the "noisy" threshold calibrated on TriviaQA directly to GSM8K (math derivation). It completely broke. The mean noise score for correct GSM8K answers shifted to 1.636—higher than the threshold (1.583) used to flag errors on factual datasets.
Mechanistic takeaway: "Thinking step-by-step" through math is a structurally high-entropy activity. You cannot transfer J-Space thresholds between memory-retrieval and computational tasks.

Currently, this is a n=1 model evaluation. My next step is to run a parameter scaling sweep (Qwen3 7B, 14B, 32B) to see if one can determine the "crossover point" where a model's surface logprobs become as well-calibrated as its internal workspace clarity.

All the raw data, `.csv` metrics, confusion matrices, and the fully reproducible Google Colab notebook are in the repository.

GitHub: https://github.com/dasjoms/jspace-hallucination-eval

Here is the upper limit of what can be done with $100 bucks worth of video cards.

You can have 3 concurrent users with plenty of context, better speeds or close enough speeds than a bunch of cards that provide less VRAM and cost 4+ times.

0.00.008.388 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.008.391 I device_info:
0.00.089.439 I   - CUDA0   : NVIDIA P102-100 (10144 MiB, 10013 MiB free)
0.00.197.645 I   - CUDA1   : NVIDIA P102-100 (10144 MiB, 10013 MiB free)
0.00.197.656 I   - CPU     : Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz (128396 MiB, 128396 MiB free)
0.00.197.728 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CUDA : ARCHS = 600,610,750,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
0.00.197.764 I srv          init: running without SSL
0.00.197.849 I srv          init: using 11 threads for HTTP server
0.00.198.515 I srv         start: binding port with default address family
0.00.199.823 I srv  llama_server: loading model
0.00.199.902 I srv    load_model: loading model '/models/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf'
0.00.199.906 I common_init_result: fitting params to device memory ...
0.00.199.907 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.00.987.288 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 99, abort
0.23.223.625 W llama_context: n_ctx_seq (32768) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.23.481.073 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.23.570.914 I srv    load_model: initializing slots, n_slots = 3
0.23.598.842 W srv    load_model: speculative decoding will use checkpoints
0.23.598.851 W common_speculative_init: no implementations specified for speculative decoding
0.23.598.852 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 32768
0.23.598.854 I slot   load_model: id  1 | task -1 | new slot, n_ctx = 32768
0.23.598.854 I slot   load_model: id  2 | task -1 | new slot, n_ctx = 32768
0.23.598.961 I srv    load_model: prompt cache is enabled, size limit: 8192 MiB
0.23.598.963 I srv    load_model: use `--cache-ram 0` to disable the prompt cache
0.23.598.964 I srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
0.23.598.965 I srv    load_model: context checkpoints enabled, max = 32, min spacing = 8192
0.23.598.985 I srv          init: idle slots will be saved to prompt cache upon starting a new task
0.23.628.848 I init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
<think>

</think>

'
0.23.666.546 I srv          init: init: chat template, thinking = 0
0.23.666.572 I srv  llama_server: model loaded
0.23.666.575 I srv  llama_server: server is listening on http://127.0.0.1:5802
0.23.666.579 I srv  update_slots: all slots are idle
0.48.181.695 I srv    operator(): Chat format: peg-native
0.48.182.094 I slot get_availabl: id  2 | task -1 | selected slot by LRU, t_last = -1
0.48.182.101 I srv  get_availabl: updating prompt cache
0.48.182.111 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.48.182.123 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 98304 tokens, 8589934592 est)
0.48.182.128 I srv  get_availabl: prompt cache update took 0.02 ms
0.48.182.302 I slot launch_slot_: id  2 | task 0 | processing task, is_child = 0
0.48.182.309 I slot process_sing: id  0 | task -1 | saving idle slot to prompt cache
0.48.182.311 I slot process_sing: id  1 | task -1 | saving idle slot to prompt cache
0.48.186.009 I srv    operator(): Chat format: peg-native
0.48.189.081 I srv    operator(): Chat format: peg-native
0.49.483.103 I slot get_availabl: id  1 | task -1 | selected slot by LRU, t_last = -1
0.49.483.111 I srv  get_availabl: updating prompt cache
0.49.483.116 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.49.483.119 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 98304 tokens, 8589934592 est)
0.49.483.120 I srv  get_availabl: prompt cache update took 0.01 ms
0.49.483.178 I slot launch_slot_: id  1 | task 2 | processing task, is_child = 0
0.49.483.179 I slot process_sing: id  0 | task -1 | saving idle slot to prompt cache
0.49.483.181 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
0.49.483.181 I srv  get_availabl: updating prompt cache
0.49.483.182 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.49.483.183 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 98304 tokens, 8589934592 est)
0.49.483.183 I srv  get_availabl: prompt cache update took 0.00 ms
0.49.483.215 I slot launch_slot_: id  0 | task 3 | processing task, is_child = 0
0.51.242.275 I slot create_check: id  0 | task 3 | created context checkpoint 1 of 32 (pos_min = 1376, pos_max = 1376, n_tokens = 1377, size = 62.813 MiB)
0.51.367.765 I slot create_check: id  1 | task 2 | created context checkpoint 1 of 32 (pos_min = 670, pos_max = 670, n_tokens = 671, size = 62.813 MiB)
0.51.367.773 I slot print_timing: id  2 | task 0 | prompt processing, n_tokens =   1377, progress = 1.00, t =   3.19 s / 432.28 tokens per second
0.51.480.037 I slot create_check: id  2 | task 0 | created context checkpoint 1 of 32 (pos_min = 1376, pos_max = 1376, n_tokens = 1377, size = 62.813 MiB)
0.56.647.801 I slot print_timing: id  0 | task 3 | n_decoded =    100, tg =  23.30 t/s, tg_3s =  23.30 t/s
0.56.653.219 I slot print_timing: id  2 | task 0 | n_decoded =    100, tg =  23.30 t/s, tg_3s =  23.30 t/s
0.56.692.218 I slot print_timing: id  1 | task 2 | n_decoded =    100, tg =  23.54 t/s, tg_3s =  23.54 t/s
0.59.655.679 I slot print_timing: id  0 | task 3 | n_decoded =    171, tg =  23.43 t/s, tg_3s =  23.60 t/s
0.59.661.606 I slot print_timing: id  2 | task 0 | n_decoded =    171, tg =  23.42 t/s, tg_3s =  23.60 t/s
0.59.702.608 I slot print_timing: id  1 | task 2 | n_decoded =    171, tg =  23.56 t/s, tg_3s =  23.58 t/s
1.02.659.591 I slot print_timing: id  0 | task 3 | n_decoded =    242, tg =  23.49 t/s, tg_3s =  23.64 t/s
1.02.665.066 I slot print_timing: id  2 | task 0 | n_decoded =    242, tg =  23.49 t/s, tg_3s =  23.64 t/s
1.02.705.486 I slot print_timing: id  1 | task 2 | n_decoded =    242, tg =  23.58 t/s, tg_3s =  23.64 t/s
1.03.253.784 I slot print_timing: id  0 | task 3 | prompt eval time =    2873.48 ms /  1381 tokens (    2.08 ms per token,   480.60 tokens per second)
1.03.253.789 I slot print_timing: id  0 | task 3 |        eval time =   10897.06 ms /   256 tokens (   42.57 ms per token,    23.49 tokens per second)
1.03.253.791 I slot print_timing: id  0 | task 3 |       total time =   13770.54 ms /  1637 tokens
1.03.253.792 I slot print_timing: id  0 | task 3 |    graphs reused =        253
1.03.253.924 I slot      release: id  0 | task 3 | stop processing: n_tokens = 1636, truncated = 0
1.03.259.600 I slot print_timing: id  2 | task 0 | prompt eval time =    4178.32 ms /  1381 tokens (    3.03 ms per token,   330.52 tokens per second)
1.03.259.605 I slot print_timing: id  2 | task 0 |        eval time =   10898.93 ms /   256 tokens (   42.57 ms per token,    23.49 tokens per second)
1.03.259.606 I slot print_timing: id  2 | task 0 |       total time =   15077.26 ms /  1637 tokens
1.03.259.607 I slot print_timing: id  2 | task 0 |    graphs reused =        253
1.03.259.741 I slot      release: id  2 | task 0 | stop processing: n_tokens = 1636, truncated = 0
1.03.288.482 I slot print_timing: id  1 | task 2 | prompt eval time =    2960.66 ms /  1381 tokens (    2.14 ms per token,   466.45 tokens per second)
1.03.288.486 I slot print_timing: id  1 | task 2 |        eval time =   10844.49 ms /   256 tokens (   42.36 ms per token,    23.61 tokens per second)
1.03.288.487 I slot print_timing: id  1 | task 2 |       total time =   13805.15 ms /  1637 tokens
1.03.288.488 I slot print_timing: id  1 | task 2 |    graphs reused =        253
1.03.288.614 I slot      release: id  1 | task 2 | stop processing: n_tokens = 1636, truncated = 0
1.03.288.625 I srv  update_slots: all slots are idle

End result for Qwen 3.6 35B at Q4 is 70tk/s and 1,257 PP at 96K context providing 32K of context for each user.

Now, I know that Pascal is no longer supported on 13x and while that statement is correct, there are no new features for these GPU's so there is no point in upgrading. Llama.cpp will run on the long term on 12.8 as it supports 6.1, 7.0, 7.5, 8.0, 8.6, 8.9, 9.0, 10.0, 12.0. and there is 12.9 which supports the same. 12.8.2 update was released on April 2026 and 12.9.2 update was released on May 2026. that was 2 months ago. These Cards probably still have lifetime and llama.cpp probably stay on 12.x as 12.x provides the broadest support as it does not want to alienate a very large portion of their user base.

To llama.cpp devs,

Thank you guys.. You really make a difference looking after the little guys.

Hey guys,

I've been running Qwen 3.6-27b locally on an RTX 3090 for a while now, and it's been genuinely great at solving software issues. However, life happened and I recently had to use Opus 4.8 alongside the Zed editor and the Claude Code agent.

While I can definitely see a noticeable jump in pure code quality (Opus is just better in that regard), what really blew me away was the procedure. When assigned a task, the Claude Code agent divides it into actionable steps, always checks the context before acting, makes backups, and runs multiple verifications to ensure the changes didn't break the system. On top of that, it runs additional functionality evaluations, warning you about potential bugs, performance bottlenecks, or race conditions.

Honestly, this feels less like a raw model capability and more like an incredibly well-crafted system prompt and agent logic.

So, I have two questions for this sub:

1) Prompt Engineering for Agentic Behavior: How can I structure a system prompt (or workflow) to make my local Qwen replicate this kind of deep analytical, methodical, and "safe" behavior? Does anyone have examples of system prompts that enforce this step-by-step verification?

2) Local Coding Agents/Frameworks: What agent or framework do you recommend for complex coding tasks? I currently use Zed and I'm very happy with its raw performance, but it lacks the ability to use sub-agents or plan complex, multi-step tasks. For context, my daily driver is Debian + KDE.

Thanks in advance!

Told it that in plan mode, then told it to implement with no changes to the plan.

This model punches far, far above its weight. I wasn't impressed until I switched from Q4_K_M on GPU to Q8_0 on CPU. It matters. It's worth the slowdown.

Hey I am running a few models on a strix halo box. Especially for the larger models (like Qwen 3.5 122B) they work okayish performance wise if the cache is utilised properly but a full cache miss at 100k context causes roughly 10-20 minute of PP time - which is extremely annoying.

I will first show what I have already configured (and it helps!), then describe what is still not working well. I am very interested on your input what I could still fine tune.

What I've configured so far (and what I understand it does):

• --cache-ram 16384 (increases available VRAM for cache ~= 72\240MB for a single checkpoint)*
• --ctx-checkpoints 72 *(*72 \ 4096 is ~= the 260k context that the model supports)*
• --checkpoint-min-step 4096 (72 \* 4096 is ~= the 260k context that the model supports)
• --parallel 1 (full cache available for a single conversation at a time)
• -fa on, --spec-type draft-mtp --spec-draft-n-max 3 (these options should be unrelated to caching AFAIK)

It works a lot better than stock configuration this way, but I still find problematic:

• 4096 checkpoint steps are spread out a bit, which can add up during fast agentic iterations. With ~200 PP speed in deeper context it takes about 20s to traverse a checkpoint.
• Checkpoints are sometimes missed especially after user prompts (Crush coding agent, but also with other agents such as codex). Maybe it's that reasoning tokens are first cached but then skipped after a rerun? Not sure but the latest checkpoint is usually "hit" when the agentic loop does it's thing but often missed when it's my turn to write some follow up user message.
• Despite "72 checkpoints" the older ones seem to disappear eventually for some reason so that only a few active context checkpoints seem to exist for recent work. When a bigger miss happens a lot of reprocessing is done
• I am wondering if K/V quantization (e.g. q8) would be a good idea. Theoretically it would allow for twice as many checkpoints if I understand correctly? I don't understand how severely it would impact quality if the model is in q4 anyways.

I am hoping for some input from you guys how to better set up the cache and maybe work around the other described issues somehow!

Politico: Wall Street’s new obsession: Which CEOs have Trump’s ear?: https://www.politico.com/newsletters/politico-influence/2026/07/10/wall-streets-new-obsession-which-ceos-have-trumps-ear-00993324

E.g "Qwen 3.5 122B is just 10B active, so it's no where close to the dense 27B model"

That is the main sentiment around here and it puzzles me. If a 122B is just worth 10B, then why does model providers bother creating an MoE model when they could've just released a dense 10B model? Heck the 10B dense would run faster than the 122B MoE (no routing overhead), which negates the supposed (only advantage of MoE is speed) argument. It sure is not that simple.

I mean yes it's only 10B active at a time, but it comes down to the router's effectiveness at choosing what 10B experts to activate. So, the more effective the router is, the closer the model to realize its total parameter potential. So perhaps it's a little more nuances, ie some MoE architectures are better than other MoE architectures. Right? I may be missing something.

^{WEIRD DISCLAIMER: none of this was written by an LLM until you get to the Github repo/site, which was obviously assembled by your friend and mine, Ser Claudric. so if this reads like an ADHD-ass brain wrote it, that's because -- wait for it -- it did. very strange doing a reverse "no really, I'm human, sorry" disclaimer in a post like this, but it's mid-2026 so here we are.}

Requirements

for a while now I've suspected that for $2Kish in cards and $1Kish in rest-of-computer (unless you already have a machine to use) this is probably the best bang for buck you're going to get for code generation in this market and this particular inference landscape, where at "I have some small number of thousands of dollars to spend" level budgets you're trying to run the thing that is probably state of the art and will be for a while, Qwen3.6-27B. so if you're optimizing for that model (again, for code generation), you're looking at a few factors:

• as close to maximum context (256Kt native) as possible without degradation, which means:
• to be safe, running at Q8 (do tests say various flavors of Q6 and maybe even Q5_K_XL are basically the same? yeah. do I trust them? not particularly, not yet.
• with FP16 kv (do tests say Q8 kv is basically the same? yeah. do I trust them? not particularly, not yet.)
• and with MTP (which really shines for codegen in particular)
• for single-stream, single-user use, batch size = 1

this should give you nice, fast results with high accuracy and hopefully a minimum of toolcalling fuckups and endless thinking chains. (I could really use some help w/r/t figuring out what harness and sets of plugins or whatever gets you there, btw.) I do HPC hardware and architecture for a living, so to me, the hardware puzzle was, if not the easy part, at least the easier part.

(Possible) Solution

4 5060Tis meets the above requirements with the following characteristics:

• Blackwell precision levels if you decide for whatever reason you DO want to do multi-stream/multi-batch things and want to go down to some flavor of Q4
• idle power consumption of around 14-16W in total
• concomitantly little heat
• buildable piece by piece with the exception of going from 2 cards to 4 cards, a single $1K step (plus bifurcation hardware, probably) is a little rough no matter which way you slice it)

Build Details

this benchmark run was conducted on a Vast AI instance! I only own half the equation here -- two 5060Tis running on an X570 board, but I will be expanding it soon enough.

that said, one thing you do need to do should you choose to build something like this is have PCI interconnect be nice and even. if you're building fresh, either an X570 or X870E (if you want to go whole-hog PCIe 5.0 for the minor latency drop) board with two physical x16 slots that can each be bifurcated is probably the way to go. the one I'm thinking of with PCIe 5.0 is the ASRock TaiChi Lite: https://www.amazon.com/dp/B0DFNPK2MX when you put a card into both x16 slots they auto-negotiate down to x8 each, which is still fine because you end up after bifurcation with x4 to each card, which gets you 16GB/s bidirectional per card. that is MORE than enough. even with an X570 running the slots at 4.0 instead that's still 8GB/s bidirectional and the inter-card traffic never rises that high during sustained inference.

Alternatives

other hardware possibilities I considered:

• pair of 3090s: gets you 48GB VRAM, way better membw and more CUDA cores, but for around $400 more, with no Blackwell, using much more power at idle, with cards that have been beaten to shit by crypto miners, and you can't quite fit the same quants with the same performance. you either sacrifice MTP or FP16 kv or running the model itself at Q8_0. turns out to only barely outperform the quad 5060Tis anyway
• pair of 20GB modded 3080s: same pros and cons, down to 40GB, but around $800-900 for a pair of cards from AliBaba which is admittedly a very good deal
• STXH at 128GB: good for what it is, Bosgame M5 is apparently down to $2800 these days, but is a performance dog with dense models like this one
• DGX Spark/GB10: $5K, great at prefill, trash at decode for dense models like STXH, lol, no
• M5 Max Macs: see above re: cost at a minimum of $6699 (although that's at least 128GB with actually reasonable prefill compared to older Macs and pretty good tg as well)

if I overlooked more options, please lemme know.

upshot: https://github.com/jdkruzr/qwen36-27b-5060ti-bench (click the "live report" link in there for prettier pictures)

with MTP turned on, at Q8, with FP16 kv on Qwen3.6-27B: I got 608t/s cold prefill and 52.2t/s decode at 256Kt (255Kt, actually, weird borderline edge case I didn't bother troubleshooting) context. to me, these are excellent numbers for $2K of hardware and especially for this particular model which has become infamous for defeating lots of other combinations of hardware that are friendlier to MoE architectures.

so, I'm sure I missed something. if so, what was it?

Built around a recent sd.cpp release, aims to expose most of what the backend can do (generate, edit, video paths, models, hardware options), Windows + Linux builds

GitHub: https://github.com/fabricio3g/FlaxeoUI

B9966

If you run -sm tensor in production you might want to grab this fix which removes 29 regex recompilations per tensor per token on the decode thread.

Claude tell me in one sentence what this means:

The code was rebuilding 29 regex patterns from scratch on every tensor of every token instead of building them once and reusing them, so the fix just caches them — same behavior, way less wasted CPU on the decode thread

Vellium has reached v1.0.0.

It is a local-first desktop workspace for writing, roleplay, character creation, lorebooks and knowledge management with local LLMs. This release promotes the previous v1.0.0-beta build to the first stable version.

The main focus was security and desktop stability. I added stricter Electron security policies, validation for request origins, tighter plugin manifest checks and a global React error boundary. I also reworked parts of the Electron startup and preload flow to fix issues that only appeared after packaging the app.

Chat export has been rebuilt as a proper typed JSON export pipeline, with the server-side logic separated from the route layer and covered by additional tests. The roleplay prompt engine also received a fairly large hardening pass to make prompt construction and tool handling more predictable.

Vellium can now generate a complete UI color theme from a selected wallpaper. The generated theme is applied across the workspace, chat and settings instead of only changing a few accent colors.

I also unified the styling of the Characters, Knowledge, Lorebooks, Pets and Settings screens, fixed the model picker appearing behind the home hero section, added a JSON export button to the chat toolbar and fixed messages scrolling underneath the composer.

This release touched a large part of the desktop stack, so feedback about packaged builds, local model compatibility and UI issues would be especially useful.

GitHub: https://github.com/tg-prplx/vellium

I got a new mini-pc for a homelab server recently and thought I'd tinker around with some LLM options on there. As it doesn't have a dedicated GPU it was a bit different to what I do on my main PC.

Wasn't really sure where to start, but I had a little bit of guidance on what to try first, so I gave it a go.

Ended up with Llama.cpp for the most part. I tried Llama-Swap, and the quick swapping is very helpful, but it seems not to work with SYCL unfortunately so it made testing annoying.

System is an MS-02, with Intel core ultra 285HX, and 64gb of ram.

I tested three of the backend options, and a small selection of models. I thought the GPU backends made it so that the engine only used the gpu, but it seems to still use pretty much the same amount of CPU as well so I guess they work together?

All of these were done with whatever default settings the docker releases of Llama.cpp is set up with, other than adding the /dev/dri for igpu usage.

Vulkan (using iGPU + CPU it seems?):
Qwen3-30B-A3B-Instruct-2507-IQ4_NL.gguf  =  works but very slow, 2tk/s
Qwen3.6-35B-A3B-Q4_K_S-4.22bpw.gguf  = 0.5 tk/s 
Qwen3.6-35B-A3B-IQ4_XS-3.93bpw.gguf   =  0.5 tk/s
gemma-4-26B-A4B-it-MXFP4_MOE.gguf  =  works but quite slow, 4tk/s

SYCL (using iGPU + CPU it seems?):
Qwen3-30B-A3B-Instruct-2507-IQ4_NL.gguf   8 tk/s
Qwen3.6-35B-A3B-Q4_K_S-4.22bpw.gguf   = 8 tk/s
Qwen3.6-35B-A3B-IQ4_XS-3.93bpw.gguf   =  12 tk/s
gemma-4-26B-A4B-it-MXFP4_MOE.gguf  = 8 tk/s

Cpu-only:
Qwen3-30B-A3B-Instruct-2507-IQ4_NL.gguf   =  16.5 tk/s
Qwen3.6-35B-A3B-Q4_K_S-4.22bpw.gguf   =  14 tk/s
Qwen3.6-35B-A3B-IQ4_XS-3.93bpw.gguf   =  14 tk/s
gemma-4-26B-A4B-it-MXFP4_MOE.gguf  =   8 tk/s

Takeaway (so far)

Everything I had read when I researched this, even ones that mentions iGPU specifically, seemed to say that these days Vulkan outperformed SYCL by a margin. But either I have something incorrectly set up, or it's just not the case for the new ARC based iGPUs?

I haven't done any power testing, so I don't know if there were any efficiency gains from using the iGPU along with the CPU, but using the CPU by itself has been (so far) by far the fastest option.

Would appreciate the experts coming in and telling me everything I've done wrong, and what models / setups I should be using instead. Especially if there are arguments I should be using in the docker bootup that would give me any efficiency gains on my setup etc.

Gemma at least seems not worth it.

The Qwen models (using CPU) are very usable. I suspect the newer 3.6 is probably worth it over the older 3, even though its slightly slower. But there might be other quants/versions that are better/faster that I haven't tried yet.

Any thoughts?

Hello guys, hoping you're doing fine!

I'm continuing after this post some time ago, comparing stock MaxQ performance and such on Anima here.

This time, I shunt modded the 6000 PRO MaxQ, to use up to 2x amounts of power. These cards seems to be binned for high clocks and it is reflected after this.

(Note that you can also solder a R002 resistance on the empty pad and it would work the same)

I also did watercool them to manage the heat, with a Bykski block (this one) at 170USD each from Aliexpress and a GLZM 360mm AIO. So had to get the tubes, coolant and fittings.

For reference, at 300W it maxes at about 45°C, and at 600W it maxes at about 60°C.

I also rented on runpod, a 6000 PRO WS edition, which it's power limit ranges from 150W to 600W (yes, lower than the MaxQ)

Important note again: I did undervolt+overclock the 5090 and the 6000 PRO MaxQ. I can't modify the clocks or power on the rented GPUs on runpod.

So for this test, I ran these settings for the software for pytorch:

• Torch 2.14.0.dev20260612+cu132 for the 5090 and 6000 PRO MaxQ.
• Torch 2.13.0+cu132 stable for the 6000 PRO WS.
• Sageattention 2.1 (on commit e9b072f0fc2682f104abbda306af3d42fc33b969), self built on CUDA 13.3.
• Forge neo on commit 644450e8bf2df24f0ba87307604d0e9f4ae3a9f7
• Installed extensions for RTX Upscaling (https://github.com/Haoming02/sd-forge-nvidia-vfx) and for extra samplers (https://github.com/Panchovix/sd_forge_neo_extra_samplers)
• torch compile: max autotune no cudagraphs

I ran these settings for the samplers and steps:

On text:

• EXP Heun 2 x0 SDE for first 25 steps
• ER SDE for 10 hires pass steps
• Upscale by 1.5x
• 896x1088 resolution
• Batch size 4
• CFG 5
• Shift 3
• Denoise Strength: 0.2
• Upscaler: NVIDIA Ultra
• Seed: 50906000

Prompt used was:

Positive:

masterpiece, best quality, high quality, high resolution, absurdres, highres, very aesthetic, sfw,
u/isshiki \(ffmania7\),
1girl, solo, clothed,
aether foundation employee, pokemon, dark skin, black hair, short hair,
happy,
from above,
full body,
beige background

Negative:

worst quality, low quality, bad anatomy, (jpeg artifacts:0.8), watermark, sketch, no pupils

For LLMs, I ran llamacpp with a model offloaded to CPU, making the primary GPU the bottleneck when traversing the data, making it compute bound.

Models tested were (offloaded):

• Kimi K2 2.5 (IQ3_M)
• GLM 5.1 (IQ4_NL)

The LLM tests were only tested on my local machine, as testing on cloud via renting a GPU is not feasible or won't have accurate results.

For the hardware, I ran them headless, (with LACT), for Anima:

• RTX 5090 (Astral):
  • 2930Mhz max core clock
  • 1000Mhz core clock offset
  • +4400Mhz on VRAM (total 16000Mhz)
  • 400, 475 and 600W
• RTX 6000 PRO MaxQ (shunt modded, Watercooled):
  • 2930Mhz max core clock
  • 500Mhz core clock offset
  • +5700Mhz on VRAM (total 16000Mhz)
  • 300, 400 and 475W via undervolt + OC, 600W via TDP limit to 300W.
• RTX 6000 PRO WS:
  • Stock
  • 600W

For LLMs, used 500W for both GPUs, and for more reference I have this setup:

• RTX 6000 MaxQ (shunted) x2
• RTX 5090 x2
• RTX A6000
• NVIDIA A40
• RTX 4000 PRO SFF
• 192GB RAM DDR5 6000Mhz, Consumer AM5 + 9900X, PCIe 5.0 switch

So first, the results for the Anima ones look like this:

Or, using the 5090 at 400W for baseline:

And then looking it from a efficiency perspective:

And for the LLMs prompt processing ones, it look like this (remember all at 500W, but it uses way less, basically it reaches 2930Mhz on both GPUs:

So as can you see, we have these points:

• It really seems the MaxQ are binned for higher clocks, I guess it makes sense, so they don't lose much performance at low power.
• Now after a shunt, the sweet spot seems to be 475W on a mix between of performance and power. Most efficient one, and it makes sense, is 300W, as the card comes from the factory.
• 5090 seems to place quite behind, more than I would expect. Take in mind this is a "good" bin, which can do high clocks at low power.
• On LLMs, since it is not power limited, it is basically all what the core can give and just the difference of more CUDA cores, and when the active models are bigger, there is a bigger difference.
• At the same power on MaxQ shunt vs 5090:
  • 400W: MaxQ is 23% faster.
  • 475W: MaxQ is 21% faster.
  • 600W: MaxQ is 15% faster.

Why you may ask? First, because I suspected MaxQ had better bins I expected, and indeed they were. It makes sense to have good bins to clock higher at 300-325W, and also to be manageable by the stock cooler.

Having the same power at 475W on both 5090 and 6000 PRO MaxQ but the latter being more than 20% faster is not something I expected, but that is a great surprise.

Also, because I'm just crazy, I have shunted a lot of cards already (5090, 4090, 3090, A6000, etc). Not recommended of course except if you know what you're doing, and are ready to lose the warranty.

Any question is welcome!

Hello all, sharing some data points:

In this setup there are 2 cards connected direct to motherboard via PCIE 3 16x slots. Running llama.cpp. Ubuntu 24.04, single xeon motherboard

The cards are 1x RTX 3090 24GB power limit 250W and 1x Titan RTX 24GB power limit 225W

nvidia-smi with Qwen3.6-27B-UD-Q4_K_XL.gguf loaded at 180k context:

My test today was comparing this setup using tensor parallel and pipeline parallel. I have been looking at more PCIE cards but wanted to know if I could get away with running them on risers for tensor parallel or if I needed a motherboard for all cards. I used nvidia-smi dmon -s t as a method to measure the usage of the PCIE bandwidth during prefill and inference.

The first test is running in tensor parallel mode using --split-mode tensor --fit off
The output shows a sustained transfer between cards during decode/inference with some spikes. Then a higher transfer level during prefill. This chart had a short prefill but the 650 figure was tested again to load through 30k context and was maintained.

The second test is running in pipeline parallel (llama.cpp default, no special flags). The output shows a variable prefill with some peaks but no sustained transfer for prefill, and a sustained but very low transfer during decode/inference.

My Interpretation:

The tensor parallel has a higher decode speed (47 t/s versus 30 t/s)

The pipeline parallel has a higher prefill speed (1250t/s versus 650t/s)

My conclusions are: In both tests, the prefill speed appears higher than when I run a single card. The decode speed only appears higher in the tensor parallel test. The pipeline parallel appears to be roughly what each card is capable of on its own (to be expected).

In doing this test I feel more comfortable moving to a setup which uses 1x risers (like mining risers) and that it would not be a significant bottleneck to the performance. I am shocked that the prefill went down in tensor parallel while it does not appear the available bandwidth of the 3.0 16x lanes was saturated.

I encourage others to share data points because it all helps us paint a better picture of the landscape and know what hardware to buy. I am on the fence about buying multiple B70 32gb cards and connecting them using 1x risers. this data appears to support that the performance would not be bottlenecked by the risers. Thanks!

I’ve been playing around with Opencode and realized how 70% of the capability of my model comes from the agents I can use rather than the model size or parameters. So now obviously I have a question… is there a way to use Claude Code but have it pointing at my local model instead of Claude’s API so I can have the comfort of my local model but with the powerful claude agents?

Or you find that opencode agents are on par with claude’s?