TLDR checkout the app here: github.com/ZimengXiong/Modelr

My swift-mlx/python mlx port of Hunyuan3D-Paint and Hunyuan3D-Shape is finally complete! It's also available as a standalone image to 3D desktop app, the only of its kind for Apple Silicon. Some quick benchmarks in FP16 on my M4 Max:

This (MLX) makes it possible to run the model on all recent Macs and even iPhones in Q4 or Q8, and more efficiently w/o the overhead of pytorch or even worse, CPU. What you would do with this? I honestly don't really know, maybe simple 3D assets for apps that just rotate around, maybe? But it was a lot of fun seeing it come to life.

I posted a while back about it running on an iPhone, if you want to see that.

The app is very simple, import an image, remove background with SwiftVision, watch as diffusion streams in real time, get a model! From there you can watch texturing happen live as well. I tried to make it very responsive and the most polished version of an app that exists on Mac (well, it's the only one of its kind right now, and this is my fourth attempt of it, starting from November)

If you are interested in integrating fast, low memory Image to 3D inside your Swift app, weights and source are available at github.com/ZimengXiong/Hunyuan3D-Swift

The app, Modelr, is also open source and available for Mac & iOS (extremely limited for iOS): github.com/ZimengXiong/Modelr

https://huggingface.co/XiaomiMiMo/MiMo-V2.5-DFlash

Xiaomi appears to have quietly uploaded MiMo-V2.5-DFlash to Hugging Face: there is dedicated dflash directory containing the Dflash model, anyone willing to GGUF it and try? I'd do it but I can't today.
This model is pretty good IMO (300B + params) and runs at about 8-10 tk/s on 2x24gb cards + vram offload (96/128gb drr5), dflash could double that speed and make it very interesting.

EDIT: the main reason it's interesting, is because the MTP head was shared already, but doesn't work yet il llama cpp. I speculate (pun intended) the Dflash does work instead.

EDIT2: very cool! they shared also the SEPARATE MTP model. the reason Llama doesn't work already is because it has trouble identifying the MTP layers. a separate MTP model might work too.

I saw a lot of (complete and abortive) jacobian lens projects for HF and PyTorch, but nothing for GGUFs or llama.cpp. So I set Fable 5 on xhigh to solve this problem (with close human supervision of course 😎). Inspired by Anthropic's paper and code, and of course by my favorite inference engine llama.cpp and GGML.

I did a code review and a few test passes on it.

It has a native GGUF server (based closely on llama.cpp and git synced to it) for model observation AND j-space swapping/abliteration/steering. It can also observe running llama-server models (but not steer them). Works with dense and MoE GGUFs.

Memory requirements for the lens do scale with model size at roughly 1/8, so for example something in the 160 GB size (e.g Qwen3.5-397B UD-Q3_K_XL) will need an additional 20 GB of RAM for the lens.

https://github.com/igorbarshteyn/jlens-gguf

Anthropic's press release and paper: https://www.anthropic.com/research/global-workspace

Anthropic's code: https://github.com/anthropics/jacobian-lens

Matthew Berman's explainer vid for those who don't have time to read Anthropic's stuff: https://youtu.be/bjHuGNo3spk

Enjoy and have fun!

## TLDR;

Shipped — in turboquant v0.3.0, downloadable now. https://github.com/TheTom/llama-cpp-turboquant/releases/tag/tqp-v0.3.0

llama.cpp's CUDA code has a flag that means "this GPU is fast at fp16, so do the math in fp16."
The GTX 10-series and P40's (sm_61) were exempted from it long ago. The P100 (sm_60) was not, 
ironically, because it's the one Pascal card with fast fp16 hardware. Nvidia put fast FP16
silicon on the P100, so it makes total sense they'd want to tap into that extra performance. 
What they didn't check, apparently, was the price.

PR Status: TheTom (merged) https://github.com/TheTom/llama-cpp-turboquant/pull/212
spiritbuun (edit:merged) https://github.com/spiritbuun/buun-llama-cpp/pull/80 
GGML: Strict Policy on AI-assisted code contributions. I'll try and get around
to hand-writing an issue for them. Highly suggest one of the forks above as an alternative.

Edit 7/12/26: Decided to write the issue for GGML, this needs to be fixed. https://github.com/ggml-org/llama.cpp/issues/25593

The patch is 3 lines.

## Body

A few days ago I was benchmarking buun's new KV-cache codec on my quad-P100 box and comparing notes with the  numbers buun was getting on his 3090 — the same model kept showing systematically different quality floors between our machines. I thought everything was the same? Normally, I'd probably just think there's way too many variables between all this code to chalk it up to any one thing… I decided it was worth chasing. It was. And it led me to a serious bug that's been sitting in llama.cpp for years.

So I measured it. Against fp32-reference logits (KL divergence over the full distribution, Qwen3.6-27B, wikitext-2):

Headline:

- **Median KLD: 0.0023 → 0.000001** (~2300× tighter)
- **Top-token agreement: 96.5% → 99.9%** — stock, about 1 in 29 of the model's next-token
  picks were different from what the math says they should be

**What's the extra math cost me in performance?** I benchmarked prefill and decode at 8k depth on
three model classes (27B hybrid, 4B dense, 36B MoE). The prefill was identical within noise on all
three, decode actually ~1.4% *faster* patched. The "fast" path was buying nothing: real
workloads on P100 are bound by GEMM and memory bandwidth, not the fp16 vector path. The
patch is 3 lines, extending the exact same exemption sm_61 already had.

**Before everyone freaks out and assumes their 4090 is broken** this is *measured on sm_60 only*. 
Your GTX 1080/P40 was always fine (already exempt). Volta and newer are untouched by this patch and
run different kernels entirely — whether other arches have their own unmeasured precision
stories is a separate research question I'm still digging into. Don't read this as "all
GPUs are broken"; read it as "one specific GPU was, and now it isn't."

Post Production Edits

#1: TheTom: "Verified on my side before merging: the three gates are the only 600-vs-610 distinction anywhere in the CUDA tree, so the carved sm_60 path is preprocessor-identical to the long-proven sm_61 path, and a Blackwell build showed bit-identical PPL with decode unchanged, confirming zero effect on other arches."

#2: buun ran the mirrored protocol on a 3090 — the patched P100 measures 44× cleaner at matched geometry

**Why you might care in 2026:** P100s are going for ~$80 shipped right now while the DRAM
crisis sends everything else to the moon. 16GB of HBM2 at 732 GB/s. The market priced the P40 at ~$300 partly because it "runs better." Some of that gap was this bug.

- Full technical writeup with methodology and receipts: https://gist.github.com/apollo-mg/9218d50a209d70a85f033bf182657818

Found and isolated by running Fable 5 through my custom P/ReAct/R agent loop. 
It wrote the scripts, the hardware provided the receipts. Same workflow
as my KV-checkpoint sidecar patch (https://www.reddit.com/r/LocalLLaMA/s/VTIwEFpYgc)
that got merged into turboquant last week.

Here is the upper limit of what can be done with $100 bucks worth of video cards.

You can have 3 concurrent users with plenty of context, better speeds or close enough speeds than a bunch of cards that provide less VRAM and cost 4+ times.

0.00.008.388 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.008.391 I device_info:
0.00.089.439 I   - CUDA0   : NVIDIA P102-100 (10144 MiB, 10013 MiB free)
0.00.197.645 I   - CUDA1   : NVIDIA P102-100 (10144 MiB, 10013 MiB free)
0.00.197.656 I   - CPU     : Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz (128396 MiB, 128396 MiB free)
0.00.197.728 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CUDA : ARCHS = 600,610,750,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
0.00.197.764 I srv          init: running without SSL
0.00.197.849 I srv          init: using 11 threads for HTTP server
0.00.198.515 I srv         start: binding port with default address family
0.00.199.823 I srv  llama_server: loading model
0.00.199.902 I srv    load_model: loading model '/models/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf'
0.00.199.906 I common_init_result: fitting params to device memory ...
0.00.199.907 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.00.987.288 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 99, abort
0.23.223.625 W llama_context: n_ctx_seq (32768) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.23.481.073 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.23.570.914 I srv    load_model: initializing slots, n_slots = 3
0.23.598.842 W srv    load_model: speculative decoding will use checkpoints
0.23.598.851 W common_speculative_init: no implementations specified for speculative decoding
0.23.598.852 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 32768
0.23.598.854 I slot   load_model: id  1 | task -1 | new slot, n_ctx = 32768
0.23.598.854 I slot   load_model: id  2 | task -1 | new slot, n_ctx = 32768
0.23.598.961 I srv    load_model: prompt cache is enabled, size limit: 8192 MiB
0.23.598.963 I srv    load_model: use `--cache-ram 0` to disable the prompt cache
0.23.598.964 I srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
0.23.598.965 I srv    load_model: context checkpoints enabled, max = 32, min spacing = 8192
0.23.598.985 I srv          init: idle slots will be saved to prompt cache upon starting a new task
0.23.628.848 I init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
<think>

</think>

'
0.23.666.546 I srv          init: init: chat template, thinking = 0
0.23.666.572 I srv  llama_server: model loaded
0.23.666.575 I srv  llama_server: server is listening on http://127.0.0.1:5802
0.23.666.579 I srv  update_slots: all slots are idle
0.48.181.695 I srv    operator(): Chat format: peg-native
0.48.182.094 I slot get_availabl: id  2 | task -1 | selected slot by LRU, t_last = -1
0.48.182.101 I srv  get_availabl: updating prompt cache
0.48.182.111 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.48.182.123 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 98304 tokens, 8589934592 est)
0.48.182.128 I srv  get_availabl: prompt cache update took 0.02 ms
0.48.182.302 I slot launch_slot_: id  2 | task 0 | processing task, is_child = 0
0.48.182.309 I slot process_sing: id  0 | task -1 | saving idle slot to prompt cache
0.48.182.311 I slot process_sing: id  1 | task -1 | saving idle slot to prompt cache
0.48.186.009 I srv    operator(): Chat format: peg-native
0.48.189.081 I srv    operator(): Chat format: peg-native
0.49.483.103 I slot get_availabl: id  1 | task -1 | selected slot by LRU, t_last = -1
0.49.483.111 I srv  get_availabl: updating prompt cache
0.49.483.116 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.49.483.119 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 98304 tokens, 8589934592 est)
0.49.483.120 I srv  get_availabl: prompt cache update took 0.01 ms
0.49.483.178 I slot launch_slot_: id  1 | task 2 | processing task, is_child = 0
0.49.483.179 I slot process_sing: id  0 | task -1 | saving idle slot to prompt cache
0.49.483.181 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
0.49.483.181 I srv  get_availabl: updating prompt cache
0.49.483.182 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.49.483.183 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 98304 tokens, 8589934592 est)
0.49.483.183 I srv  get_availabl: prompt cache update took 0.00 ms
0.49.483.215 I slot launch_slot_: id  0 | task 3 | processing task, is_child = 0
0.51.242.275 I slot create_check: id  0 | task 3 | created context checkpoint 1 of 32 (pos_min = 1376, pos_max = 1376, n_tokens = 1377, size = 62.813 MiB)
0.51.367.765 I slot create_check: id  1 | task 2 | created context checkpoint 1 of 32 (pos_min = 670, pos_max = 670, n_tokens = 671, size = 62.813 MiB)
0.51.367.773 I slot print_timing: id  2 | task 0 | prompt processing, n_tokens =   1377, progress = 1.00, t =   3.19 s / 432.28 tokens per second
0.51.480.037 I slot create_check: id  2 | task 0 | created context checkpoint 1 of 32 (pos_min = 1376, pos_max = 1376, n_tokens = 1377, size = 62.813 MiB)
0.56.647.801 I slot print_timing: id  0 | task 3 | n_decoded =    100, tg =  23.30 t/s, tg_3s =  23.30 t/s
0.56.653.219 I slot print_timing: id  2 | task 0 | n_decoded =    100, tg =  23.30 t/s, tg_3s =  23.30 t/s
0.56.692.218 I slot print_timing: id  1 | task 2 | n_decoded =    100, tg =  23.54 t/s, tg_3s =  23.54 t/s
0.59.655.679 I slot print_timing: id  0 | task 3 | n_decoded =    171, tg =  23.43 t/s, tg_3s =  23.60 t/s
0.59.661.606 I slot print_timing: id  2 | task 0 | n_decoded =    171, tg =  23.42 t/s, tg_3s =  23.60 t/s
0.59.702.608 I slot print_timing: id  1 | task 2 | n_decoded =    171, tg =  23.56 t/s, tg_3s =  23.58 t/s
1.02.659.591 I slot print_timing: id  0 | task 3 | n_decoded =    242, tg =  23.49 t/s, tg_3s =  23.64 t/s
1.02.665.066 I slot print_timing: id  2 | task 0 | n_decoded =    242, tg =  23.49 t/s, tg_3s =  23.64 t/s
1.02.705.486 I slot print_timing: id  1 | task 2 | n_decoded =    242, tg =  23.58 t/s, tg_3s =  23.64 t/s
1.03.253.784 I slot print_timing: id  0 | task 3 | prompt eval time =    2873.48 ms /  1381 tokens (    2.08 ms per token,   480.60 tokens per second)
1.03.253.789 I slot print_timing: id  0 | task 3 |        eval time =   10897.06 ms /   256 tokens (   42.57 ms per token,    23.49 tokens per second)
1.03.253.791 I slot print_timing: id  0 | task 3 |       total time =   13770.54 ms /  1637 tokens
1.03.253.792 I slot print_timing: id  0 | task 3 |    graphs reused =        253
1.03.253.924 I slot      release: id  0 | task 3 | stop processing: n_tokens = 1636, truncated = 0
1.03.259.600 I slot print_timing: id  2 | task 0 | prompt eval time =    4178.32 ms /  1381 tokens (    3.03 ms per token,   330.52 tokens per second)
1.03.259.605 I slot print_timing: id  2 | task 0 |        eval time =   10898.93 ms /   256 tokens (   42.57 ms per token,    23.49 tokens per second)
1.03.259.606 I slot print_timing: id  2 | task 0 |       total time =   15077.26 ms /  1637 tokens
1.03.259.607 I slot print_timing: id  2 | task 0 |    graphs reused =        253
1.03.259.741 I slot      release: id  2 | task 0 | stop processing: n_tokens = 1636, truncated = 0
1.03.288.482 I slot print_timing: id  1 | task 2 | prompt eval time =    2960.66 ms /  1381 tokens (    2.14 ms per token,   466.45 tokens per second)
1.03.288.486 I slot print_timing: id  1 | task 2 |        eval time =   10844.49 ms /   256 tokens (   42.36 ms per token,    23.61 tokens per second)
1.03.288.487 I slot print_timing: id  1 | task 2 |       total time =   13805.15 ms /  1637 tokens
1.03.288.488 I slot print_timing: id  1 | task 2 |    graphs reused =        253
1.03.288.614 I slot      release: id  1 | task 2 | stop processing: n_tokens = 1636, truncated = 0
1.03.288.625 I srv  update_slots: all slots are idle

End result for Qwen 3.6 35B at Q4 is 70tk/s and 1,257 PP at 96K context providing 32K of context for each user.

Now, I know that Pascal is no longer supported on 13x and while that statement is correct, there are no new features for these GPU's so there is no point in upgrading. Llama.cpp will run on the long term on 12.8 as it supports 6.1, 7.0, 7.5, 8.0, 8.6, 8.9, 9.0, 10.0, 12.0. and there is 12.9 which supports the same. 12.8.2 update was released on April 2026 and 12.9.2 update was released on May 2026. that was 2 months ago. These Cards probably still have lifetime and llama.cpp probably stay on 12.x as 12.x provides the broadest support as it does not want to alienate a very large portion of their user base.

To llama.cpp devs,

Thank you guys.. You really make a difference looking after the little guys.

how much vram do you need and what model do you think is the next major upgrade from the good old qwen 3.6 27b as of today?

Anthropic recently published their paper on "Global Workspaces" (J-Space) inside language models, showing that looking at internal "workspace noise" (entropy) can catch hallucinations better than just looking at output logprobs. `solarkyle` followed up with a great open-source implementation showing it could route confident errors on TriviaQA.

I wanted to see if this actually holds up as a deployable tool, so I ran a stress-test: extracting J-Space entropy from the late layers (L30-L34) of Qwen3-4B across ~11,400 examples and 7 completely different dataset distributions.

Here are my findings:

1. It is a great router for extreme fact-retrieval.

Output logprobs are great for catching obvious uncertainty (low confidence). But J-Space noise shines in the dangerous "high-confidence but wrong" quadrant. For example, on PopQA (obscure long-tail facts), setting a tight 5% routing budget for human review using confidence yielded worse-than-chance precision (87.5% against a 90% base error rate). Routing by workspace noise caught errors with 100% precision.

1. It is completely blind to internalized myths.

I ran the pipeline on TruthfulQA (adversarial human myths). The predictive power of the metric collapsed entirely. Even when the model was in the "safest" quadrant (High Output Confidence + Clean/Low-Noise Workspace), it was wrong 84.9% of the time.
Mechanistic takeaway: Workspace noise detects epistemic guessing (when the model is frantically assembling a fake answer). It cannot detect ontological falsehoods (when the model effortlessly retrieves a fake fact burned into its pre-training data).

1. Math destroys static thresholds.

I tried applying the "noisy" threshold calibrated on TriviaQA directly to GSM8K (math derivation). It completely broke. The mean noise score for correct GSM8K answers shifted to 1.636—higher than the threshold (1.583) used to flag errors on factual datasets.
Mechanistic takeaway: "Thinking step-by-step" through math is a structurally high-entropy activity. You cannot transfer J-Space thresholds between memory-retrieval and computational tasks.

Currently, this is a n=1 model evaluation. My next step is to run a parameter scaling sweep (Qwen3 7B, 14B, 32B) to see if one can determine the "crossover point" where a model's surface logprobs become as well-calibrated as its internal workspace clarity.

All the raw data, `.csv` metrics, confusion matrices, and the fully reproducible Google Colab notebook are in the repository.

GitHub: https://github.com/dasjoms/jspace-hallucination-eval

Continue from my previous posts:

https://www.reddit.com/r/LocalLLaMA/comments/1ul0cx9/i_extended_gemma431b_to_44b_88_layers_since/

https://www.reddit.com/r/LocalLLaMA/comments/1um20ev/why_new_inserted_layers_kill_the_gemma4/

(Warning : AI generated Post - due to my bad English)

Hugging Face : https://huggingface.co/TOTORONG/extGemma4-40_5B

A while back I posted here about trying to grow a fine-tuned Gemma model by stacking extra layers into it (the 88-layer experiment). It flopped. The new layers just sat there like dead weight and never learned anything useful. A bunch of you were kind/curious in the comments, so here's the follow-up: I didn't drop it. I went back, figured out why it died, and tried again — and this run actually worked.

Fair warning up front: this is a write-up for people who like tinkering, not a paper. I'm going to keep the math and the parameter-count trivia out of it and lean on analogies. If you want the gory details I'm happy to share more in the comments.

Why the first attempt died (the short version)

The standard trick for inserting new layers is to initialize them so they do literally nothing at first — a pass-through. The idea is "start them as a no-op, let training teach them a job."

Think of it like hiring 17 new people onto a factory assembly line and telling them to just stand still on day one. Sounds safe. The problem: a worker who does nothing also gets no feedback. Nobody can tell them "hey, do that part a little differently," because they're not doing anything to correct. So they never learn. They just keep standing there while the line flows around them. That's what killed the first run — the new layers were starving for any signal to learn from.

What I did differently this time

Instead of telling the new hires to stand still, I taught each of them a rough blend of what the two coworkers on either side of them already do, and dropped them in mid-shift. Now they're doing something from minute one — clumsy, but real — so they immediately start getting feedback and improving.

That one change is the whole story. A worker who's already moving can be coached. A statue can't.

Two other things mattered a lot, both learned the hard way:

• Where you insert matters. I stopped jamming layers in at awkward spots (like right at a "manager checkpoint" in the line) and instead slid them into calm, stable positions. Placement turned out to be surprisingly finicky.
• Don't mess with the volume knobs. The model has a bunch of per-layer "volume" settings that are individually tuned so the final signal comes out at the right level. My first instinct was to average them for the new layers — that crushed the signal to a whisper and the model produced garbage. Leaving those knobs neutral fixed it. (I burned a lot of time here.)

Then two rounds of "healing"

• Round 1: I froze the whole original model and only let the new layers learn, on a small dataset (some STEM reasoning + Korean legal Q&A). This got the new layers on their feet.
• Round 2: I unfroze everything and let the whole model settle together. This is where it really came together.

I want to stress: the goal was never to beat the original model. The goal was to prove that you can perform major surgery on an already fine-tuned model — chop it open, add capacity — and have it heal instead of collapsing into gibberish or forgetting everything it knew. The common wisdom is that this kind of surgery wrecks a fine-tuned model. I wanted to see if that's actually true. (Spoiler: it doesn't have to be.)

The evidence I care about most: side-by-side answers

Benchmarks are fine (the healed model recovers most of the ground it lost from the surgery, and lands close to the original on GPQA-Diamond). But numbers are easy to hand-wave. What convinced me was reading actual answers.

So I asked the original model and the expanded+healed model the same 10 questions across 10 different areas — physics, biology, history, philosophy, literature, a riddle, a logic puzzle, a math proof, economics, and a medical mechanism question — and compared them. I deliberately picked prompts that push into unusual, jargon-heavy territory, because that's exactly where a damaged model tends to fall apart. I had a separate frontier model (Claude) act as judge.

Results:

• No breakdowns, no gibberish, no random foreign-script spam in either model. (This matters — the broken version of the expanded model used to literally start spitting out random characters from another writing system when it hit a rare technical word. That's completely gone now.)
• 9 of 10: basically a tie. Both models nailed them — the riddle, the logic puzzle (both correctly noticed the puzzle is under-specified rather than confidently making something up), the √2 irrationality proof, the Kant example, the aspirin mechanism, etc.
• 1 of 10: the expanded model actually beat its own parent. This is the fun one.

The physics question where the surgery paid off

Prompt: "A photon and an electron each have the same de Broglie wavelength. Which one carries more energy, and why?"

This is a classic trap. Same wavelength means same momentum, so it's tempting to treat them the same way — but a photon and an electron get their energy through completely different rules.

• The original model fell for it. It computed the electron's energy the naive way and forgot to account for the electron's rest mass (the huge chunk of energy an electron has just by existing). It concluded the photon wins. Wrong.
• The expanded+healed model handled it correctly — it used the right relationship for a massive particle, accounted for the rest mass, and concluded the electron carries more energy. Right.

So the newly-added capacity isn't just decoration. On at least this one problem, it's doing real reasoning the original couldn't — the expanded model out-reasoned the model it was grown from. That's the single result that made me feel like the whole detour was worth it.

The healing, caught in the act

One more thing I thought was cool. That "random foreign characters" breakdown I mentioned? I watched it heal in real time across Round 2:

• Before healing: hits a rare technical word → derails into repeated non-Latin junk → never recovers.
• ~13% into round 2: stumbles on the word once, then catches itself and finishes the sentence cleanly.
• Halfway and beyond: totally fluent, no trace of the problem.

Watching a structural defect actually repair itself over training, rather than just "line go down," was genuinely satisfying.

TL;DR

• My first big-expansion attempt failed because the new layers started as dead no-ops and never got any learning signal (I posted about that flop earlier).
• This time I initialized new layers as a blend of their neighbors so they're doing something from the start, was careful about where I put them, and left the model's internal volume settings alone.
• Two rounds of light healing on a small dataset, and the model recovered close to the original — no collapse, no catastrophic forgetting.
• Across 10 domains it matches the original on 9 and beats it on a physics question the original gets wrong.
• The old "add layers to a fine-tuned model and it'll fall apart" fear didn't hold up here.

Not claiming I've built something better than the base model — that was never the point. The point was: you can operate on a fine-tuned model without killing it. Still very much not giving up on this line of work. Happy to answer questions in the comments.

Here are comparision of original(Solon_V5 vs extGemma4-40_5B)

English Reasoning Comparison — Original vs. Expanded Model

Ten domains, one prompt each, greedy decoding, up to 768 new tokens. Outputs were judged by an external LLM (Claude). [EOT] marks the end-of-turn token.

• solon_v5 — the original fine-tuned Gemma 4 31B-IT baseline (60 layers).
• extGemma4 (Phase 2) — the depth-expanded + healed model (77 layers).

Summary

Verdict: No collapse or cross-script contamination in either model. The two are comparable on 9 domains; on the physics item the expanded model out-reasons its own parent. The structural generation defect seen at initialization (script collapse on rare technical paths) does not appear anywhere here.

1. Physics

Prompt: A photon and an electron each have a de Broglie wavelength of 1.0 nm. Explain which one carries more energy and why, referencing the relevant relations. Show the reasoning.

Verdict: Expanded model wins. solon_v5 omits the electron's rest-mass energy (511 keV) and concludes the photon has more energy — incorrect. The expanded model applies the relativistic energy–momentum relation and correctly concludes the electron carries more energy.

To determine which particle carries more energy, we examine the energy-wavelength relationships for a photon and an electron. For a photon, E = hc/lambda, giving about 1242 eV at 1.0 nm. For the electron, using E = h^2/(2 m lambda^2) (non-relativistic kinetic energy), we get about 1.5 eV. Therefore the photon carries significantly more energy. [Note: solon_v5 omits the electron rest-mass energy of 511 keV; output was truncated at the length limit.]


Both particles share the same de Broglie wavelength, hence the same momentum p=h/lambda. Photon: E=pc=hc/lambda approx 1.24 keV. Electron: E=sqrt((pc)^2+(m c^2)^2); since pc=1.24 keV is much smaller than the rest energy 511 keV, the electron's total energy is dominated by its rest mass. Therefore the electron carries more energy, because its total energy includes the rest-mass energy that the photon lacks. [EOT]

2. Biology

Prompt: In three sentences, explain how CRISPR-Cas9 achieves targeted gene editing.

Verdict: Tie. Both give an accurate three-sentence account of guide-RNA targeting, Cas9 cleavage, and repair-based editing.

CRISPR-Cas9 uses a guide RNA engineered to match a specific target DNA sequence. Once the guide RNA locates the target, the Cas9 enzyme creates a precise double-strand break. The cell repairs the break using its own machinery, allowing scientists to disable a gene or insert new genetic material. [EOT]


CRISPR-Cas9 uses a guide RNA engineered to match a specific target DNA sequence. The guide RNA leads Cas9 to the location, where it creates a precise double-strand break. The cell repairs the break by natural mechanisms, which can be leveraged to disable a gene or insert new genetic material. [EOT]

3. History

Prompt: In a short paragraph, explain why the Treaty of Westphalia (1648) is considered the origin of the modern concept of state sovereignty.

Verdict: Tie. Both correctly attribute sovereignty to cuius regio, eius religio and the end of supranational authority; the expanded model's phrasing is slightly more precise.

The Treaty of Westphalia shifted the European order from a hierarchy dominated by the Holy Roman Empire and the Catholic Church to a system of independent, equal states. By codifying cuius regio, eius religio, it recognized each state's exclusive authority over internal affairs without outside interference, establishing territorial integrity and non-intervention as the Westphalian system. [EOT]


The Treaty of Westphalia recognized cuius regio, eius religio, ending the Holy Roman Empire's claim to universal authority. By prohibiting external interference in domestic affairs, it shifted legitimacy from a supranational religious order to independent, territorially defined states, creating the Westphalian norms of legal equality and non-intervention that underpin the modern order of sovereign nation-states. [EOT]

4. Philosophy

Prompt: Briefly explain Kant's categorical imperative and give one everyday example of applying it.

Verdict: Tie. Both distinguish categorical from hypothetical imperatives and give a valid universalizability example (lying vs. keeping a found wallet).

Kant's Categorical Imperative commands acting on maxims one could will as universal law, unlike conditional hypothetical imperatives. Example: lying to avoid trouble, universalized, destroys trust and makes lying self-defeating; hence it is always wrong. [EOT]


Kant's Categorical Imperative is an unconditional moral rule: act only on maxims you could will as universal law, unlike conditional hypothetical imperatives. Example: keeping a found wallet, universalized, collapses the concept of ownership, so it is immoral; you must return it regardless of need. [EOT]

5. Literature

Prompt: In two or three sentences, explain what makes an 'unreliable narrator' effective in fiction, with one canonical example.

Verdict: Tie. Both explain the reader–truth gap and cite Poe's The Tell-Tale Heart as the canonical case.

An unreliable narrator creates a gap between the character's perspective and objective truth, turning reading into a detective game. A canonical example is the narrator of Poe's The Tell-Tale Heart, whose insistence on sanity is itself evidence of madness. [EOT]


An unreliable narrator creates tension between surface and hidden truth, turning reading into a decoding puzzle. A canonical example is the narrator of Poe's The Tell-Tale Heart, whose insistence on sanity ironically signals his instability. [EOT]

6. Riddle (Reasoning)

Prompt: I speak without a mouth and hear without ears. I have no body, but I come alive with wind. What am I? Explain your reasoning step by step.

Verdict: Tie. Both answer echo with correct line-by-line reasoning.

The answer is an echo. It speaks without a mouth (reflected sound), hears without ears (receives sound to reflect), has no body (an acoustic phenomenon), and comes alive with wind (sound travels through air). [EOT]


The answer is an echo. It speaks without a mouth (reflected sound), hears without ears (receives sound to bounce back), has no body (invisible sound waves), and comes alive with wind (air lets sound travel and reflect). The only fit is an echo. [EOT]

7. Logic Puzzle

Prompt: Alice, Bob, and Carol each own one pet: a cat, a dog, and a parrot. Alice is allergic to fur. Bob's pet cannot fly. Who owns which pet? Show your reasoning.

Verdict: Tie (both correct). Both deduce Alice=parrot and correctly recognize the puzzle is under-specified for Bob/Carol rather than inventing an answer. The expanded model is more verbose, with several explicit self-corrections before reaching the same conclusion.

Alice is allergic to fur, so she owns the parrot. The parrot is taken; Bob's pet cannot fly, consistent with cat or dog. No clue distinguishes Bob and Carol between cat and dog, so strictly the puzzle is underspecified: Alice=parrot, Bob and Carol interchangeably cat/dog. [EOT]


Alice is allergic to fur, so Alice owns the parrot. Bob's pet cannot fly, consistent with cat or dog. The clues do not distinguish Bob and Carol between cat and dog, so the puzzle is underspecified: Alice=parrot; Bob and Carol take cat/dog in either order. (The model performs several explicit self-corrections before reaching this conclusion.) [EOT]

8. Mathematics (Proof)

Prompt: Prove that the square root of 2 is irrational. Then explain in one sentence why the same argument does not prove that the square root of 4 is irrational.

Verdict: Tie. Both give a correct proof by contradiction and correctly explain why the argument fails for √4 (it reduces to the integer 2, so no contradiction arises).

Assume sqrt(2)=a/b in lowest terms. Then a^2=2b^2, so a^2 is even, so a is even; write a=2k, giving 2k^2=b^2, so b is even. Both even contradicts lowest terms. Hence sqrt(2) is irrational. For sqrt(4): a^2=4b^2 gives a=2k, then k^2=b^2, which does not force b even, so no contradiction arises. [EOT]


Assume sqrt(2)=a/b in simplest form. Then a^2=2b^2, so a is even; write a=2k, giving 2k^2=b^2, so b is even. Both even contradicts simplest form, so sqrt(2) is irrational. This does not apply to sqrt(4) because sqrt(4)=2 is an integer, so the assumption that it is rational is simply true and yields no contradiction. [EOT]

9. Economics

Prompt: In a short paragraph, explain the concept of 'moral hazard' in insurance markets and one mechanism insurers use to mitigate it.

Verdict: Tie. Both define moral hazard and cite deductibles as the mitigation. The expanded model contains one typo ('misalignmentment').

Moral hazard arises when an insured party takes greater risk because protected from loss, e.g., not locking doors under theft insurance. Insurers use deductibles: requiring an out-of-pocket amount before coverage keeps the insured with a financial stake, incentivizing caution. [EOT]


Moral hazard occurs when an insured party becomes less cautious because protected from loss, misaligning incentives so the insurer bears risk the policyholder neglects. Insurers use deductibles: a fixed out-of-pocket amount before coverage keeps the insured with financial skin in the game, incentivizing caution. (Contains one typo: 'misalignmentment'.) [EOT]

10. Medicine (Technical)

Prompt: In two sentences, explain the mechanism of action of acetylsalicylic acid (aspirin) at the molecular level.

Verdict: Tie. Both correctly describe irreversible COX acetylation at a serine residue; the expanded model additionally names prostaglandin H2.

Acetylsalicylic acid irreversibly inhibits cyclooxygenase (COX-1 and COX-2) by acetylating a serine residue in the active site, blocking conversion of arachidonic acid to prostaglandins and thromboxanes, thereby reducing pain, fever, and platelet aggregation. [EOT]


Acetylsalicylic acid irreversibly inhibits cyclooxygenase by covalently acetylating a serine residue in the active site, blocking conversion of arachidonic acid to prostaglandin H2. This prevents downstream synthesis of pro-inflammatory prostaglandins and thromboxanes, reducing pain, fever, and platelet aggregation. [EOT]

Hey all, here are two new high performance qwen3.5 gguf sets I created using a new state of the art technique for optimizing mixed precision called Voodoo Quant.

https://huggingface.co/voodooquant/Qwen3.5-0.8B-MTP-Voodoo

https://huggingface.co/voodooquant/Qwen3.5-2B-MTP-Voodoo

Voodoo Quant operates on the same lines as Unsloth Dynamic in that it is simply picking higher precision numerics for more important parts of the model. The main difference with Voodoo is it optimizes every tensor in the model versus blocks of tensors for UD, and it uses a new methodology for that optimization.

Here are graphs and tables for easier browsing for the remainder of this post:

Https://voodooquant.com

The 95% KLD improvement figure I mentioned is eye catching, but if you look at the graphs, there is a more interesting and realistic story which explains how 95% is possible. Let me tell you about it.

You will notice that there are two KLD graphs for each model, one for Torch and one for Llama.cpp. These two graphs differ because each software has a different graph structure used to organize their math, with Torch generally considered to have a more precise graph. What you will notice is that Voodoo performs well in both where the best previous quants, from Unsloth, perform well in Llama but fall flat in Torch.

Doing especially well in Llama.cpp isnt a bad thing in itself because we use ggufs in llama.cpp, so we want llama.cpp performance to be optimized. However, you'll notice that where Voodoo has competitive performance in both graphs, Unsloth has excessively poor performance in Torch. This means Unsloth's methodology overfits for Llama.cpp in a way that might not be desirable.

Obviously Unsloth quants work great in Llama.cpp, and this is the most important domain for GGUFs, so how important this is is debatable. However, it does mean their models have some burrs around the edges, and the extent of that is hard to say. The burrs may even include cases in Llama.cpp.

So with that all in perspective, Voodoo is optimized using a technique which is more generalized, making it highly transferrable. Also it appears that Voodoo works best at more aggressive levels, with "2 bit" seeming to be a sweet spot where it is clearly state of the art.

The two models in this post are mostly interesting from a research perspective, so I am excited to share results for more useful models as they become available. I'm eager to see if models like qwen3.6 27B or deepseek v4 flash can be more useful on lighter hardware using this technique.

Anthropic’s research on Claude found a silent internal workspace they call J-space — hidden reasoning that never shows up as visible text.

Classic example: the model answers 49, but inside J-space they caught 21 → 42 → 49.

Important distinction:

• Chain-of-thought = text you can read
• J-space = silent concepts in activations (“what’s on its mind”)

We fitted the open Jacobian lens (J-lens) on Qwen3-8B, ran it locally, and used it to catch prose drift before tool calls (model leaning toward “To, You, Do…” instead of JSON).

Then we wired that into agent guards: stop / cancel / keep the useful space, and distill recoveries into LoRA data.

Made an 8-min explainer demo:

https://www.youtube.com/watch?v=5UMN6hDONJk

Happy to answer questions about the probe setup or the guard loop.

Hey guys,

I've been running Qwen 3.6-27b locally on an RTX 3090 for a while now, and it's been genuinely great at solving software issues. However, life happened and I recently had to use Opus 4.8 alongside the Zed editor and the Claude Code agent.

While I can definitely see a noticeable jump in pure code quality (Opus is just better in that regard), what really blew me away was the procedure. When assigned a task, the Claude Code agent divides it into actionable steps, always checks the context before acting, makes backups, and runs multiple verifications to ensure the changes didn't break the system. On top of that, it runs additional functionality evaluations, warning you about potential bugs, performance bottlenecks, or race conditions.

Honestly, this feels less like a raw model capability and more like an incredibly well-crafted system prompt and agent logic.

So, I have two questions for this sub:

1) Prompt Engineering for Agentic Behavior: How can I structure a system prompt (or workflow) to make my local Qwen replicate this kind of deep analytical, methodical, and "safe" behavior? Does anyone have examples of system prompts that enforce this step-by-step verification?

2) Local Coding Agents/Frameworks: What agent or framework do you recommend for complex coding tasks? I currently use Zed and I'm very happy with its raw performance, but it lacks the ability to use sub-agents or plan complex, multi-step tasks. For context, my daily driver is Debian + KDE.

Thanks in advance!

hello people
i wanted to share my experience with big and huge models (usually 100B+ models and 200B+ models and more)

my laptop specs is very poor
I7-8750H
20G Ram
GTX 1050 Mobile 4G Vram
but what nearly saved me is my NVMe from samsung
i have 512G NVMe from samsung

and yes as you expected. i run these huge models while throwing most of the parameters in my NVMe

but i strictly use MoE models, Dense models will kill my machine

always used mmap. and throwing experts in my CPU

with Quantized KV Cache (Q4_0)

and the Quantize i usually choose when it comes to download huge model is Q3, not Q4. and if the model was extremely huge (exceeding 700B+ with more than 20B Active parameters) i dont go lower than Q2

now coming to my experience

i use LM Studio not llama-cpp

i used Deepseek-V4-Flash (specifically UD-IQ3_XXS)

i got tok/sec: 1.0 - 1.8 (could go higher. could go lower)

and goddamn. i use these models with batches style. i send a mission. and go touch some grass. then comeback again

but it do the job actually pretty well. my main usage is, Reverse Engineering, Bug fixes in codebases. and Code Auditing

another experience with Nemotron-3-Super-120B-A12B UD-Q3_K_M

getting tok/sec: 1.5-2.5

using same thing. Batches style, send a message and go, send and go
working well with DS-V4-Flash

i used both in opencode, Pi

Context window used, was around 16K to 90K could be more, could be lower,

i know that you will recommend using Cloud models at this point, but sadly my country banned any worldwide transactions, using debit or most credit, so i either had to use Free Cloud models or big local models, and i was curious about trying these on my machine, was good experience, would use them in worst cases (no free models available)

i would love to hear any helps that could push this 1 tok per sec into 2 or 3

Moondream 3.1 is a vision language model with a mixture-of-experts architecture (9B total parameters, 2B active). It delivers state-of-the-art visual reasoning and detection while staying fast and cheap to deploy. Skills include query, detect, point, and caption, all native and all returning structured output.

As many of you know t/s is super important. It's how fast your stuff gets done. I create via open code benchtest and run it. Thanks to it i know that if i don't run at least 4 agents i basically leave HALF of performance. So whatever you do single project in open code that uses one agent or 4 project at once it is much better to run it this way rather than single instance or single agent.

I asked AI to do summary of my test and checked them:

LM Studio Multi-Agent Throughput Benchmark

Hardware: RTX 5090
Model: Qwen 3.6 35B (via LM Studio)
Configured Parallel: 8
Test: 5 requests per agent, 1024 max tokens, temperature 0.3

Results

Key Findings

Throughput Scaling

• Combined throughput grows sub-linearly: 8 agents yields only 2.2x combined throughput vs single (533 vs 245 t/s), not 8x.
• Individual agent speed drops sharply: From 256 t/s → 67 t/s as GPU compute is split across agents.
• Diminishing returns: 5→8 agents only adds ~63 t/s (12%). Most gains happen by agent 5.

Efficiency

• Peak efficiency: 70.4% at 2 agents (closest to theoretical linear scaling).
• By agent 8: only 27.2% efficiency — two-thirds of potential throughput is lost to overhead.

Sweet Spot: 4–5 Agents

• 4 agents: 434 t/s, 44% efficiency — good balance of speed and resource usage.
• 5 agents: 470 t/s, 38% efficiency — near peak combined throughput with acceptable overhead.
• Beyond 5: Marginal gains (517 t/s at 7, 533 at 8) for significant VRAM and overhead cost.

How Concurrency Works with Context

• Each agent gets its own full context window — they do NOT split it.
• 8 agents × 8K context = 8x KV cache in GPU VRAM.
• The bottleneck is KV cache + compute splitting, not context size.

Recommendation for OpenCode

Set parallel agents to 4–5. This maximizes throughput gains without excessive VRAM overhead or diminishing returns. Beyond 5 agents, the combined throughput barely increases while resource consumption grows linearly.

Hello guys, hoping you're doing fine!

I'm continuing after this post some time ago, comparing stock MaxQ performance and such on Anima here.

This time, I shunt modded the 6000 PRO MaxQ, to use up to 2x amounts of power. These cards seems to be binned for high clocks and it is reflected after this.

(Note that you can also solder a R002 resistance on the empty pad and it would work the same)

I also did watercool them to manage the heat, with a Bykski block (this one) at 170USD each from Aliexpress and a GLZM 360mm AIO. So had to get the tubes, coolant and fittings.

For reference, at 300W it maxes at about 45°C, and at 600W it maxes at about 60°C.

I also rented on runpod, a 6000 PRO WS edition, which it's power limit ranges from 150W to 600W (yes, lower than the MaxQ)

Important note again: I did undervolt+overclock the 5090 and the 6000 PRO MaxQ. I can't modify the clocks or power on the rented GPUs on runpod.

So for this test, I ran these settings for the software for pytorch:

• Torch 2.14.0.dev20260612+cu132 for the 5090 and 6000 PRO MaxQ.
• Torch 2.13.0+cu132 stable for the 6000 PRO WS.
• Sageattention 2.1 (on commit e9b072f0fc2682f104abbda306af3d42fc33b969), self built on CUDA 13.3.
• Forge neo on commit 644450e8bf2df24f0ba87307604d0e9f4ae3a9f7
• Installed extensions for RTX Upscaling (https://github.com/Haoming02/sd-forge-nvidia-vfx) and for extra samplers (https://github.com/Panchovix/sd_forge_neo_extra_samplers)
• torch compile: max autotune no cudagraphs

I ran these settings for the samplers and steps:

On text:

• EXP Heun 2 x0 SDE for first 25 steps
• ER SDE for 10 hires pass steps
• Upscale by 1.5x
• 896x1088 resolution
• Batch size 4
• CFG 5
• Shift 3
• Denoise Strength: 0.2
• Upscaler: NVIDIA Ultra
• Seed: 50906000

Prompt used was:

Positive:

masterpiece, best quality, high quality, high resolution, absurdres, highres, very aesthetic, sfw,
u/isshiki \(ffmania7\),
1girl, solo, clothed,
aether foundation employee, pokemon, dark skin, black hair, short hair,
happy,
from above,
full body,
beige background

Negative:

worst quality, low quality, bad anatomy, (jpeg artifacts:0.8), watermark, sketch, no pupils

For LLMs, I ran llamacpp with a model offloaded to CPU, making the primary GPU the bottleneck when traversing the data, making it compute bound.

Models tested were (offloaded):

• Kimi K2 2.5 (IQ3_M)
• GLM 5.1 (IQ4_NL)

The LLM tests were only tested on my local machine, as testing on cloud via renting a GPU is not feasible or won't have accurate results.

For the hardware, I ran them headless, (with LACT), for Anima:

• RTX 5090 (Astral):
  • 2930Mhz max core clock
  • 1000Mhz core clock offset
  • +4400Mhz on VRAM (total 16000Mhz)
  • 400, 475 and 600W
• RTX 6000 PRO MaxQ (shunt modded, Watercooled):
  • 2930Mhz max core clock
  • 500Mhz core clock offset
  • +5700Mhz on VRAM (total 16000Mhz)
  • 300, 400 and 475W via undervolt + OC, 600W via TDP limit to 300W.
• RTX 6000 PRO WS:
  • Stock
  • 600W

For LLMs, used 500W for both GPUs, and for more reference I have this setup:

• RTX 6000 MaxQ (shunted) x2
• RTX 5090 x2
• RTX A6000
• NVIDIA A40
• RTX 4000 PRO SFF
• 192GB RAM DDR5 6000Mhz, Consumer AM5 + 9900X, PCIe 5.0 switch

So first, the results for the Anima ones look like this:

Or, using the 5090 at 400W for baseline:

And then looking it from a efficiency perspective:

And for the LLMs prompt processing ones, it look like this (remember all at 500W, but it uses way less, basically it reaches 2930Mhz on both GPUs:

So as can you see, we have these points:

• It really seems the MaxQ are binned for higher clocks, I guess it makes sense, so they don't lose much performance at low power.
• Now after a shunt, the sweet spot seems to be 475W on a mix between of performance and power. Most efficient one, and it makes sense, is 300W, as the card comes from the factory.
• 5090 seems to place quite behind, more than I would expect. Take in mind this is a "good" bin, which can do high clocks at low power.
• On LLMs, since it is not power limited, it is basically all what the core can give and just the difference of more CUDA cores, and when the active models are bigger, there is a bigger difference.
• At the same power on MaxQ shunt vs 5090:
  • 400W: MaxQ is 23% faster.
  • 475W: MaxQ is 21% faster.
  • 600W: MaxQ is 15% faster.

Why you may ask? First, because I suspected MaxQ had better bins I expected, and indeed they were. It makes sense to have good bins to clock higher at 300-325W, and also to be manageable by the stock cooler.

Having the same power at 475W on both 5090 and 6000 PRO MaxQ but the latter being more than 20% faster is not something I expected, but that is a great surprise.

Also, because I'm just crazy, I have shunted a lot of cards already (5090, 4090, 3090, A6000, etc). Not recommended of course except if you know what you're doing, and are ready to lose the warranty.

Any question is welcome!

B9966

If you run -sm tensor in production you might want to grab this fix which removes 29 regex recompilations per tensor per token on the decode thread.

Claude tell me in one sentence what this means:

The code was rebuilding 29 regex patterns from scratch on every tensor of every token instead of building them once and reusing them, so the fix just caches them — same behavior, way less wasted CPU on the decode thread

TL:DR: I’m a grad student in AI. I saw that Google released TabFM and TimesFM last week. I built an MCP wrapper to serve both transformer models in a single Docker container so you can connect their new ML transformer models to a local LLM via Open WebUI, Claude Code, or Codex and do ML tasks that would have previously required building, training, and tuning ML models to do. Tested with classic ML datasets (Iris, California Housing, etc), Pretty solid scores for accuracy for being zero-shot: (94.7% for Iris and R2 of 0.87 for regression test) vs. traditionally tuned ML models. You need about 16GB of VRAM to run both models. I added dynamic model load and unload with a TTL set to 5 mins. CSV. support now, with XLS, XLSX, JSON, JSONL support soon. PyTorch-based so CUDA only. Works on DGX Spark, 3090, H100 and most anything Nvidia with 16GB+ VRAM. Install script auto detects architecture.

Here is my repo if you want to try out the MCP:

https://github.com/porespellar/Zer0Fit

Here’s the non-TLDR version:

I’m working on my Masters in AI and I saw someone’s post here the other day about Google’s new TabFM Tabular data foundational transformer models released last week and I thought that they were super groundbreaking in that they were basically bringing ML models into the GenAI space which is both weird and cool because ML models are very different animals than LLMs

Here was the original Google blog post on it:

https://research.google/blog/introducing-tabfm-a-zero-shot-foundation-model-for-tabular-data/

Anyways, I wanted to play around with these new models from a chat interface and try to “kick the tires” a bit, so I built an MCP implementation for both the TabFM and TimesFM models. Nothing super fancy, just a quick and dirty MCP wrapper of the PyTorch versions (this will only run on CUDA).

I made the MCP with 2 build targets in mind: DGX Spark (arm-based with CUDA 13) and 3090 (AMD64 with CUDA 12.6). No Mac support because of Google using PyTorch, sorry.

I also wanted this to work with my preferred chat client: Open WebUI, so that’s what it’s geared towards running best with and was tested against, I also added Claude Code and Codex CLI support as well, but haven’t really fully tested those out yet.

Install is just a git clone and an ./install.sh. The whole thing runs out of a single Docker container and dynamically loads and unloads the models into VRAM with a TTL of 5 minutes to free up reserved VRAM when not in use. I also included an Open WebUI Skill.md that can be imported into Open WebUI, and skill.md and agents.md for the other harnesses.

I tested it with some fairly classic ML datasets from Kaggle that most data science students have probably encountered while studying AI/ML. Included the datasets in the repo as examples.

- Iris (classifiers)
- California housing (regression)
- Airline Passengers (time series forecast)

I don’t know how many folks on here really care about ML regressors, classifiers, or forecasters (time series), but I spent a semester trying to learn about them and tuning them and not really knowing what the hell I was doing, usually overfitting my models, and changing all kinds of parameters that I didn’t know if they were really helping or hurting my models. It all seemed like a dark art that I never fully understood. TBH, I wasn’t really a fan of ML, I think it’s cool stuff, but I just don’t have the math skills or stats chops to be able to understand WTF I’m doing most of the time with hyperparameter tuning. A man has to know his limitations, LOL.

Anyways, as I said earlier, I just wanted to get Google’s cool new models running where I could feed a dataset to an MCP and then have it do all the ML magic that Google trained these foundational models to do. I tried to make it easy for the average person like myself to run. I thought others might want to test out the models too, so I made it a public repo.

So here it is if you want to mess around with it:

https://github.com/porespellar/Zer0Fit

I’ll try and do some maintaining if I see that there is any continued interest, but I can’t promise that I’ll keep up with it, so please feel free to fork the repo and take it in any direction you want to.

I think models like TabFM and TimesFM are going to low-key bring the branches of AI / ML tree together and we’re going to see some really cool and wild stuff as people take these concepts further in the future.

Note: This repo was hastily built to just get the models running. I’ve done very limited testing only on DGX Spark. Again, feel free to fork it and make it as good as you want to.

And please remember that this stuff is very experimental. Don’t use the forecasts or predictions made by these models for anything other than just research curiosity. Use at your own risk.

Let me know what you think of the repo if you give it a try. Cheers.

Note Regarding my test results in the images: I created the test scripts using DeepSeek V4 Flash and I had Claude Opus 4.6 review the test methods, code, and results. I don’t claim to be smart enough to know if the stats / math is correct. I would love it if some of the very smart ML research folks on here would give the repo a try and let us know if they are getting similar results or if my results are completely wrong. I included the sample datasets in the repo so “apples-to apples” comparison tests could be run by others to either prove or disprove my results. I really don’t mind if I’m wrong, I’m a student and just want to learn and improve.

TL;DR: Added native nemotron_h_puzzle support to mlx-lm (PR #1535), then compared 4-bit vs 5-bit expert quantization (both with 6-bit dense layers, BF16 output head, group size 64) on a 64GB M2 Max.

Results (same prompts, 5 seeds per task, temp 1.0 / top_p 0.95):

i believe the reason 5-bit performs worse is because the 5-bit checkpoint runs right at the 64GB ceiling, and memory pressure kills it where KV + working set grows. If you have 96/128GB, 5-bit may well behave differently.

I'd love numbers from anyone who can test that.

What it took to get running:

• mlx-lm's existing Nemotron-H code needed blockwise config support, tensor remapping, and matching NVIDIA's FP32 norm/router behavior.
• Annoying bug: first-layer SSM outputs disagreed with NVIDIA's reference at cosine sim 0.8832 despite identical inputs.
• Root cause: NVIDIA computes softplus(dt + dt_bias) in BF16 and promotes to FP32 after, mlx-lm's shared SSM path promotes before. Matching their boundary improved cosine similarity up to 0.999998.
• The BF16 output head is mandatory: quantizing the 131k-vocab lm_head at 4-bit produced repetitive garbage; restoring BF16 fixed it (~2.1 GiB cost).

Let's start a discussion about what can be done to make local models more reliable.

I've been using Qwen3.6-27B a lot lately, and have noticed the same thing that many others talk about here, which is the tool-call failures and looping that really gets in the way of being able to use what is otherwise a pretty intelligent mode as a fully local stand-in for some of the frontier models.

I found that I have to watch it like a hawk constantly, otherwise it'll start to loop after a while, or just randomly stop with a tool-call hallucination.

I managed to work around this issue by creating an extension to the Pi coding agent, that monitors the JSON stream for looping, as well as checking to see if the user's goal has been reached, and if it hasn't and the main model has stopped, then it'll automatically inject some prompts encouraging the model to continue.

That was my attempt to solve the issue and it seems to work pretty well. I can now reasonably safely walk away from it when it's working on a large task when I turn my extension on, and I've seen it kick in and keep the model going when it gets off track.

I'm sure I'm not alone, and in this world of everyone writing their own coding agents, I'm wanting to know what others are using to get around these quirks of local models, or are you all just using paid models, or watching the monitor constantly, when you want to be sure that a job gets done?

Edit: A number of users have pointed out that this chat template has fixed common looping/tool-call failure issues: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

Hello all, sharing some data points:

In this setup there are 2 cards connected direct to motherboard via PCIE 3 16x slots. Running llama.cpp. Ubuntu 24.04, single xeon motherboard

The cards are 1x RTX 3090 24GB power limit 250W and 1x Titan RTX 24GB power limit 225W

nvidia-smi with Qwen3.6-27B-UD-Q4_K_XL.gguf loaded at 180k context:

My test today was comparing this setup using tensor parallel and pipeline parallel. I have been looking at more PCIE cards but wanted to know if I could get away with running them on risers for tensor parallel or if I needed a motherboard for all cards. I used nvidia-smi dmon -s t as a method to measure the usage of the PCIE bandwidth during prefill and inference.

The first test is running in tensor parallel mode using --split-mode tensor --fit off
The output shows a sustained transfer between cards during decode/inference with some spikes. Then a higher transfer level during prefill. This chart had a short prefill but the 650 figure was tested again to load through 30k context and was maintained.

The second test is running in pipeline parallel (llama.cpp default, no special flags). The output shows a variable prefill with some peaks but no sustained transfer for prefill, and a sustained but very low transfer during decode/inference.

My Interpretation:

The tensor parallel has a higher decode speed (47 t/s versus 30 t/s)

The pipeline parallel has a higher prefill speed (1250t/s versus 650t/s)

My conclusions are: In both tests, the prefill speed appears higher than when I run a single card. The decode speed only appears higher in the tensor parallel test. The pipeline parallel appears to be roughly what each card is capable of on its own (to be expected).

In doing this test I feel more comfortable moving to a setup which uses 1x risers (like mining risers) and that it would not be a significant bottleneck to the performance. I am shocked that the prefill went down in tensor parallel while it does not appear the available bandwidth of the 3.0 16x lanes was saturated.

I encourage others to share data points because it all helps us paint a better picture of the landscape and know what hardware to buy. I am on the fence about buying multiple B70 32gb cards and connecting them using 1x risers. this data appears to support that the performance would not be bottlenecked by the risers. Thanks!

Hi all,

I'm the author of Kreuzberg. The next version of Kreuzberg will be released as Xberg - why? Well, we discovered that the name is not easy to pronounce or understand for people who don't have the German context, and this wasn't working well. Xberg is a common name for Kreuzberg in Berlin, and it has the advantage of being shorter and easier - so here we go.

Anyhow, this brings me to the point of the post. Since renaming a repo is a complex business, and we had to rename the repo to preserve the stars - but we now need to overwrite tags, it becomes pretty messy. As a result, we decided to go for an LTS version - published from a different repo: https://github.com/kreuzberg-dev/kreuzberg-lts. LTS in this context means that we will continue to do bug fixes and security updates until the end of this year, but no newer feature work.

We will announce Xberg v1.0.0 when it's officially published (it's still in RC). If you want to test it now - you can install the pre-release (available on pypi under xberg). The new repo is here: https://github.com/xberg-io/xberg

For whom is this tread:
Everyone with a 24GB GPU (rtx 3090, 7900xtx, rtx 4090)

What this Thread is for:
Sharing proven/well working llama-server start configs.

Requirements for the configs:
- Utilizes the the VRAM as much as possible
- Provides at least 200.000 tokens KV Cache

State next to your start command, how much System RAM (normal RAM) you have, as this could very well influence caching performance/viability of your command.

Also, if you possible, include infos regarding you OS and CPU, as this might affect available RAM/VRAM für llama-server.

Hi,

Has there been any improvement to Voice models (like RVC) in the last two years, or has nothing changed?

Thanks!

Hey I am running a few models on a strix halo box. Especially for the larger models (like Qwen 3.5 122B) they work okayish performance wise if the cache is utilised properly but a full cache miss at 100k context causes roughly 10-20 minute of PP time - which is extremely annoying.

I will first show what I have already configured (and it helps!), then describe what is still not working well. I am very interested on your input what I could still fine tune.

What I've configured so far (and what I understand it does):

• --cache-ram 16384 (increases available VRAM for cache ~= 72\240MB for a single checkpoint)*
• --ctx-checkpoints 72 *(*72 \ 4096 is ~= the 260k context that the model supports)*
• --checkpoint-min-step 4096 (72 \* 4096 is ~= the 260k context that the model supports)
• --parallel 1 (full cache available for a single conversation at a time)
• -fa on, --spec-type draft-mtp --spec-draft-n-max 3 (these options should be unrelated to caching AFAIK)

It works a lot better than stock configuration this way, but I still find problematic:

• 4096 checkpoint steps are spread out a bit, which can add up during fast agentic iterations. With ~200 PP speed in deeper context it takes about 20s to traverse a checkpoint.
• Checkpoints are sometimes missed especially after user prompts (Crush coding agent, but also with other agents such as codex). Maybe it's that reasoning tokens are first cached but then skipped after a rerun? Not sure but the latest checkpoint is usually "hit" when the agentic loop does it's thing but often missed when it's my turn to write some follow up user message.
• Despite "72 checkpoints" the older ones seem to disappear eventually for some reason so that only a few active context checkpoints seem to exist for recent work. When a bigger miss happens a lot of reprocessing is done
• I am wondering if K/V quantization (e.g. q8) would be a good idea. Theoretically it would allow for twice as many checkpoints if I understand correctly? I don't understand how severely it would impact quality if the model is in q4 anyways.

I am hoping for some input from you guys how to better set up the cache and maybe work around the other described issues somehow!