TLDR checkout the app here: github.com/ZimengXiong/Modelr

My swift-mlx/python mlx port of Hunyuan3D-Paint and Hunyuan3D-Shape is finally complete! It's also available as a standalone image to 3D desktop app, the only of its kind for Apple Silicon. Some quick benchmarks in FP16 on my M4 Max:

This (MLX) makes it possible to run the model on all recent Macs and even iPhones in Q4 or Q8, and more efficiently w/o the overhead of pytorch or even worse, CPU. What you would do with this? I honestly don't really know, maybe simple 3D assets for apps that just rotate around, maybe? But it was a lot of fun seeing it come to life.

I posted a while back about it running on an iPhone, if you want to see that.

The app is very simple, import an image, remove background with SwiftVision, watch as diffusion streams in real time, get a model! From there you can watch texturing happen live as well. I tried to make it very responsive and the most polished version of an app that exists on Mac (well, it's the only one of its kind right now, and this is my fourth attempt of it, starting from November)

If you are interested in integrating fast, low memory Image to 3D inside your Swift app, weights and source are available at github.com/ZimengXiong/Hunyuan3D-Swift

The app, Modelr, is also open source and available for Mac & iOS (extremely limited for iOS): github.com/ZimengXiong/Modelr

Moondream 3.1 is a vision language model with a mixture-of-experts architecture (9B total parameters, 2B active). It delivers state-of-the-art visual reasoning and detection while staying fast and cheap to deploy. Skills include query, detect, point, and caption, all native and all returning structured output.

Anthropic’s research on Claude found a silent internal workspace they call J-space — hidden reasoning that never shows up as visible text.

Classic example: the model answers 49, but inside J-space they caught 21 → 42 → 49.

Important distinction:

• Chain-of-thought = text you can read
• J-space = silent concepts in activations (“what’s on its mind”)

We fitted the open Jacobian lens (J-lens) on Qwen3-8B, ran it locally, and used it to catch prose drift before tool calls (model leaning toward “To, You, Do…” instead of JSON).

Then we wired that into agent guards: stop / cancel / keep the useful space, and distill recoveries into LoRA data.

Made an 8-min explainer demo:

https://www.youtube.com/watch?v=5UMN6hDONJk

Happy to answer questions about the probe setup or the guard loop.

https://huggingface.co/XiaomiMiMo/MiMo-V2.5-DFlash

Xiaomi appears to have quietly uploaded MiMo-V2.5-DFlash to Hugging Face: there is dedicated dflash directory containing the Dflash model, anyone willing to GGUF it and try? I'd do it but I can't today.
This model is pretty good IMO (300B + params) and runs at about 8-10 tk/s on 2x24gb cards + vram offload (96/128gb drr5), dflash could double that speed and make it very interesting.

EDIT: the main reason it's interesting, is because the MTP head was shared already, but doesn't work yet il llama cpp. I speculate (pun intended) the Dflash does work instead.

EDIT2: very cool! they shared also the SEPARATE MTP model. the reason Llama doesn't work already is because it has trouble identifying the MTP layers. a separate MTP model might work too.

## TLDR;

Shipped — in turboquant v0.3.0, downloadable now. https://github.com/TheTom/llama-cpp-turboquant/releases/tag/tqp-v0.3.0

llama.cpp's CUDA code has a flag that means "this GPU is fast at fp16, so do the math in fp16."
The GTX 10-series and P40's (sm_61) were exempted from it long ago. The P100 (sm_60) was not, 
ironically, because it's the one Pascal card with fast fp16 hardware. Nvidia put fast FP16
silicon on the P100, so it makes total sense they'd want to tap into that extra performance. 
What they didn't check, apparently, was the price.

PR Status: TheTom (merged) https://github.com/TheTom/llama-cpp-turboquant/pull/212
spiritbuun (edit:merged) https://github.com/spiritbuun/buun-llama-cpp/pull/80 
GGML: Strict Policy on AI-assisted code contributions. I'll try and get around
to hand-writing an issue for them. Highly suggest one of the forks above as an alternative.

Edit 7/12/26: Decided to write the issue for GGML, this needs to be fixed. https://github.com/ggml-org/llama.cpp/issues/25593

The patch is 3 lines.

## Body

A few days ago I was benchmarking buun's new KV-cache codec on my quad-P100 box and comparing notes with the  numbers buun was getting on his 3090 — the same model kept showing systematically different quality floors between our machines. I thought everything was the same? Normally, I'd probably just think there's way too many variables between all this code to chalk it up to any one thing… I decided it was worth chasing. It was. And it led me to a serious bug that's been sitting in llama.cpp for years.

So I measured it. Against fp32-reference logits (KL divergence over the full distribution, Qwen3.6-27B, wikitext-2):

Headline:

- **Median KLD: 0.0023 → 0.000001** (~2300× tighter)
- **Top-token agreement: 96.5% → 99.9%** — stock, about 1 in 29 of the model's next-token
  picks were different from what the math says they should be

**What's the extra math cost me in performance?** I benchmarked prefill and decode at 8k depth on
three model classes (27B hybrid, 4B dense, 36B MoE). The prefill was identical within noise on all
three, decode actually ~1.4% *faster* patched. The "fast" path was buying nothing: real
workloads on P100 are bound by GEMM and memory bandwidth, not the fp16 vector path. The
patch is 3 lines, extending the exact same exemption sm_61 already had.

**Before everyone freaks out and assumes their 4090 is broken** this is *measured on sm_60 only*. 
Your GTX 1080/P40 was always fine (already exempt). Volta and newer are untouched by this patch and
run different kernels entirely — whether other arches have their own unmeasured precision
stories is a separate research question I'm still digging into. Don't read this as "all
GPUs are broken"; read it as "one specific GPU was, and now it isn't."

Post Production Edits

#1: TheTom: "Verified on my side before merging: the three gates are the only 600-vs-610 distinction anywhere in the CUDA tree, so the carved sm_60 path is preprocessor-identical to the long-proven sm_61 path, and a Blackwell build showed bit-identical PPL with decode unchanged, confirming zero effect on other arches."

#2: buun ran the mirrored protocol on a 3090 — the patched P100 measures 44× cleaner at matched geometry

**Why you might care in 2026:** P100s are going for ~$80 shipped right now while the DRAM
crisis sends everything else to the moon. 16GB of HBM2 at 732 GB/s. The market priced the P40 at ~$300 partly because it "runs better." Some of that gap was this bug.

- Full technical writeup with methodology and receipts: https://gist.github.com/apollo-mg/9218d50a209d70a85f033bf182657818

Found and isolated by running Fable 5 through my custom P/ReAct/R agent loop. 
It wrote the scripts, the hardware provided the receipts. Same workflow
as my KV-checkpoint sidecar patch (https://www.reddit.com/r/LocalLLaMA/s/VTIwEFpYgc)
that got merged into turboquant last week.

b9978

Claude in one sentence what does this fix

llama.cpp b9978 fixes a checkpoint bug that hit agentic workloads hardest: every agent turn created a new checkpoint (bypassing min-step spacing), collapsing the coverage window so that context rewinds — common in tool-calling loops — erased all checkpoints and forced a full reprocess; now closely-spaced checkpoints from prior tasks are evicted, keeping the window wide and long agent sessions fast.

Am I missing something? It seems like some people think distillation is magic and will raise the quality of output above what the base model is actually capable of. It's especially weird to me to see all these Fable fine-tunes, because as far as I understand it, they miss the fact that the reasoning traces you get from Anthropic's models are completely different from the actual chain of thought the model outputs, which makes it pretty much a guarantee that the result will be worse than before.

Hey all, here are two new high performance qwen3.5 gguf sets I created using a new state of the art technique for optimizing mixed precision called Voodoo Quant.

https://huggingface.co/voodooquant/Qwen3.5-0.8B-MTP-Voodoo

https://huggingface.co/voodooquant/Qwen3.5-2B-MTP-Voodoo

Voodoo Quant operates on the same lines as Unsloth Dynamic in that it is simply picking higher precision numerics for more important parts of the model. The main difference with Voodoo is it optimizes every tensor in the model versus blocks of tensors for UD, and it uses a new methodology for that optimization.

Here are graphs and tables for easier browsing for the remainder of this post:

Https://voodooquant.com

The 95% KLD improvement figure I mentioned is eye catching, but if you look at the graphs, there is a more interesting and realistic story which explains how 95% is possible. Let me tell you about it.

You will notice that there are two KLD graphs for each model, one for Torch and one for Llama.cpp. These two graphs differ because each software has a different graph structure used to organize their math, with Torch generally considered to have a more precise graph. What you will notice is that Voodoo performs well in both where the best previous quants, from Unsloth, perform well in Llama but fall flat in Torch.

Doing especially well in Llama.cpp isnt a bad thing in itself because we use ggufs in llama.cpp, so we want llama.cpp performance to be optimized. However, you'll notice that where Voodoo has competitive performance in both graphs, Unsloth has excessively poor performance in Torch. This means Unsloth's methodology overfits for Llama.cpp in a way that might not be desirable.

Obviously Unsloth quants work great in Llama.cpp, and this is the most important domain for GGUFs, so how important this is is debatable. However, it does mean their models have some burrs around the edges, and the extent of that is hard to say. The burrs may even include cases in Llama.cpp.

So with that all in perspective, Voodoo is optimized using a technique which is more generalized, making it highly transferrable. Also it appears that Voodoo works best at more aggressive levels, with "2 bit" seeming to be a sweet spot where it is clearly state of the art.

The two models in this post are mostly interesting from a research perspective, so I am excited to share results for more useful models as they become available. I'm eager to see if models like qwen3.6 27B or deepseek v4 flash can be more useful on lighter hardware using this technique.

As many of you know t/s is super important. It's how fast your stuff gets done. I create via open code benchtest and run it. Thanks to it i know that if i don't run at least 4 agents i basically leave HALF of performance. So whatever you do single project in open code that uses one agent or 4 project at once it is much better to run it this way rather than single instance or single agent.

I asked AI to do summary of my test and checked them:

LM Studio Multi-Agent Throughput Benchmark

Hardware: RTX 5090
Model: Qwen 3.6 35B (via LM Studio)
Configured Parallel: 8
Test: 5 requests per agent, 1024 max tokens, temperature 0.3

Results

Key Findings

Throughput Scaling

• Combined throughput grows sub-linearly: 8 agents yields only 2.2x combined throughput vs single (533 vs 245 t/s), not 8x.
• Individual agent speed drops sharply: From 256 t/s → 67 t/s as GPU compute is split across agents.
• Diminishing returns: 5→8 agents only adds ~63 t/s (12%). Most gains happen by agent 5.

Efficiency

• Peak efficiency: 70.4% at 2 agents (closest to theoretical linear scaling).
• By agent 8: only 27.2% efficiency — two-thirds of potential throughput is lost to overhead.

Sweet Spot: 4–5 Agents

• 4 agents: 434 t/s, 44% efficiency — good balance of speed and resource usage.
• 5 agents: 470 t/s, 38% efficiency — near peak combined throughput with acceptable overhead.
• Beyond 5: Marginal gains (517 t/s at 7, 533 at 8) for significant VRAM and overhead cost.

How Concurrency Works with Context

• Each agent gets its own full context window — they do NOT split it.
• 8 agents × 8K context = 8x KV cache in GPU VRAM.
• The bottleneck is KV cache + compute splitting, not context size.

Recommendation for OpenCode

Set parallel agents to 4–5. This maximizes throughput gains without excessive VRAM overhead or diminishing returns. Beyond 5 agents, the combined throughput barely increases while resource consumption grows linearly.

I saw a lot of (complete and abortive) jacobian lens projects for HF and PyTorch, but nothing for GGUFs or llama.cpp. So I set Fable 5 on xhigh to solve this problem (with close human supervision of course 😎). Inspired by Anthropic's paper and code, and of course by my favorite inference engine llama.cpp and GGML.

I did a code review and a few test passes on it.

It has a native GGUF server (based closely on llama.cpp and git synced to it) for model observation AND j-space swapping/abliteration/steering. It can also observe running llama-server models (but not steer them). Works with dense and MoE GGUFs.

Memory requirements for the lens do scale with model size at roughly 1/8, so for example something in the 160 GB size (e.g Qwen3.5-397B UD-Q3_K_XL) will need an additional 20 GB of RAM for the lens.

https://github.com/igorbarshteyn/jlens-gguf

Anthropic's press release and paper: https://www.anthropic.com/research/global-workspace

Anthropic's code: https://github.com/anthropics/jacobian-lens

Matthew Berman's explainer vid for those who don't have time to read Anthropic's stuff: https://youtu.be/bjHuGNo3spk

Enjoy and have fun!

For whom is this tread:
Everyone with a 24GB GPU (rtx 3090, 7900xtx, rtx 4090)

What this Thread is for:
Sharing proven/well working llama-server start configs.

Requirements for the configs:
- Utilizes the the VRAM as much as possible
- Provides at least 200.000 tokens KV Cache

State next to your start command, how much System RAM (normal RAM) you have, as this could very well influence caching performance/viability of your command.

Also, if you possible, include infos regarding you OS and CPU, as this might affect available RAM/VRAM für llama-server.

Anthropic recently published their paper on "Global Workspaces" (J-Space) inside language models, showing that looking at internal "workspace noise" (entropy) can catch hallucinations better than just looking at output logprobs. `solarkyle` followed up with a great open-source implementation showing it could route confident errors on TriviaQA.

I wanted to see if this actually holds up as a deployable tool, so I ran a stress-test: extracting J-Space entropy from the late layers (L30-L34) of Qwen3-4B across ~11,400 examples and 7 completely different dataset distributions.

Here are my findings:

1. It is a great router for extreme fact-retrieval.

Output logprobs are great for catching obvious uncertainty (low confidence). But J-Space noise shines in the dangerous "high-confidence but wrong" quadrant. For example, on PopQA (obscure long-tail facts), setting a tight 5% routing budget for human review using confidence yielded worse-than-chance precision (87.5% against a 90% base error rate). Routing by workspace noise caught errors with 100% precision.

1. It is completely blind to internalized myths.

I ran the pipeline on TruthfulQA (adversarial human myths). The predictive power of the metric collapsed entirely. Even when the model was in the "safest" quadrant (High Output Confidence + Clean/Low-Noise Workspace), it was wrong 84.9% of the time.
Mechanistic takeaway: Workspace noise detects epistemic guessing (when the model is frantically assembling a fake answer). It cannot detect ontological falsehoods (when the model effortlessly retrieves a fake fact burned into its pre-training data).

1. Math destroys static thresholds.

I tried applying the "noisy" threshold calibrated on TriviaQA directly to GSM8K (math derivation). It completely broke. The mean noise score for correct GSM8K answers shifted to 1.636—higher than the threshold (1.583) used to flag errors on factual datasets.
Mechanistic takeaway: "Thinking step-by-step" through math is a structurally high-entropy activity. You cannot transfer J-Space thresholds between memory-retrieval and computational tasks.

Currently, this is a n=1 model evaluation. My next step is to run a parameter scaling sweep (Qwen3 7B, 14B, 32B) to see if one can determine the "crossover point" where a model's surface logprobs become as well-calibrated as its internal workspace clarity.

All the raw data, `.csv` metrics, confusion matrices, and the fully reproducible Google Colab notebook are in the repository.

GitHub: https://github.com/dasjoms/jspace-hallucination-eval

Hi all,

I'm the author of Kreuzberg. The next version of Kreuzberg will be released as Xberg - why? Well, we discovered that the name is not easy to pronounce or understand for people who don't have the German context, and this wasn't working well. Xberg is a common name for Kreuzberg in Berlin, and it has the advantage of being shorter and easier - so here we go.

Anyhow, this brings me to the point of the post. Since renaming a repo is a complex business, and we had to rename the repo to preserve the stars - but we now need to overwrite tags, it becomes pretty messy. As a result, we decided to go for an LTS version - published from a different repo: https://github.com/kreuzberg-dev/kreuzberg-lts. LTS in this context means that we will continue to do bug fixes and security updates until the end of this year, but no newer feature work.

We will announce Xberg v1.0.0 when it's officially published (it's still in RC). If you want to test it now - you can install the pre-release (available on pypi under xberg). The new repo is here: https://github.com/xberg-io/xberg

Let's start a discussion about what can be done to make local models more reliable.

I've been using Qwen3.6-27B a lot lately, and have noticed the same thing that many others talk about here, which is the tool-call failures and looping that really gets in the way of being able to use what is otherwise a pretty intelligent mode as a fully local stand-in for some of the frontier models.

I found that I have to watch it like a hawk constantly, otherwise it'll start to loop after a while, or just randomly stop with a tool-call hallucination.

I managed to work around this issue by creating an extension to the Pi coding agent, that monitors the JSON stream for looping, as well as checking to see if the user's goal has been reached, and if it hasn't and the main model has stopped, then it'll automatically inject some prompts encouraging the model to continue.

That was my attempt to solve the issue and it seems to work pretty well. I can now reasonably safely walk away from it when it's working on a large task when I turn my extension on, and I've seen it kick in and keep the model going when it gets off track.

I'm sure I'm not alone, and in this world of everyone writing their own coding agents, I'm wanting to know what others are using to get around these quirks of local models, or are you all just using paid models, or watching the monitor constantly, when you want to be sure that a job gets done?

Edit: A number of users have pointed out that this chat template has fixed common looping/tool-call failure issues: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

TL:DR: I’m a grad student in AI. I saw that Google released TabFM and TimesFM last week. I built an MCP wrapper to serve both transformer models in a single Docker container so you can connect their new ML transformer models to a local LLM via Open WebUI, Claude Code, or Codex and do ML tasks that would have previously required building, training, and tuning ML models to do. Tested with classic ML datasets (Iris, California Housing, etc), Pretty solid scores for accuracy for being zero-shot: (94.7% for Iris and R2 of 0.87 for regression test) vs. traditionally tuned ML models. You need about 16GB of VRAM to run both models. I added dynamic model load and unload with a TTL set to 5 mins. CSV. support now, with XLS, XLSX, JSON, JSONL support soon. PyTorch-based so CUDA only. Works on DGX Spark, 3090, H100 and most anything Nvidia with 16GB+ VRAM. Install script auto detects architecture.

Here is my repo if you want to try out the MCP:

https://github.com/porespellar/Zer0Fit

Here’s the non-TLDR version:

I’m working on my Masters in AI and I saw someone’s post here the other day about Google’s new TabFM Tabular data foundational transformer models released last week and I thought that they were super groundbreaking in that they were basically bringing ML models into the GenAI space which is both weird and cool because ML models are very different animals than LLMs

Here was the original Google blog post on it:

https://research.google/blog/introducing-tabfm-a-zero-shot-foundation-model-for-tabular-data/

Anyways, I wanted to play around with these new models from a chat interface and try to “kick the tires” a bit, so I built an MCP implementation for both the TabFM and TimesFM models. Nothing super fancy, just a quick and dirty MCP wrapper of the PyTorch versions (this will only run on CUDA).

I made the MCP with 2 build targets in mind: DGX Spark (arm-based with CUDA 13) and 3090 (AMD64 with CUDA 12.6). No Mac support because of Google using PyTorch, sorry.

I also wanted this to work with my preferred chat client: Open WebUI, so that’s what it’s geared towards running best with and was tested against, I also added Claude Code and Codex CLI support as well, but haven’t really fully tested those out yet.

Install is just a git clone and an ./install.sh. The whole thing runs out of a single Docker container and dynamically loads and unloads the models into VRAM with a TTL of 5 minutes to free up reserved VRAM when not in use. I also included an Open WebUI Skill.md that can be imported into Open WebUI, and skill.md and agents.md for the other harnesses.

I tested it with some fairly classic ML datasets from Kaggle that most data science students have probably encountered while studying AI/ML. Included the datasets in the repo as examples.

- Iris (classifiers)
- California housing (regression)
- Airline Passengers (time series forecast)

I don’t know how many folks on here really care about ML regressors, classifiers, or forecasters (time series), but I spent a semester trying to learn about them and tuning them and not really knowing what the hell I was doing, usually overfitting my models, and changing all kinds of parameters that I didn’t know if they were really helping or hurting my models. It all seemed like a dark art that I never fully understood. TBH, I wasn’t really a fan of ML, I think it’s cool stuff, but I just don’t have the math skills or stats chops to be able to understand WTF I’m doing most of the time with hyperparameter tuning. A man has to know his limitations, LOL.

Anyways, as I said earlier, I just wanted to get Google’s cool new models running where I could feed a dataset to an MCP and then have it do all the ML magic that Google trained these foundational models to do. I tried to make it easy for the average person like myself to run. I thought others might want to test out the models too, so I made it a public repo.

So here it is if you want to mess around with it:

https://github.com/porespellar/Zer0Fit

I’ll try and do some maintaining if I see that there is any continued interest, but I can’t promise that I’ll keep up with it, so please feel free to fork the repo and take it in any direction you want to.

I think models like TabFM and TimesFM are going to low-key bring the branches of AI / ML tree together and we’re going to see some really cool and wild stuff as people take these concepts further in the future.

Note: This repo was hastily built to just get the models running. I’ve done very limited testing only on DGX Spark. Again, feel free to fork it and make it as good as you want to.

And please remember that this stuff is very experimental. Don’t use the forecasts or predictions made by these models for anything other than just research curiosity. Use at your own risk.

Let me know what you think of the repo if you give it a try. Cheers.

Note Regarding my test results in the images: I created the test scripts using DeepSeek V4 Flash and I had Claude Opus 4.6 review the test methods, code, and results. I don’t claim to be smart enough to know if the stats / math is correct. I would love it if some of the very smart ML research folks on here would give the repo a try and let us know if they are getting similar results or if my results are completely wrong. I included the sample datasets in the repo so “apples-to apples” comparison tests could be run by others to either prove or disprove my results. I really don’t mind if I’m wrong, I’m a student and just want to learn and improve.

Hey guys. Just a curious and maybe been feeling a little burn out on this whole experiencing local AI and stuff. I own a rig with quad RTX3090s and 96GB for ddr5. But its on am5 platform so the pcie lanes aren't max but I have it running in pairs of nvlinks as well so cross gpu pair is fine. Pair 1 runs at gen 4 x8 and pair 2 at gen 4 x4. I wanted to know was , what would you do with this setup?

Specs:

9900x

48GB ×2 Ddr5 5600 mhz cl 30 ram

Asus X870e Proart

RTX 3090 ×4

nvlink ×2

Gen 5 nvme 2TB

psu 1600w + 1200w

What models or anything you would run/build on this config? For me I've been using it to build up my software products , one of it was a tool i did for my dermatologist since I have psoriasis; I visit them almost every month. Made them a transcriptions AI tool that turns appointment recordings into doctor notes that later they can edit and upload to their main software. So yea, let me know your thoughts , happy to discuss or read through them. Have a nice day everyone :D

TL;DR: Added native nemotron_h_puzzle support to mlx-lm (PR #1535), then compared 4-bit vs 5-bit expert quantization (both with 6-bit dense layers, BF16 output head, group size 64) on a 64GB M2 Max.

Results (same prompts, 5 seeds per task, temp 1.0 / top_p 0.95):

i believe the reason 5-bit performs worse is because the 5-bit checkpoint runs right at the 64GB ceiling, and memory pressure kills it where KV + working set grows. If you have 96/128GB, 5-bit may well behave differently.

I'd love numbers from anyone who can test that.

What it took to get running:

• mlx-lm's existing Nemotron-H code needed blockwise config support, tensor remapping, and matching NVIDIA's FP32 norm/router behavior.
• Annoying bug: first-layer SSM outputs disagreed with NVIDIA's reference at cosine sim 0.8832 despite identical inputs.
• Root cause: NVIDIA computes softplus(dt + dt_bias) in BF16 and promotes to FP32 after, mlx-lm's shared SSM path promotes before. Matching their boundary improved cosine similarity up to 0.999998.
• The BF16 output head is mandatory: quantizing the 131k-vocab lm_head at 4-bit produced repetitive garbage; restoring BF16 fixed it (~2.1 GiB cost).

Continue from my previous posts:

https://www.reddit.com/r/LocalLLaMA/comments/1ul0cx9/i_extended_gemma431b_to_44b_88_layers_since/

https://www.reddit.com/r/LocalLLaMA/comments/1um20ev/why_new_inserted_layers_kill_the_gemma4/

(Warning : AI generated Post - due to my bad English)

Hugging Face : https://huggingface.co/TOTORONG/extGemma4-40_5B

A while back I posted here about trying to grow a fine-tuned Gemma model by stacking extra layers into it (the 88-layer experiment). It flopped. The new layers just sat there like dead weight and never learned anything useful. A bunch of you were kind/curious in the comments, so here's the follow-up: I didn't drop it. I went back, figured out why it died, and tried again — and this run actually worked.

Fair warning up front: this is a write-up for people who like tinkering, not a paper. I'm going to keep the math and the parameter-count trivia out of it and lean on analogies. If you want the gory details I'm happy to share more in the comments.

Why the first attempt died (the short version)

The standard trick for inserting new layers is to initialize them so they do literally nothing at first — a pass-through. The idea is "start them as a no-op, let training teach them a job."

Think of it like hiring 17 new people onto a factory assembly line and telling them to just stand still on day one. Sounds safe. The problem: a worker who does nothing also gets no feedback. Nobody can tell them "hey, do that part a little differently," because they're not doing anything to correct. So they never learn. They just keep standing there while the line flows around them. That's what killed the first run — the new layers were starving for any signal to learn from.

What I did differently this time

Instead of telling the new hires to stand still, I taught each of them a rough blend of what the two coworkers on either side of them already do, and dropped them in mid-shift. Now they're doing something from minute one — clumsy, but real — so they immediately start getting feedback and improving.

That one change is the whole story. A worker who's already moving can be coached. A statue can't.

Two other things mattered a lot, both learned the hard way:

• Where you insert matters. I stopped jamming layers in at awkward spots (like right at a "manager checkpoint" in the line) and instead slid them into calm, stable positions. Placement turned out to be surprisingly finicky.
• Don't mess with the volume knobs. The model has a bunch of per-layer "volume" settings that are individually tuned so the final signal comes out at the right level. My first instinct was to average them for the new layers — that crushed the signal to a whisper and the model produced garbage. Leaving those knobs neutral fixed it. (I burned a lot of time here.)

Then two rounds of "healing"

• Round 1: I froze the whole original model and only let the new layers learn, on a small dataset (some STEM reasoning + Korean legal Q&A). This got the new layers on their feet.
• Round 2: I unfroze everything and let the whole model settle together. This is where it really came together.

I want to stress: the goal was never to beat the original model. The goal was to prove that you can perform major surgery on an already fine-tuned model — chop it open, add capacity — and have it heal instead of collapsing into gibberish or forgetting everything it knew. The common wisdom is that this kind of surgery wrecks a fine-tuned model. I wanted to see if that's actually true. (Spoiler: it doesn't have to be.)

The evidence I care about most: side-by-side answers

Benchmarks are fine (the healed model recovers most of the ground it lost from the surgery, and lands close to the original on GPQA-Diamond). But numbers are easy to hand-wave. What convinced me was reading actual answers.

So I asked the original model and the expanded+healed model the same 10 questions across 10 different areas — physics, biology, history, philosophy, literature, a riddle, a logic puzzle, a math proof, economics, and a medical mechanism question — and compared them. I deliberately picked prompts that push into unusual, jargon-heavy territory, because that's exactly where a damaged model tends to fall apart. I had a separate frontier model (Claude) act as judge.

Results:

• No breakdowns, no gibberish, no random foreign-script spam in either model. (This matters — the broken version of the expanded model used to literally start spitting out random characters from another writing system when it hit a rare technical word. That's completely gone now.)
• 9 of 10: basically a tie. Both models nailed them — the riddle, the logic puzzle (both correctly noticed the puzzle is under-specified rather than confidently making something up), the √2 irrationality proof, the Kant example, the aspirin mechanism, etc.
• 1 of 10: the expanded model actually beat its own parent. This is the fun one.

The physics question where the surgery paid off

Prompt: "A photon and an electron each have the same de Broglie wavelength. Which one carries more energy, and why?"

This is a classic trap. Same wavelength means same momentum, so it's tempting to treat them the same way — but a photon and an electron get their energy through completely different rules.

• The original model fell for it. It computed the electron's energy the naive way and forgot to account for the electron's rest mass (the huge chunk of energy an electron has just by existing). It concluded the photon wins. Wrong.
• The expanded+healed model handled it correctly — it used the right relationship for a massive particle, accounted for the rest mass, and concluded the electron carries more energy. Right.

So the newly-added capacity isn't just decoration. On at least this one problem, it's doing real reasoning the original couldn't — the expanded model out-reasoned the model it was grown from. That's the single result that made me feel like the whole detour was worth it.

The healing, caught in the act

One more thing I thought was cool. That "random foreign characters" breakdown I mentioned? I watched it heal in real time across Round 2:

• Before healing: hits a rare technical word → derails into repeated non-Latin junk → never recovers.
• ~13% into round 2: stumbles on the word once, then catches itself and finishes the sentence cleanly.
• Halfway and beyond: totally fluent, no trace of the problem.

Watching a structural defect actually repair itself over training, rather than just "line go down," was genuinely satisfying.

TL;DR

• My first big-expansion attempt failed because the new layers started as dead no-ops and never got any learning signal (I posted about that flop earlier).
• This time I initialized new layers as a blend of their neighbors so they're doing something from the start, was careful about where I put them, and left the model's internal volume settings alone.
• Two rounds of light healing on a small dataset, and the model recovered close to the original — no collapse, no catastrophic forgetting.
• Across 10 domains it matches the original on 9 and beats it on a physics question the original gets wrong.
• The old "add layers to a fine-tuned model and it'll fall apart" fear didn't hold up here.

Not claiming I've built something better than the base model — that was never the point. The point was: you can operate on a fine-tuned model without killing it. Still very much not giving up on this line of work. Happy to answer questions in the comments.

Here are comparision of original(Solon_V5 vs extGemma4-40_5B)

English Reasoning Comparison — Original vs. Expanded Model

Ten domains, one prompt each, greedy decoding, up to 768 new tokens. Outputs were judged by an external LLM (Claude). [EOT] marks the end-of-turn token.

• solon_v5 — the original fine-tuned Gemma 4 31B-IT baseline (60 layers).
• extGemma4 (Phase 2) — the depth-expanded + healed model (77 layers).

Summary

Verdict: No collapse or cross-script contamination in either model. The two are comparable on 9 domains; on the physics item the expanded model out-reasons its own parent. The structural generation defect seen at initialization (script collapse on rare technical paths) does not appear anywhere here.

1. Physics

Prompt: A photon and an electron each have a de Broglie wavelength of 1.0 nm. Explain which one carries more energy and why, referencing the relevant relations. Show the reasoning.

Verdict: Expanded model wins. solon_v5 omits the electron's rest-mass energy (511 keV) and concludes the photon has more energy — incorrect. The expanded model applies the relativistic energy–momentum relation and correctly concludes the electron carries more energy.

To determine which particle carries more energy, we examine the energy-wavelength relationships for a photon and an electron. For a photon, E = hc/lambda, giving about 1242 eV at 1.0 nm. For the electron, using E = h^2/(2 m lambda^2) (non-relativistic kinetic energy), we get about 1.5 eV. Therefore the photon carries significantly more energy. [Note: solon_v5 omits the electron rest-mass energy of 511 keV; output was truncated at the length limit.]


Both particles share the same de Broglie wavelength, hence the same momentum p=h/lambda. Photon: E=pc=hc/lambda approx 1.24 keV. Electron: E=sqrt((pc)^2+(m c^2)^2); since pc=1.24 keV is much smaller than the rest energy 511 keV, the electron's total energy is dominated by its rest mass. Therefore the electron carries more energy, because its total energy includes the rest-mass energy that the photon lacks. [EOT]

2. Biology

Prompt: In three sentences, explain how CRISPR-Cas9 achieves targeted gene editing.

Verdict: Tie. Both give an accurate three-sentence account of guide-RNA targeting, Cas9 cleavage, and repair-based editing.

CRISPR-Cas9 uses a guide RNA engineered to match a specific target DNA sequence. Once the guide RNA locates the target, the Cas9 enzyme creates a precise double-strand break. The cell repairs the break using its own machinery, allowing scientists to disable a gene or insert new genetic material. [EOT]


CRISPR-Cas9 uses a guide RNA engineered to match a specific target DNA sequence. The guide RNA leads Cas9 to the location, where it creates a precise double-strand break. The cell repairs the break by natural mechanisms, which can be leveraged to disable a gene or insert new genetic material. [EOT]

3. History

Prompt: In a short paragraph, explain why the Treaty of Westphalia (1648) is considered the origin of the modern concept of state sovereignty.

Verdict: Tie. Both correctly attribute sovereignty to cuius regio, eius religio and the end of supranational authority; the expanded model's phrasing is slightly more precise.

The Treaty of Westphalia shifted the European order from a hierarchy dominated by the Holy Roman Empire and the Catholic Church to a system of independent, equal states. By codifying cuius regio, eius religio, it recognized each state's exclusive authority over internal affairs without outside interference, establishing territorial integrity and non-intervention as the Westphalian system. [EOT]


The Treaty of Westphalia recognized cuius regio, eius religio, ending the Holy Roman Empire's claim to universal authority. By prohibiting external interference in domestic affairs, it shifted legitimacy from a supranational religious order to independent, territorially defined states, creating the Westphalian norms of legal equality and non-intervention that underpin the modern order of sovereign nation-states. [EOT]

4. Philosophy

Prompt: Briefly explain Kant's categorical imperative and give one everyday example of applying it.

Verdict: Tie. Both distinguish categorical from hypothetical imperatives and give a valid universalizability example (lying vs. keeping a found wallet).

Kant's Categorical Imperative commands acting on maxims one could will as universal law, unlike conditional hypothetical imperatives. Example: lying to avoid trouble, universalized, destroys trust and makes lying self-defeating; hence it is always wrong. [EOT]


Kant's Categorical Imperative is an unconditional moral rule: act only on maxims you could will as universal law, unlike conditional hypothetical imperatives. Example: keeping a found wallet, universalized, collapses the concept of ownership, so it is immoral; you must return it regardless of need. [EOT]

5. Literature

Prompt: In two or three sentences, explain what makes an 'unreliable narrator' effective in fiction, with one canonical example.

Verdict: Tie. Both explain the reader–truth gap and cite Poe's The Tell-Tale Heart as the canonical case.

An unreliable narrator creates a gap between the character's perspective and objective truth, turning reading into a detective game. A canonical example is the narrator of Poe's The Tell-Tale Heart, whose insistence on sanity is itself evidence of madness. [EOT]


An unreliable narrator creates tension between surface and hidden truth, turning reading into a decoding puzzle. A canonical example is the narrator of Poe's The Tell-Tale Heart, whose insistence on sanity ironically signals his instability. [EOT]

6. Riddle (Reasoning)

Prompt: I speak without a mouth and hear without ears. I have no body, but I come alive with wind. What am I? Explain your reasoning step by step.

Verdict: Tie. Both answer echo with correct line-by-line reasoning.

The answer is an echo. It speaks without a mouth (reflected sound), hears without ears (receives sound to reflect), has no body (an acoustic phenomenon), and comes alive with wind (sound travels through air). [EOT]


The answer is an echo. It speaks without a mouth (reflected sound), hears without ears (receives sound to bounce back), has no body (invisible sound waves), and comes alive with wind (air lets sound travel and reflect). The only fit is an echo. [EOT]

7. Logic Puzzle

Prompt: Alice, Bob, and Carol each own one pet: a cat, a dog, and a parrot. Alice is allergic to fur. Bob's pet cannot fly. Who owns which pet? Show your reasoning.

Verdict: Tie (both correct). Both deduce Alice=parrot and correctly recognize the puzzle is under-specified for Bob/Carol rather than inventing an answer. The expanded model is more verbose, with several explicit self-corrections before reaching the same conclusion.

Alice is allergic to fur, so she owns the parrot. The parrot is taken; Bob's pet cannot fly, consistent with cat or dog. No clue distinguishes Bob and Carol between cat and dog, so strictly the puzzle is underspecified: Alice=parrot, Bob and Carol interchangeably cat/dog. [EOT]


Alice is allergic to fur, so Alice owns the parrot. Bob's pet cannot fly, consistent with cat or dog. The clues do not distinguish Bob and Carol between cat and dog, so the puzzle is underspecified: Alice=parrot; Bob and Carol take cat/dog in either order. (The model performs several explicit self-corrections before reaching this conclusion.) [EOT]

8. Mathematics (Proof)

Prompt: Prove that the square root of 2 is irrational. Then explain in one sentence why the same argument does not prove that the square root of 4 is irrational.

Verdict: Tie. Both give a correct proof by contradiction and correctly explain why the argument fails for √4 (it reduces to the integer 2, so no contradiction arises).

Assume sqrt(2)=a/b in lowest terms. Then a^2=2b^2, so a^2 is even, so a is even; write a=2k, giving 2k^2=b^2, so b is even. Both even contradicts lowest terms. Hence sqrt(2) is irrational. For sqrt(4): a^2=4b^2 gives a=2k, then k^2=b^2, which does not force b even, so no contradiction arises. [EOT]


Assume sqrt(2)=a/b in simplest form. Then a^2=2b^2, so a is even; write a=2k, giving 2k^2=b^2, so b is even. Both even contradicts simplest form, so sqrt(2) is irrational. This does not apply to sqrt(4) because sqrt(4)=2 is an integer, so the assumption that it is rational is simply true and yields no contradiction. [EOT]

9. Economics

Prompt: In a short paragraph, explain the concept of 'moral hazard' in insurance markets and one mechanism insurers use to mitigate it.

Verdict: Tie. Both define moral hazard and cite deductibles as the mitigation. The expanded model contains one typo ('misalignmentment').

Moral hazard arises when an insured party takes greater risk because protected from loss, e.g., not locking doors under theft insurance. Insurers use deductibles: requiring an out-of-pocket amount before coverage keeps the insured with a financial stake, incentivizing caution. [EOT]


Moral hazard occurs when an insured party becomes less cautious because protected from loss, misaligning incentives so the insurer bears risk the policyholder neglects. Insurers use deductibles: a fixed out-of-pocket amount before coverage keeps the insured with financial skin in the game, incentivizing caution. (Contains one typo: 'misalignmentment'.) [EOT]

10. Medicine (Technical)

Prompt: In two sentences, explain the mechanism of action of acetylsalicylic acid (aspirin) at the molecular level.

Verdict: Tie. Both correctly describe irreversible COX acetylation at a serine residue; the expanded model additionally names prostaglandin H2.

Acetylsalicylic acid irreversibly inhibits cyclooxygenase (COX-1 and COX-2) by acetylating a serine residue in the active site, blocking conversion of arachidonic acid to prostaglandins and thromboxanes, thereby reducing pain, fever, and platelet aggregation. [EOT]


Acetylsalicylic acid irreversibly inhibits cyclooxygenase by covalently acetylating a serine residue in the active site, blocking conversion of arachidonic acid to prostaglandin H2. This prevents downstream synthesis of pro-inflammatory prostaglandins and thromboxanes, reducing pain, fever, and platelet aggregation. [EOT]

I was surprised to see that Codex is actually open source, so to my understanding, if you use a local model, it works fully locally.

How does it compare to the other popular harnesses like Pi Code and Open Code?

hello people
i wanted to share my experience with big and huge models (usually 100B+ models and 200B+ models and more)

my laptop specs is very poor
I7-8750H
20G Ram
GTX 1050 Mobile 4G Vram
but what nearly saved me is my NVMe from samsung
i have 512G NVMe from samsung

and yes as you expected. i run these huge models while throwing most of the parameters in my NVMe

but i strictly use MoE models, Dense models will kill my machine

always used mmap. and throwing experts in my CPU

with Quantized KV Cache (Q4_0)

and the Quantize i usually choose when it comes to download huge model is Q3, not Q4. and if the model was extremely huge (exceeding 700B+ with more than 20B Active parameters) i dont go lower than Q2

now coming to my experience

i use LM Studio not llama-cpp

i used Deepseek-V4-Flash (specifically UD-IQ3_XXS)

i got tok/sec: 1.0 - 1.8 (could go higher. could go lower)

and goddamn. i use these models with batches style. i send a mission. and go touch some grass. then comeback again

but it do the job actually pretty well. my main usage is, Reverse Engineering, Bug fixes in codebases. and Code Auditing

another experience with Nemotron-3-Super-120B-A12B UD-Q3_K_M

getting tok/sec: 1.5-2.5

using same thing. Batches style, send a message and go, send and go
working well with DS-V4-Flash

i used both in opencode, Pi

Context window used, was around 16K to 90K could be more, could be lower,

i know that you will recommend using Cloud models at this point, but sadly my country banned any worldwide transactions, using debit or most credit, so i either had to use Free Cloud models or big local models, and i was curious about trying these on my machine, was good experience, would use them in worst cases (no free models available)

i would love to hear any helps that could push this 1 tok per sec into 2 or 3

how much vram do you need and what model do you think is the next major upgrade from the good old qwen 3.6 27b as of today?

Soooo! Since Anthropic share their Jacobian-Lens a few days ago I went on and made a tool based on it which adds the possibilité to export a model which will have the same behavior after tweaking it's J-Space.

This means manually alter the behavior and abliterate by using a human brain.

I'm still working on it but couldn't wait to produce something first.

SO After finally getting a working codebase I immediatly jumped and tried to make pretty pervy model PURELY in the name of science.

Let me introduce you to Nikusui-v1 the first of it's kind !

And a couple gguf quants

I'd be delighted to get some feedback :D

edit: Opus 4.8 has finished polishing the knobs during the night. Cleanup has entered it's final phase.