6 local LLMs I’ve used that prove they’re not just smaller versions of cloud models
For most of the last two years, running an LLM at home has meant the same thing. For most people, that’s meant accepting a smaller, quantized, or otherwise compromised version of the kind of model they could also access through a cloud API. Whether it’s a smaller Llama variant or a distilled Qwen, most of them are trying to reach the same target through different methods while ultimately ending up in similar places. That’s starting to change quite a bit, and not just in a “models are getting better” kind of way. Models built in 2026 aren’t just smarter than last year’s; they’re built differently, trained differently, or in some cases, they don’t even generate text the way anything else you’ve used does. A surprising number of them are aimed at running on your home computer rather than on a server in a data center. Quantizing a model’s weights only solves half the problem of running it locally; the other half is the KV cache, which is the running memory of the conversation itself, and it grows with every token of context in a way conventional weight quantization can’t impact. A Llama-2-7B at 128K tokens needs something like 64GB just for that cache in FP16. A lot of the models here are doing some interesting things to deal with that side of the equation. A lot of the benchmark numbers here come from the labs that built the models, and I’ll point out when that’s the case. Also, a couple of these I can’t run to their full effect locally, like in the case of Zaya1. While none of them will dethrone a cloud model on an individual level, taken together, they’re a culmination of why I think local AI will change significantly.
Zaya1-8B changes how we think about context windows
One small model becomes multiple at once
Credit: Zyphra
Zaya1, from Zyphra, is the model I’ve been most impressed by, and the one I think is the most promising for a number of reasons. It’s an 8.4B mixture-of-experts model with only about 760M parameters active per token, but that’s not the only interesting part. No, it’s that it uses Compressed Convolutional Attention, which squeezes the query, key, and value projections into a shared latent space and does its sequence mixing with convolutions. In practice that works out to an eight-fold KV-cache compression with no obvious quality drop in my own testing. The other trick it uses is Markovian Recursive Self-Aggregation, a test-time reasoning method that Zaya1 was explicitly co-trained to use. It combines the tail end of multiple parallel reasoning traces into one bounded aggregation context, letting the model reason for longer without carrying the full chain forward. I gave it a modified AIME 2026 problem, a system of three logarithmic equations rewritten so no memorized answer would help, where the obvious approach of solving for each variable turns into a mess of fractions. Instead of grinding through that, it spotted that it only needed one particular combination of the three equations and set up a small linear system to collapse them straight onto the answer, went through about 7,400 reasoning tokens, and eventually settled on the right value. GPT-5.5 and Claude Opus 4.8, handed the same problem, came back with wrong answers on multiple tries. Zyphra reports 91.9% on AIME 2025 with the reasoning method switched on, and while those are the lab’s own numbers, they line up with what I saw. Running it locally I got about 7 tokens a second in BF16 on an M4 Pro, and around 42 with an MXFP4 quant through vMLX. There’s a real catch, though: the full parallel-trace version of the reasoning is cloud-only for now.
VibeThinker-3B shows that reasoning compresses further than we thought
Hate the name, love the model
Zaya1 is an interesting model, but VibeThinker is weird in the best way. It’s a 3B dense model built on Qwen2.5-Coder-3B, and its whole reason for existing is what the creators call the Parametric Compression-Coverage Hypothesis: the idea that verifiable reasoning, the math-and-code kind where an answer is definitively right or wrong, can be packed into a smaller model than most would typically assume. The training recipe layers a two-stage supervised setup, including a reinforcement learning phase with their own MaxEnt-guided policy optimization, and offline self-distillation on top. Its benchmark claims are interesting; 94.3% on AIME 2026, 70.2% on GPQA Diamond, a 96.1% acceptance rate on LeetCode, all pitting it against models significantly larger. Those are lab-reported values, and the numbers are disputed alongside claims that it wasn’t even measured correctly. The authors are also explicit that this is a model for verifiable math, code, and STEM-style reasoning, not a replacement for larger general-purpose systems; they specifically point to GPQA-Diamond as a place where small models still trail broader, knowledge-heavy models. Outside of the uses it was trained for it falls apart, but inside it, it earns its keep. I gave it the same modified AIME 2026 problem I’d thrown at Zaya1, and it got the right answer, which is more than either GPT-5.5 or Claude Opus 4.8 managed when I ran them on it, both getting the system wrong. I modified another question and it got the right answer as well, which GPT-5.5 got wrong. The predecessor 1.5B model was reportedly post-trained for around $7,800, against estimates well into six figures for comparable frontier reasoning models. If reasoning of this kind really is this cheap to instill, then what stands between you and a capable local reasoner is the way you train it, not the hardware you own.
DeepSeek V4 Flash runs a 284B model on more accessible hardware than you’d think
It actually works decently well
DeepSeek V4 Flash is a 284B mixture-of-experts with only about 13B active per token. It’s MIT-licensed with a one-million-token context window. What makes that context possible to run locally is a hybrid attention design DeepSeek calls Compressed Sparse Attention paired with Heavily Compressed Attention, and holding a full million tokens ends up costing roughly 10% of the KV cache that DeepSeek-V3.2 would need. antirez’s ds4, or “DwarfStar 4,” is a self-contained inference engine written in C by the creator of Redis. It’s a runtime built for DeepSeek V4 rather than a generalized GGUF runner, and it places the KV cache on disk so you can use the model with streaming from storage. It has an OpenAI and Anthropic-compatible API server, and has a coding agent built into the process itself. It needs a machine with proper unified memory to work, so a Mac with around 96GB or a DGX Spark-esque machine will work. I’ve been running V4 Flash on a Lenovo ThinkStation PGX, where the GB10 Grace Blackwell chip and 128GB of unified LPDDR5x meet the requirements. It isn’t the only huge model that you can run on a device with 128GB of unified memory. StepFun’s Step-3.5 Flash is similar, and it’s a model with 197B parameters, 11B active that uses sliding window attention to keep memory low. I’ve run it on the ThinkStation PGX at Q4_K_S with a 140K token context window.
Qwen 3.6 keeps its attention memory from growing at all
Gated DeltaNet is incredibly interesting
Qwen’s recent models, including the Qwen 3.6 family of models, use Gated DeltaNet, a form of linear attention. It’s present in most of its layers, running roughly three DeltaNet layers to every full-attention one, with that fourth layer acting as a kind of precision checkpoint. Where a standard transformer’s attention memory grows with every token you feed it, a linear-attention layer keeps a fixed-size recurrent state that never grows with context at all. It stores the delta against what the state already holds, and gating adds decay so old information goes away. My favorite demonstration of what that gives you comes in the form of Qwen3-Coder-Next, which I’ve run on the ThinkStation PGX. It’s an 80B model, but an ultra-sparse mixture-of-experts with only 3B active per token, and 36 of its 48 layers are Gated DeltaNet, which means the entire fixed state adds up to around 18MB at 170K tokens, with just 12 full-attention layers using 2-head grouped-query attention. On the PGX’s 128GB of unified memory I run it at Q4_K_M, around 46GB, or Q8_0 at around 85GB, with a 170K context window, served through vLLM in a Docker container with Claude Code pointed straight at the local endpoint, where it generates at a rate between 25 and 40 tokens a second. Qwen 3.6 has a 27B dense variant and a 35B-A3B MoE, with 262K of native context that stretches toward a million, and I keep the 27B on my home server and run the 35B-A3B on the PGX. Qwen isn’t the only model to have done this, though. MiniMax M2 had a similar thing in its architecture, but the lab dropped it, citing reasoning and multi-turn accuracy problems. Qwen is a bet, right now, that linear attention is good enough for real work.
DiffusionGemma doesn’t generate text left to right
Watch your text appear like an image
Of everything on this list, DiffusionGemma is the one that most literally redefines what a local LLM can be, because it doesn’t generate text in the usual left-to-right autoregressive way. Built on a Gemma 4 26B/4B-active MoE base, it uses discrete text diffusion, meaning that instead of committing to the next token and then the one after that, it starts from a canvas of 256 random tokens. It refines the whole block in parallel with bidirectional attention, so the start and end of a passage influence each other before anything is locked in. If you watch it run, the text doesn’t stream in left to right; a whole block resolves at once, like an image coming out of noise. I tried it as a 4-bit GGUF through Unsloth’s custom llama.cpp fork on my M4 Pro, and gave it a Flappy Bird clone in Flask to write. It produced working code in about 138 seconds across 123 denoising steps and 9 blocks. The game ran, though the bird’s gravity was comically aggressive and it took some coaxing to get the physics sane. It’s unusual enough that you need a specific llama-diffusion-cli and a visual-diffusion flag just to watch it think, so it’s not really a model you can just casually drop into a workflow just yet.
There’s a catch when it comes to speed, though, as Google advertises a 4x speedup with 1,000-plus tokens per second on an H100. That speedup comes from parallel denoising on a compute-heavy accelerator, and Apple Silicon’s bandwidth-bound unified memory doesn’t get the same benefit, so it wasn’t that fast on my Mac. Google also says that autoregressive Gemma 4 still wins on outright quality, making this little more than a cool proof of concept right now. It needs at least 16GB of VRAM at Q4 and it’s Apache-2.0, but as a working proof that local generation doesn’t have to be autoregressive, it’s incredibly exciting.
Gemma 4 E2B and E4B see and hear on a phone
Audio is unique for an LLM
This model does something that none of the others on this list can: it hears. Gemma 4’s E2B and E4B are on-device multimodal models handling text and images, and the “E” stands for effective parameters. E2B behaves like a roughly 2B model and E4B like a roughly 4.5B one, small enough to run on a phone, but they punch well above those weights thanks to a couple of tricks that are unusual for models this size. The first is MatFormer, a Matryoshka-style nesting where the smaller E2B is co-trained inside the larger E4B, so you can pull a smaller model out without retraining anything and one download gives you several deployment targets. The second is Per-Layer Embeddings, which claw back efficiency by giving each decoder layer its own token embeddings in cheap lookup tables. I ran E4B as a phone-hosted server on an Oppo Find N5, a Snapdragon 8 Elite with 16GB of RAM, building llama.cpp inside Termux and serving the model over an OpenAI-compatible API at Q8_0, about 4.3GB, with a BF16 multimodal projector adding another 900MB or so. It sat at roughly 6GB of RAM, stayed loaded overnight, and generated at a rate of about 7 to 8 tokens a second with sub-second time to first token. It described images, though a small one takes about 10 seconds to encode before generation even starts and another 10 to 20 to write the description, and it transcribed audio through its native conformer encoder, while tool calling worked but felt rather fragile. Still, a phone that sees, hears, and calls tools locally is pretty cool, even if it’s nowhere near a finished product yet.
There’s a bigger pattern emerging
While these models are wildly different, they all have one thing in common: a novel approach to at least one part of the assumed LLM architecture. None of them on its own has changed the world of LLMs, but there are several promising leads here that have all come from models the everyday user can run on their laptop. All of these models taken together are why I think predictions that soon we’ll all have AI models running locally aren’t so far fetched. I love seeing the new technology and innovations coming out of these models, and they’re far more exciting than most of what I see coming out of the frontier-model space. They may not be as powerful, but they sure are exciting.
Diterbitkan : 2026-07-11 12:30:00
sumber : www.xda-developers.com
