Why a Local LLM Seems Dumber Than It Really Is

A thorough technical breakdown authored by a user named thr3e has appeared on the Level1Techs forum, addressing a situation many will recognize: everyone praises some model, you download it (usually in quantized form) — and you're disappointed. The central claim is simple and reassuring: your local setup is “being dumb,” but so is everyone else's — just in its own way.

The reason is that every combination of hardware and software computes the next token a little differently. Different GPU generations, different instruction sets, different CUDA kernels, and on top of all that a mountain of dependencies: in a nightly build of the vLLM container the author counted 734 packages, each with its own bugs and quirks. He separately reminds readers about sampler settings: these should be taken from the model card, and too low a temperature is a classic reason Qwen gets stuck looping in its own “reasoning.”

Math Is Math

The most curious part is the experiments on the Qwen3.6-27B model, where the author captured the full logits and compared the results. It turned out that changing even the single attention mechanism (FlashAttention 2, Flash Inference, or Triton) — with the weights, drivers, and prompt all unchanged — is enough to make the model start choosing different tokens. The divergences don't appear gradually as the context grows, but in clusters tied to the content of the prompt. Meanwhile, repeated runs on the same backend produce bit-for-bit identical logits — meaning the culprit is purely the arithmetic of matrix multiplication.

KV-cache quantization is even more striking. Cut down to int4, the model falls apart after a certain amount of context and can't correctly complete a tool call; int8 makes mistakes but recovers; bf16 works normally.

The Battle of the Quants

In a comparison of five weight variants, INT8 (W8A16) came out on top, beating even the vendor's own FP8. NVIDIA's NVFP4 release, on the other hand, came in last, racking up about 50% top-1 divergence by 88,000 tokens. Both NVFP4 and the AWQ quant not only failed to close out function calls but also garbled Cisco command-line syntax: instead of show arp they'd run show run.

In the second part the author dissects specific failures: because of a single “flipped” token, FlashAttention 2 addresses the wrong router interface — in production this could cause a serious network outage. He also shows that with tensor parallelism the result changes illogically (TP1 — correct, TP2 — error, TP4 — correct again), and NCCL is most often to blame.

The third part is devoted to “uncensored” fine-tunes of Qwen3.8. Some (Heretic-ARA, Huihui) barely spoil the working behavior, whereas others introduce crude errors: AEON, for instance, turned the port 5432 into 543ql and corrupted hostnames in commands.

The overall takeaway from the discussion in the comments is sober: small models shouldn't be trusted with critical tasks, and the pretty KLD figures on quant cards are meaningless without a full disclosure of the measurement methodology.

Source: Hacker News