On-Device AI Models Are Getting Scary Good — Here's Why It Matters

An 8-billion-parameter model at 4-bit quantization is about 4.5 GB of weights. Add a kilobyte or two per token of KV cache and you are looking at roughly 5 GB resident for a usable context length. That fits in the unified memory of a base-model laptop from three years ago, and it fits on a current flagship phone with room left for the OS. The interesting question stopped being "can it fit" somewhere around 2024. The interesting question now is how fast it runs, and the answer has almost nothing to do with the chip's FLOPS number.

The arithmetic that decides local tokens per second

Autoregressive decoding at batch size 1 has a property that makes it trivially easy to predict: to produce one token, the hardware has to read every weight in the model exactly once. There is no reuse. Each weight participates in one multiply against a single activation vector and then it is done. That is an arithmetic intensity of about 1 FLOP per byte loaded, which is two orders of magnitude below what modern accelerators need to stay busy.

So the ceiling is a division:

tokens_per_sec_max = memory_bandwidth / bytes_read_per_token

# Apple M-series base tier, ~100 GB/s, 8B model at 4-bit (~5 GB):
100e9 / 5e9  = 20 tok/s

# M-series Max tier, ~400 GB/s, same model:
400e9 / 5e9  = 80 tok/s

# Same 100 GB/s machine, but FP16 weights (~16 GB):
100e9 / 16e9 = 6.25 tok/s

Real throughput lands at 60 to 80 percent of that ceiling once you account for cache misses, attention over the KV cache, and sampling overhead. But the ceiling is the ceiling, and it explains something people find surprising: quantizing from FP16 to 4-bit gives roughly a 3x speedup on decode even though the arithmetic did not get cheaper. You did not make the math faster. You made the model smaller, and the model's size in bytes is the denominator.

This is also why the same 8B model runs at wildly different speeds on machines with similar TFLOPS ratings. Bandwidth is the spec sheet number that predicts local performance. If you are choosing hardware to run models on, sort by GB/s and ignore the rest.

Why server inference behaves completely differently

On a server, requests get batched. Thirty-two sequences decode simultaneously against the same weight matrices, so one read of the weights produces 32 tokens instead of 1. Arithmetic intensity goes from ~1 to ~32 FLOPs per byte and the workload crosses over into compute-bound territory, which is where GPUs are designed to live. That crossover is the entire reason hosted inference is cheap per token and local inference is not: the datacenter amortizes the weight read across users, and you cannot.

The practical consequence is that local inference will never win on throughput per dollar and does not need to. Prefill is a separate story and is compute-bound even at batch 1, since the whole prompt is processed as a matrix multiply, which is why a long prompt has a noticeable time-to-first-token on a laptop and then streams smoothly afterward. If you want to see all of this instrumented, the llama.cpp benchmark output separates prompt-eval from token-eval timings for exactly this reason.

What actually moves local

Twenty tokens per second is faster than most people read. That is enough for a real class of work, and the class is defined by two properties rather than by capability:

• Data that should not leave the device. Full-text search over personal email, transcription of a medical conversation, code completion against a repo under NDA. The model does not need to be frontier-class. It needs to be adequate and it needs to not phone home.
• Always-on background work where per-call latency is irrelevant. Watching a filesystem and summarizing changes, extracting structured entries from a clipboard history, tagging screenshots. Nobody is waiting. A model running continuously at 20 tok/s at zero marginal cost beats an API call that costs money every time it fires, which is why these workloads have basically not existed until now.

What does not move local is anything where you would notice the quality gap. An 8B model is not going to do multi-file refactoring or careful long-form reasoning, and pretending otherwise wastes everyone's time. The realistic architecture is hybrid: a small local model handling the high-frequency, privacy-sensitive, latency-tolerant tier, escalating to a frontier model when the task warrants the round trip.

The part hybrid breaks

Split inference across a local model and a hosted one and you immediately have a state problem. The local model on your laptop learned that you moved the ingestion service to Go. The hosted model you escalate to has never heard of it. The local model on your phone is a separate process with a separate scratch space and knows neither.

Weights are not the state. Context is the state, and context is exactly the thing that does not replicate for free. A model is a few gigabytes of read-only file you can copy to every device you own once. Your accumulated context is small, mutable, written from multiple devices concurrently, and needs conflict resolution, which makes it a genuinely harder distributed systems problem than distributing the model ever was.

Local inference solves where the compute happens. It does nothing about where the memory lives, and the second problem gets harder as the first one gets easier.

This is the shape of the trend worth watching. As on-device models get good enough for a real tier of work, the number of independent inference endpoints in a person's life goes up, not down: a phone model, a laptop model, two or three hosted frontier models. Every one of them starts from zero unless something outside the model holds the state.

That layer is what Unimatrix is: a memory store the local and hosted models both read from over MCP, so the phone model and the frontier model are working from the same facts. Running it on your own hardware keeps the privacy property that made local inference appealing in the first place, which is covered in security.