Why on-device LLM assistants have felt slow, and how NPUs fix it

If you've tried building an on-device AI assistant that reads the screen, summarizes a document, or automates a multi-step task, you've probably hit the same wall: the first response takes forever, even on a capable phone. That delay isn't the model "thinking" in a useful sense. It's the prefill stage, the part where the model processes your prompt before it can generate a single token, and it's usually the actual bottleneck.

A paper from Peking University's Key Lab of HCST, presented at ASPLOS 2025, quantifies exactly how bad this gets and proposes a concrete fix. The authors, Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Mengwei Xu, and Xuanzhe Liu, measured that for UI automation tasks, prefill accounts for 94.4% to 98.8% of total end-to-end latency. In their example, a single step for a 1.8B-parameter model took 8.1 seconds, meaning a routine 5-step UI task took over 40 seconds before producing a useful result. That's not a model quality problem. It's an architecture problem: commodity phones weren't designed to run this workload efficiently on CPU or GPU alone.

Their system, called llm.npu, targets the Neural Processing Unit that already ships in most modern phones but sits underused for LLM inference specifically because NPUs are built for fixed-shape, batch-oriented workloads, not variable-length prompts and irregular activation patterns. The paper's contribution is a set of three restructuring techniques that make prompts and models NPU-friendly: chunking variable-length prompts into fixed-size pieces while preserving dependencies, pulling outlier tensor values out to run on CPU/GPU in parallel, and scheduling transformer blocks out-of-order across CPU, GPU, and NPU based on which hardware suits them best.

The measured results are substantial. Compared to five established baselines, including llama.cpp, TFLite, MNN, MLC-LLM, and PowerInfer-v2, llm.npu achieves a 22.4x faster average prefill speed and 30.7x average energy savings, with up to 32.8x speedup in a real end-to-end application. It's also, per the authors, the first system to break 1,000 tokens per second of prefill throughput for a billion-parameter model on commodity off-the-shelf hardware, while keeping accuracy loss under 1% compared to full-precision FP16.

Why this matters beyond the benchmark: energy and latency are the two constraints that actually decide whether an on-device AI feature ships or gets cut. A feature that drains battery or makes users wait ten seconds for a screen read doesn't survive a product review, no matter how good the model is underneath. Results like a 30x energy reduction change what's viable to build on-device at all, from real-time UI agents to offline document summarization to privacy-preserving assistants that never send a prompt to a server.

The practical takeaway for teams building mobile or AI products isn't "go implement custom NPU kernels tomorrow." It's that the hardware headroom for genuinely responsive on-device AI already exists in the silicon shipping in current phones. The bottleneck has been software and scheduling, not raw compute. As NPU-aware inference techniques like this one make their way into mainstream runtimes and SDKs, on-device features that felt impractical a year ago, because of latency or battery cost, are becoming a realistic default rather than an experimental stretch goal. Teams evaluating on-device AI roadmaps should treat NPU offloading as an architecture decision worth planning for now, not a future nice-to-have.

Source: Xu, Zhang, Yang, Liu, Huang, Xu, and Liu, "Fast On-device LLM Inference with NPUs," ASPLOS 2025.