On-Device Machine Learning

This is a note from my public notes. View the canonical version: On-Device Machine Learning.

Running machine learning models locally on the user's device rather than sending data to a remote server for inference. Also called edge ML or client-side ML. Core value proposition of the WebMachineLearning initiative and the Prompt API.

Why It Matters

Key Enablers

• Hardware acceleration: NPUs, GPUs, and specialized ML chips in modern devices
• Model compression: quantization, pruning, and distillation make large models fit on-device
• Browser APIs: WebNN API, Prompt API give web apps access to device hardware
• OS-level models: browsers can surface OS-provided models (e.g., Apple's Core ML, Google's Gemini Nano on Android)

Trade-offs

Advantages:

• Privacy by default — no data transmitted
• Works offline
• No API costs
• Low latency for real-time use cases

Limitations:

• Model capability bounded by device compute
• Large model downloads for first run
• Consistency varies across devices and hardware
• Smaller context windows than cloud models

Web Platform Connection

WebMachineLearning standardizes browser access to on-device ML. WebNN API provides the low-level hardware interface; Prompt API and Writing Assistance APIs expose higher-level LLM capabilities.

References

• https://github.com/webmachinelearning
• https://www.w3.org/groups/wg/webmachinelearning/

Related

• Machine Learning (ML)
• AI Inference
• AI Privacy
• WebMachineLearning
• Prompt API
• WebNN API
• Browser-Provided Language Models
• Large Language Models (LLMs)
• Edge AI
• Edge Computing
• Neural Processing Unit (NPU)
• Gemini Nano
• Transformers.js
• ONNX Runtime Web
• Web Assembly (WASM)
• WebGPU

About Sébastien

Ready to get to the next level?