A 7MB Model Runs in the Browser CPU in 5ms, the Products You Can Build on Server-Free Inference

A 7MB model runs on a browser CPU in five milliseconds

A Show HN project called Ternlight hit the point precisely. A 7MB embedding model runs fully on the browser CPU through WebAssembly, about 5ms per call, with zero server calls. It produces 384-dimensional vectors and measures semantic similarity between texts. What it exposes is not a single tool but a gap. To add semantic search, intent classification, FAQ matching, or personalization to a web product today, most teams round-trip every input to a server embedding API. Each round-trip adds network latency, piles up token cost with usage, ships whatever the user typed off the device to someone else’s server, and dies completely in airplane mode or on a locked-down corporate network. The most common AI features sit on the heaviest possible infrastructure.

Two curves just crossed

Two curves just crossed. One is that models got small enough. Ternlight runs a Mamba-style architecture with binary weights and no floating-point math, small enough to ship a 5MB mini variant. The other is that the browser runtime matured: WASM already hits practical speeds and WebGPU backs it up. A third force compounds them. Cloud inference cost is collapsing as open weights arrive, but on-device does not lower the unit cost, it takes it to zero. Layer on the regulatory pressure to keep data on the device and never send it anywhere. Ultra-light on-device inference is standing right at the gap where an individual demo is about to harden into a standard product layer.

Push both the embedding and the vector index into the browser

Push the embedding into the browser and keep the vector index local too. The user types, the model produces a vector inside the browser, similarity is computed against a local index, and the result comes back without a single network call. The product surface is wide: browser-extension semantic search, instant search inside offline note and document apps, a privacy-first FAQ bot that never exports its data, client-side form classification, server-free edge personalization. The tooling surface is real too: an on-device embedding SDK, a browser-local vector store, a quantization and distillation pipeline for small models.

flowchart LR
  A[User text] --> B[In-browser model, WASM on CPU]
  B --> C[On-device vector index]
  C --> D[Ranked results in about 5ms]
  B --> E[No server, private, offline]

Enter narrow. Start with a drop-in SDK aimed at one embedding-shaped workload, say document search, that installs in a single line, then add a local vector store with classification and ranking, then widen through framework integrations. Charge for the developer SDK per seat or by usage, and layer a subscription on the finished end products.

The first few megabytes must not cost you the visitor

First, the bundle-size budget. A few extra megabytes on first load cannot drive users away, so lazy loading and return-visit caching have to guarantee zero network from the second visit on. Second, an honest quality boundary. A small model is not as accurate as a frontier embedding. Trust is earned by splitting the workloads that on-device serves well from the ones a server still does better. Third, a narrow wedge. Do not chase all of on-device AI; whoever first proves “this is better local” on one workload is the one who gets to move to the next.