A 7MB Model Runs in the Browser CPU in 5ms, the Products You Can Build on Server-Free Inference

The Problem

A Show HN project called Ternlight hit the point precisely. A 7MB embedding model runs fully on the browser CPU through WebAssembly, about 5ms per call, with zero server calls. It produces 384-dimensional vectors and measures semantic similarity between texts. What it exposes is not a single tool but a gap. To add semantic search, intent classification, FAQ matching, or personalization to a web product today, most teams round-trip every input to a server embedding API. Each round-trip adds network latency, piles up token cost with usage, ships whatever the user typed off the device to someone else’s server, and dies completely in airplane mode or on a locked-down corporate network. The most common AI features sit on the heaviest possible infrastructure.

Why Now

Two curves just crossed. One is that models got small enough. Ternlight runs a Mamba-style architecture with binary weights and no floating-point math, small enough to ship a 5MB mini variant. The other is that the browser runtime matured: WASM already hits practical speeds and WebGPU backs it up. A third force compounds them. Cloud inference cost is collapsing as open weights arrive, but on-device does not lower the unit cost, it takes it to zero. Layer on the regulatory pressure to keep data on the device and never send it anywhere. Ultra-light on-device inference is standing right at the gap where an individual demo is about to harden into a standard product layer.

How to Build It

Push the embedding into the browser and keep the vector index local too. The user types, the model produces a vector inside the browser, similarity is computed against a local index, and the result comes back without a single network call. The product surface is wide: browser-extension semantic search, instant search inside offline note and document apps, a privacy-first FAQ bot that never exports its data, client-side form classification, server-free edge personalization. The tooling surface is real too: an on-device embedding SDK, a browser-local vector store, a quantization and distillation pipeline for small models.

flowchart LR
  A[User text] --> B[In-browser model, WASM on CPU]
  B --> C[On-device vector index]
  C --> D[Ranked results in about 5ms]
  B --> E[No server, private, offline]

Enter narrow. Start with a drop-in SDK aimed at one embedding-shaped workload, say document search, that installs in a single line, then add a local vector store with classification and ranking, then widen through framework integrations. Charge for the developer SDK per seat or by usage, and layer a subscription on the finished end products.

Success Criteria

First, the bundle-size budget. A few extra megabytes on first load cannot drive users away, so lazy loading and return-visit caching have to guarantee zero network from the second visit on. Second, an honest quality boundary. A small model is not as accurate as a frontier embedding. Trust is earned by splitting the workloads that on-device serves well from the ones a server still does better. Third, a narrow wedge. Do not chase all of on-device AI; whoever first proves “this is better local” on one workload is the one who gets to move to the next.