# How to run AI models directly on your PC and browser: generation speed is governed by memory bandwidth

- Link: https://zenn.dev/port_inc/articles/ff974c7456f4db
- Discussion: https://startupxo.com/en/1408/
- Posted by startupxo at 2026-11-17T12:28:20.676Z
- 0 points, 0 comments, kind story, origin curated
- Written in ko, machine translated to en

## Summary

- An AI model is essentially a file containing a huge number of values called weights. You can choose to run it in the cloud, on an in-house server, or on a personal device, and if you run it on your device, your input never leaves the device. After the first download, it can be used without a connection.
- The speed of generating text one token at a time is governed by memory bandwidth rather than compute performance. By the author's calculation, llama.cpp benchmark measurements reached 6 to 8 tenths of the theoretical upper limit, which is bandwidth divided by model size. When you feed in a long document, the KV cache (storage for earlier computation results) grows and speed drops.
- With 4-bit quantization (a method of coarsely reducing numbers to cut size), examples were given such as Qwen3.5-4B for GPU memory of the 8GB class and Qwen3.6-27B for the 24GB class. Epoch AI compared that the best model running on 1 RTX 5090 cards lags the top-tier models by an average of 6 to 12 months.
- In the browser, you can run models with WebLLM, which is based on WebGPU, and wllama, the browser version of llama.cpp. The author explains that small models can be used for classification, reading only the probabilities of options without writing text, or can be tailored to a purpose with distillation and LoRA (a method of additionally training only some of the weights).

## Sources

- Zenn: https://zenn.dev/port_inc/articles/ff974c7456f4db

## Comments (0)

No comments yet.

JSON: https://startupxo.com/api/v1/item.json?id=1408&lang=en
