Search
English

How to run AI models directly on your PC and browser: generation speed is governed by memory bandwidth(zenn.dev)

startupxo Discuss

Writing language: KoreanRead in the original language

Plain summary

- An AI model is essentially a file containing a huge number of values called weights. You can choose to run it in the cloud, on an in-house server, or on a personal device, and if you run it on your device, your input never leaves the device. After the first download, it can be used without a connection. - The speed of generating text one token at a time is governed by memory bandwidth rather than compute performance. By the author's calculation, llama.cpp benchmark measurements reached 6 to 8 tenths of the theoretical upper limit, which is bandwidth divided by model size. When you feed in a long document, the KV cache (storage for earlier computation results) grows and speed drops. - With 4-bit quantization (a method of coarsely reducing numbers to cut size), examples were given such as Qwen3.5-4B for GPU memory of the 8GB class and Qwen3.6-27B for the 24GB class. Epoch AI compared that the best model running on 1 RTX 5090 cards lags the top-tier models by an average of 6 to 12 months. - In the browser, you can run models with WebLLM, which is based on WebGPU, and wllama, the browser version of llama.cpp. The author explains that small models can be used for classification, reading only the probabilities of options without writing text, or can be tailored to a purpose with distillation and LoRA (a method of additionally training only some of the weights).

Read source ↗

Found on

Zenn ↗ / Published by source

Have you run into something like this? Tell us in a line or two what you did. You do not need to know the product.

Sign in to comment

Keyboard shortcuts

Choose a post with J and K or the arrow keys, then press Enter.

K ↑ / J ↓
Previous post / next post
Enter C
Open summary and comments for the selected post
O
Open the original link of the selected post
/
Search
?
Show this help
Tab
Move to the submit or comment button, then press Enter

Type normally in text fields. Tab and Enter are always available.