Search
English

How many people can use an internal document search chatbot on L40S GPU 1 unit, measured by model and setting(zenn.dev)

startupxo Discuss

Writing language: KoreanRead in the original language

Plain summary

- The author measured how many people can use a chatbot that finds and answers from internal documents at the same time when it runs on one GPU (NVIDIA L40S 48GB). Inputs averaged about 7,050 tokens, and the criteria were a first token within 10 seconds and at least 20 tokens per second. - The number of people who met the criteria ranged from 1 to 60 depending on the configuration. The highest was 60 people with Gemma 4 26B-A4B compressed to 4 bits and run on vLLM, and the lowest was 1 people with Qwen3.8-27B run on llama.cpp. - Compressing to 4 bits on vLLM increased the number of people while the average accuracy dropped slightly. The task of copying unit prices from long documents did not get worse, but for the tax-inclusive total calculation, every configuration got only 0 of 50 questions right. - Placing the retrieval embedding model on the same GPU met the criteria when only question search was done. When document embedding ran continuously, output speed dropped by 41 to 46%, and even 1 people fell short of the criteria. Limitations: 2 models, 1 GPU, and accuracy measured 1 times per configuration.

Read source ↗

Found on

Zenn ↗ / Published by source

Have you run into something like this? Tell us in a line or two what you did. You do not need to know the product.

Sign in to comment

Keyboard shortcuts

Choose a post with J and K or the arrow keys, then press Enter.

K ↑ / J ↓
Previous post / next post
Enter C
Open summary and comments for the selected post
O
Open the original link of the selected post
/
Search
?
Show this help
Tab
Move to the submit or comment button, then press Enter

Type normally in text fields. Tab and Enter are always available.