{"id":1193,"kind":"story","title":"Using a mask in PyTorch attention inference spiked GPU memory; the author reduced it by changing the fallback path","url":"https://zenn.dev/genelab_999/articles/387bf151caf716","host":"zenn.dev","by":"startupxo","score":0,"comments":1,"editorial_comments":1,"time":"2026-10-24T12:37:57.526Z","origin":"curated","language":{"source":"ko","reading":"en","translated":true},"summary":["The model Xiaomi-Lens-0, which creates masks marking image regions, switches to the MultiheadAttention path when flash-attn is not installed. When a mask was passed, that path created a matrix sized as 16 heads × the square of the patch count.","According to the author's measurements, for a 3-megapixel image the extra VRAM peak in the mask step was 18.02GB on the original path and 2.28GB on the SDPA path. The test environment was RTX 5090 32GB, Windows 11, with flash-attn not installed.","The PyTorch documentation says an optimized path is used when need_weights=False. However, when the author read the CUDA source, the path was not entered when a mask was present. In CPU profiling, the same layer in training mode used SDPA, so the issue reproduced only in eval inference.","The author reported that across a comparison of 6 samples, tokens and boxes matched, and mask IoU ranged from 0.9976 to 0.9998. This is a limited comparison, not a full quality evaluation."],"discussion":"https://startupxo.com/en/1193/","markdown":"https://startupxo.com/en/1193.md","api":"https://startupxo.com/api/v1/item.json?id=1193&lang=en","text":null,"edited":null,"sources":[{"site":"rss","name":"Zenn","url":"https://zenn.dev/genelab_999/articles/387bf151caf716","score":null,"comments":null,"observed_at":"2026-10-24T12:37:14Z"}],"thread":[{"id":464,"by":"startupxo","score":0,"time":"2026-10-24T12:37:57.526Z","text":"O: An engineer who runs an in-house inference server could measure the GPU memory peak once with a padding mask, under eval and inference_mode, and compare it with the SDPA path.\nX: The figure is the extra peak added by the mask step in the author's environment, not total VRAM. First, it is worth checking that results can differ if the GPU, resolution, or model structure is different.","editorial":true,"replies":[]}]}