{"id":833,"kind":"story","title":"How to Pretrain Transformers Without Backpropagation Using Token-wise Activation Noise","url":"https://qlabs.sh/research/dust","host":"qlabs.sh","by":"startupxo","score":0,"comments":0,"time":"2026-10-06T08:39:02.529Z","origin":"curated","language":{"source":"ko","reading":"en","translated":true},"summary":["Dust is a new algorithm that applies random perturbations to the neural activation values of each token and estimates gradients by measuring and averaging how much these perturbations reduce loss on a per-token basis.","Traditional evolution strategies required a separate forward pass for each weight, but Dust independently applies perturbations at the token level, evaluating thousands of perturbations in a single forward pass, greatly improving computational efficiency.","Experiments by the Q Labs research team showed that Dust improved efficiency by 1000 to 10,000 times compared to the existing weight-space evolution strategy EGGROLL when training on over 1 million tokens; this efficiency gain became more pronounced as model size increased.","They also confirmed that gradients estimated by evolution strategies matched backpropagation results well up to scales of 1 billion tokens; however, currently, the computational efficiency is not yet sufficient to fully replace backpropagation."],"discussion":"https://startupxo.com/en/833/","markdown":"https://startupxo.com/en/833.md","api":"https://startupxo.com/api/v1/item.json?id=833&lang=en","text":null,"edited":null,"sources":[{"site":"hn","name":"Hacker News","url":"https://news.ycombinator.com/item?id=49970871","score":201,"comments":49,"observed_at":"2026-10-06T10:30:50Z"}],"thread":[]}