Search
English

How to Pretrain Transformers Without Backpropagation Using Token-wise Activation Noise(qlabs.sh)

startupxo Discuss

Writing language: KoreanRead in the original language

Plain summary

- Dust is a new algorithm that applies random perturbations to the neural activation values of each token and estimates gradients by measuring and averaging how much these perturbations reduce loss on a per-token basis. - Traditional evolution strategies required a separate forward pass for each weight, but Dust independently applies perturbations at the token level, evaluating thousands of perturbations in a single forward pass, greatly improving computational efficiency. - Experiments by the Q Labs research team showed that Dust improved efficiency by 1000 to 10,000 times compared to the existing weight-space evolution strategy EGGROLL when training on over 1 million tokens; this efficiency gain became more pronounced as model size increased. - They also confirmed that gradients estimated by evolution strategies matched backpropagation results well up to scales of 1 billion tokens; however, currently, the computational efficiency is not yet sufficient to fully replace backpropagation.

Read source ↗

Found on

Hacker News ↗ / 201 votes / 49 comments

Have you run into something like this? Tell us in a line or two what you did. You do not need to know the product.

Sign in to comment

Keyboard shortcuts

Choose a post with J and K or the arrow keys, then press Enter.

K ↑ / J ↓
Previous post / next post
Enter C
Open summary and comments for the selected post
O
Open the original link of the selected post
/
Search
?
Show this help
Tab
Move to the submit or comment button, then press Enter

Type normally in text fields. Tab and Enter are always available.