Most deep learning practitioners reach for Adam by default. But when training on tasks with noisy or sparse gradients (like GANs, reinforcement learning, or large-scale language models), Adam can sometimes struggle with sudden large gradient updates that destabilize training.
Yogi won't replace Adam everywhere, but it's an excellent tool to keep in your optimizer toolbox – especially when gradients get wild. yogi optimizer
Enter (You Only Gradient Once).
Yogi adds a tiny bit of compute per step and may need slightly more memory. In practice, it's negligible for most models. Most deep learning practitioners reach for Adam by default
Developed by researchers at Google and Stanford, Yogi modifies Adam's adaptive learning rate mechanism to make it more robust to noisy gradients. Enter (You Only Gradient Once)
Try it on your next unstable training run. You might be surprised. 🚀
Beyond Adam: Meet Yogi – The Optimizer That Tames Noisy Gradients
join our email list
We will send you occasional updates about our latest products and deals, but it won’t be annoying and you can unsubscribe any time. See our privacy policy
Copyright © 2020 Koji Color.
All other trademarks and copyrights are the property of their respective owners.