Lion

Lion (Evolved Sign Momentum Optimizer) is a next-generation optimization algorithm introduced in 2023 by researchers at Google DeepMind. Named after the concept of "Evolved Sign Momentum," Lion (Layer-wise Optimizer) aims to provide efficient, accurate, and low-resource consumption optimization, particularly for large-scale models. It has achieved notable success in computationally intensive networks such as transformer-based large language models.

Unlike classical optimization methods such as Adam and SGD, Lion uses only the sign of the gradients to update weights. This approach enhances memory efficiency and reduces oscillations.

At the formula level, the Lion algorithm is expressed as:

$m_{t+1}={\beta }_1\cdot m_t+(1-{\beta }_1)\cdot \nabla L({\theta }_t)$

Lion achieves efficient optimization through a simplified parameter update that relies on the direction of the momentum vector.