`memory-efficiency` Related Article Results

Adafactor is an efficient, low-memory optimization algorithm developed by Google, specifically designed for memory-intensive models such as large-scale language models. It was first introduced in 2018 in the paper titled "Adafactor: Adaptive Learning Rates with Sublinear Memory Cost". Like the Adam algorithm, Adafactor performs moment-based updates but computes second-moment estimates using significantly less memory, thereby enabling the training of large models. Adafactor Optimization Algorithm