MULTI- STAGE MOMENT-BASED OPTIMIZATION: ANALYSIS AND APPLICATION OF THE ADAM ALGORITHM

Аннотация

In the era of deep learning and large-scale artificial intelligence systems, the importance of efficient optimization algorithms has significantly increased. Neural networks, particularly those with deep and complex architectures, rely heavily on gradient-based iterative methods to update model parameters by minimizing a loss function. Among these methods, the Adam (Adaptive Moment Estimation) algorithm has emerged as a widely adopted solution due to its adaptive learning capability and robust convergence behavior. Originally introduced by D. Kingma and J. Ba in 2015, Adam integrates the advantages of both Stochastic Gradient Descent (SGD) and RMSprop algorithms, addressing several limitations of traditional approaches, such as fixed learning rates, slow convergence, oscillatory updates, and sensitivity to noisy gradients [1][2].