Editing Learning Rate (section)

== Learning Rate Scheduling ==
'''Learning rate scheduling''' or '''learning rate decay''' is a strategy to adjust the learning rate during the training process. The idea is to start with a relatively high learning rate to benefit from fast learning, and then reduce the learning rate as training progresses to allow the model to converge.

There are several strategies for learning rate scheduling:
* '''Step Decay''': The learning rate is reduced by a factor after a fixed number of epochs.
* '''Exponential Decay''': The learning rate is reduced exponentially, often after each batch or epoch.
* '''1/t Decay''': The learning rate is reduced as the inverse of the square root of the epoch number.
* '''Warm-up and Cool-down periods''': The learning rate is initially increased for a certain number of epochs (warm-up), and then gradually decreased towards the end of training (cool-down).

The choice of learning rate and schedule can greatly impact the quality of the final model and is often determined through experimentation and [[Hyperparameter tuning]].