Stochastic gradient descent

Stochastic gradient descent (SGD) is an optimization technique that is widely used in machine learning applications, especially for training neural networks.

The goal of training a neural network is to find the optimal values of the model parameters (weights) that minimize a loss function, which measures how well the model fits the data. The loss function is usually defined as the average of a per-example loss function over the entire training set, such as the mean squared error or the cross-entropy. However, computing the loss function and its gradient (the slope of the loss function with respect to the weights) over the entire training set can be very expensive and time-consuming, especially when the training set is large and the model is complex, so iterative methods are required.

SGD is an iterative method that approximates the gradient of the loss function by using a randomly selected subset of the training set, called a batch or a minibatch. The batch size is a hyperparameter that controls how many examples are used to estimate the gradient in each iteration. A smaller batch size means a faster and more frequent update of the weights, but also a higher variance and noise in the gradient estimate. A larger batch size means a more accurate and stable estimate of the gradient, but also a slower and less frequent update of the weights.

Algorithm

The basic algorithm of SGD for neural network training is as follows:

• Initialize the weights randomly or with some heuristic method.
• Repeat until convergence or a maximum number of iterations is reached:
  • Shuffle the training set and divide it into batches of equal size.
  • For each batch:
    • Pass the input through the layers and obtain the output (the forward pass).
    • Determine the error in the output compared to the known answers. Propagate the error from the output to the input and obtain the gradient of the loss function with respect to each weight (the backward pass).
    • Update each weight by subtracting a fraction of its gradient, i.e., \\(w_{t+1} = w_t - \\eta \\nabla L(w_t)\\), where \\(w_t\\) is the weight at iteration \\(t\\), \\(\\eta\\) is the learning rate (another hyperparameter that controls how much the weights are changed in each iteration), and \\(\\nabla L(w_t)\\) is the gradient of the loss function at iteration \\(t\\).

Advantages and Disadvantages

SGD has several advantages over other optimization techniques for neural network training, such as:

• Efficiency: SGD can handle large-scale and sparse data sets efficiently, as it only requires a small subset of data to compute each update.
• Simplicity: SGD is easy to implement and tune, as it only requires two hyperparameters: batch size and learning rate.
• Flexibility: SGD can be combined with various extensions and variations, such as momentum, adaptive learning rate, regularization, etc., to improve its performance and convergence.

However, SGD also has some disadvantages and challenges, such as:

• Sensitivity: SGD is sensitive to feature scaling, initialization, hyperparameter selection, etc., as they can affect its convergence rate and accuracy.
• Local minima: SGD may get stuck in local minima or saddle points of non-convex loss functions, which are common in neural networks.
• Oscillation: SGD may oscillate around the optimal solution due to its high variance and noise in the gradient estimate.

Therefore, it is important to monitor and evaluate the performance of SGD on different data sets and tasks, and to apply appropriate techniques to avoid or mitigate its limitations.

See Also

• wikipedia:Stochastic gradient descent
• Stochastic Gradient Descent — scikit-learn 1.3.0 documentation. [1]
• Stochastic gradient descent - Cornell University Computational .... [2]
• Stochastic Gradient Descent Algorithm With Python and NumPy. [3]
• Difference Between Backpropagation and Stochastic Gradient Descent .... [4]