Gradient descent is an iterative optimization algorithm where on each iteration, or step, the parameters of the model are updated so that the loss function sequentially moves in the direction of its minimum until a convergence criteria is achieved. The size of the step taken to reach the optimum is determined by the learning rate parameter. If the learning rate is large, the algorithm might converge faster, but it also risks overstepping the minimum and oscillating in its vicinity. On the other hand, a small learning rate will take the algorithm a long time to train.
What is Gradient Descent?
Help us improve this post by suggesting in comments below:
– modifications to the text, and infographics
– video resources that offer clear explanations for this question
– code snippets and case studies relevant to this concept
– online blogs, and research publications that are a “must read” on this topic
Partner Ad
