What heuristics exist in gradient descent methods?

Question

Algorithms

At the university, a teacher during a laboratory defense asked how the concept of "heuristic" is related to the gradient descent algorithm, what kind of heuristics exist in gradient descent, etc. What did he mean? As far as I know, a heuristic approach is applied when choosing a direction in stochastic gradient descent, but this is hardly the answer to the question.

Answer 1

Most likely, he meant stochastic gradient descent. There are several tricks \ algorithms that allow you to speed up its convergence. The simplest is the use of the moment.