How to form a correct stochastic gradient descent algorithm?

Question

Mathematics

Hello, it is not possible to implement the execution of the step algorithm for stochastic gradient descent.
Here is the model I used:

Here is the step algorithm:

def stochastic_gradient_step(X, y, w, train_ind, eta=0.01):
    r = []
    X_train = X[train_ind]
   
    r.append(w[0] - 0.5 * eta * X_train[0] * (w[0] * X_train[0] + w[1] * X_train[1] + w[2] * X_train[2] + w[3]* X_train[3] - y[train_ind]))
    r.append(w[1] - 0.5 * eta * X_train[1] * (w[0] * X_train[0] + w[1] * X_train[1] + w[2] * X_train[2] + w[3]* X_train[3] - y[train_ind]))
    r.append(w[2] - 0.5 * eta * X_train[2] * (w[0] * X_train[0] + w[1] * X_train[1] + w[2] * X_train[2] + w[3]* X_train[3] - y[train_ind]))
    r.append(w[3] - 0.5 * eta * X_train[3] * (w[0] * X_train[0] + w[1] * X_train[1] + w[2] * X_train[2] + w[3]* X_train[3] - y[train_ind]))
    return r

I suspect the formula is wrong.

Answer 1

You can, for example, like this:
def stochastic_gradient_step(X, y, w, train_ind, eta=0.01):
l=len(y)
return w+(2*eta*X[train_ind]/l)*(y[train_ind]-sum ((xi*wi) for xi,wi in zip(X[train_ind],w)))
If I understand correctly what problem you are solving :)
But I have a similar problem - gradient descent works, the resulting weight vector is "like the truth", the mean square error function is implemented correctly. But the answer is no!

Answer 2

There is a typo in the task, if nothing is corrected in the task, then you need to apply a different formula:

Answer 3

the topicstarter does not have a division by l (in your case by 4). Without this, the gradient is incorrectly calculated, so there will be no descent.

Answer 4

matrunich sergey Also faced this problem. Can you share what was the problem? Already checked everything...