Answer the question
In order to leave comments, you need to log in
How to speed up np.dot()?
Colleagues, good day!
The following interests:
I from a DB receive df in the size of 10 million lines on n columns;
I select one column from this df further;
I make a copy of it consisting of inverse values;
I make the resulting copy as a string;
Then I multiply in a matrix way through np.dot () the column by the formed line.
-------------------------------------------------- -------------------------------------------------- --------------
Chunk - the first way to speed up, to multiply by parts - matrices chunk_size by 10 million
There are other options how it could be accelerated? Perhaps there is some optimal value for the chunk parameter?
chunk_size = 25
df_values = DF.to_numpy()
v = df_values[:, criteria_number - 1:criteria_number]
length = v.shape[0]
kolvo_shagov = int(length / chunk_size) + int(
(length % chunk_size) / ((length % chunk_size - 1)))
b = 1 / v
if vector is None:
vector = np.zeros([v.shape[0], 1])
i = -1
for shag in range(kolvo_shagov):
i += 1
matrix = v[chunk_size * i:chunk_size * (i + 1), :].dot(bT)
gm = stats.gmean((matrix), axis=1)
gm = gm.reshape([len(gm), 1])
vector[chunk_size * i:chunk_size * (i + 1)] = gm
vector_sum = vector.sum()
vector = vector / vector_sum
Answer the question
In order to leave comments, you need to log in
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question