A
A
andrepavlov2018-05-16 11:12:37
MySQL
andrepavlov, 2018-05-16 11:12:37

How to speed up executemany in Python on INSERT IGNORE (mysql)?

Python 3, mysql 5.7, mysql.connector. I insert a large number of values ​​into the database (~ 5 million rows, in batches of 5000 rows) through executemany. Some values ​​are not unique. In this case, you just need to skip this insert / update / replace it. If you use INSERT - if there is at least one error in a batch of 5000 rows, if there is a duplicate, the entire batch is not inserted. Insertion speed is fast. To solve the problem, I tried INSERT IGNORE, the speed drops many times, the impression is that queries are executed separately, not in batches. How can this problem be solved without sacrificing speed?

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question