How to commit a transaction in a loop?

Question

Django

I have a project that started with Django 1.4. Once an hour, a special management command parses a json file with about 200,000 entries and loads them into the database. Naturally, this is a very slow process. Previously, it was accelerated by manual transaction management: before the cycle, the transaction began and was committed every 1000 records. The trouble is that now the project is running on 1.9, in which this method no longer works. In 1.9, there is a decorator and a transaction.atomic context manager, which for obvious reasons is not applicable in a cycle, and a pair of transaction.set_autocommit(False) and transaction.commit() works only on the most primitive operations, such as sequential creation of new unrelated records in the database. Not my case. What to do? It would not be desirable to get rid of ORM'a very much.

Answer 1

# data - твой датасет из 200000 элементов
while data:
    with atomic():
        do_something_atomic(data[:1000])
        del data[:1000]

But I still don't understand why you need transactions. Isn't it better to put 1000 values ​​into one insert?