Why is List(QuerySet) conversion so slow in Django?

Question

Django

I love writing raw queries in Django. And it’s not that I don’t know how to cook ORM (although I don’t care), but sometimes even in SQL it’s impossible to write what you want (see my previous question). And now, having rolled up another raw with a bunch of all sorts of JOINs from six tables (call the result of its execution QueySet2 ), and looking at the profiler how much it eats, I decided to drag all the insane number of repeating fields from the previous raw query (let's call its result QuerySet1 ). Anyway, it sits in the cache, the fields are the same, the id is not repeated ... in my case, it seemed convenient to me. All the same, you need to shovel this QueySet2 to send it to the template, throwing out 90% of the duplicate data ...
To make it clearer, something like the formation of a table is going on... I "flip through" the QueySet2 records , arrange them into columns and rows, add to each column and row some reference data obtained from the calculated fields and fields of the linked tables of the heaped QueySet2 .. But the descriptions of these columns, rows and each cell can be taken from QyerySet1 . And now I am accessing QyerySet1 like this: QyerySet1[i].id ... There are only 35 such calls in the cycle (counted). And it will take 4.71 seconds . But since to determine the length of QyerySet1 , it was still necessary -- to scroll through the cells in rows -- to determine its length len(list(QyerySet1)), then for the sake of experiment, I decided to check how it would work if we first do ListSet1 = list(QyerySet1) and then apply ListSet1[i].id .... And the same 35 queries were completed in 0.25 seconds! Those. 18 times faster!
The time was measured without taking into account the execution of the SQL itself (I looked at the SQL execution time in the SQL profiler and subtracted from both measurements ... If you do not subtract the difference will be 12 times, which is also decent).
After experimenting, I made sure that the call to QyerySet1[i].id seems to turn into List(QyerySet1)[i].id every time inside Django (I’m not a programmer, so I don’t know how to get into the source code, and I draw conclusions simply by comparing the execution time different options).
The question is why is it so slow? How to win it?

Answer 1

Learn to work with Django
Learn to work with Python (in particular, pep8 and iterable)
Learn to use ORM (only, defer, select/prefetch related) and use raw as an exception, and not vice versa
Learn to build a database architecture (in particular,

a bunch of all sorts of JOINs from six tables

)
And there will be no such slow requests.
And the answer to the question will be something like this:
If QuerySet1 is not yet fetched, then QyerySet1[i].id makes a query to the database (35 queries in total)
list(QyerySet1) makes a fetch and .id is already taken from the operative.
PS Use django-debug-toolbar. You will see all your SQL queries and their execution time.