Answer the question
In order to leave comments, you need to log in
How to get rid of accumulation of postgres temporary tables?
When executing recursive queries, postgres creates temporary tables that remain on disk after the query completes until the entire script completes, which is done in the form of a django management command using multiprocessing. You cannot close the connection to the database in the parent process - it is used as an iterative source of initial data.
The query is executed as a raw-query through a cursor like this:
def process_datum(datum):
with db.connections['world'].cursor() as cursor:
cursor.execute("SELECT ... from query_function(%s)", (datum.id,))
rows = cursor.fetchall()
for row in rows:
try:
...
A_Model.objects.create(...)
except db.IntegrityError as e:
logger.warning("%s: %s", path, e)
select ... from features limit 1000 offset xxxx;
with recursive recursion(child_id, parent_id, node_id, path) as (
select h.child_id, h.parent_id, h.parent_id as node_id, ARRAY[h.parent_id]
from hierarchy h
where h.direct=true and h.child_id=$1
union all
select h.child_id, h.parent_id, r.node_id, r.path || ARRAY[h.parent_id]
from recursion r join hierarchy h on h.child_id = r.parent_id
where h.direct=true and h.parent_id != r.node_id
)
select * from recursion
insert into hierarchy (parent_id, child_id, direct, path) values (%d, %d, false, %s::bigint[])
Answer the question
In order to leave comments, you need to log in
The problem is in the infinite recursion due to looped data.
The recursive query needs to be fixed.
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question