M
M
Maxim Vasiliev2017-01-25 11:05:44
Django
Maxim Vasiliev, 2017-01-25 11:05:44

How to get rid of accumulation of postgres temporary tables?

When executing recursive queries, postgres creates temporary tables that remain on disk after the query completes until the entire script completes, which is done in the form of a django management command using multiprocessing. You cannot close the connection to the database in the parent process - it is used as an iterative source of initial data.
The query is executed as a raw-query through a cursor like this:

def process_datum(datum):
   with db.connections['world'].cursor() as cursor:
        cursor.execute("SELECT ... from query_function(%s)", (datum.id,))
        rows = cursor.fetchall()
   for row in rows:
        try:
            ...
            A_Model.objects.create(...)
        except db.IntegrityError as e:
            logger.warning("%s: %s", path, e)

The `process_datum` function is called from the pool worker, the `query_function` function on the server side implements a recursive query.
Dofigischa of these requests and temporary tables litter the entire disk.
How to be?
PS
Original query: Recursive query:
select ... from features limit 1000 offset xxxx;
with recursive recursion(child_id, parent_id, node_id, path) as (
  select h.child_id, h.parent_id, h.parent_id as node_id, ARRAY[h.parent_id] 
  from hierarchy h
  where h.direct=true and h.child_id=$1
union all
  select h.child_id, h.parent_id, r.node_id, r.path || ARRAY[h.parent_id]
  from recursion r join hierarchy h on h.child_id = r.parent_id
  where h.direct=true and h.parent_id != r.node_id
)
select * from recursion

Creating request:
insert into hierarchy (parent_id, child_id, direct, path) values (%d, %d, false, %s::bigint[])

Answer the question

In order to leave comments, you need to log in

1 answer(s)
M
Maxim Vasiliev, 2017-01-25
@qmax

The problem is in the infinite recursion due to looped data.
The recursive query needs to be fixed.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question