Answer the question
In order to leave comments, you need to log in
How to quickly transform data from table A of 500M+ rows into an aggregated table B?
In order to optimize access to certain queries, it was decided to create an aggregated table.
It is necessary to perform a one-time transfer of data to an aggregated table with row-by-row processing.
If you write a program with logic outside the DBMS, in my opinion, it will work for a long time due to round-trip traffic.
Couldn't google a suitable answer.
Please tell me the best way.
Answer the question
In order to leave comments, you need to log in
If the aggregation logic is simple and fits in one query without any cursors, then it is better to make sql an SQL script within one transaction.
If it is more difficult and you have to use a cursor, but not more than one, then also in the sql script. (may be overlaid with indexes)
In all other cases, put it in an application with wide selections.
500M+ is not very much.
In the program, by the way, no one bothers you to make wide selections, process them and paste them back.
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question