A
A
Alexander2018-03-24 13:24:50
MySQL
Alexander, 2018-03-24 13:24:50

Is there a limit on the length of the SQL query (big IN)?

Greetings!
There is a big table on 30 columns and 5 million lines. It is necessary to go through each line, and perform additional data processing. I decided to select rows in batches of 100 thousand rows at a time.
I do it in two passes:
1. First, I select 100-300 thousand id in a cycle 2. By the second request, I pull out the remaining columns by these id: Is it correct to list 100 thousand id in the IN block? And 300 thousand? Is there a request length limit? How to achieve maximum execution speed? I would be grateful if you share your experience. Thank you.
SELECT id FROM tbl LIMIT X, 100000
SELECT * FROM tbl WHERE id IN (...)

Answer the question

In order to leave comments, you need to log in

2 answer(s)
A
Alexander Aksentiev, 2018-03-24
@Sanasol

And what is the sacred meaning of choosing first IDs with LIMIT OFFSET, and then making a second request through IN?
Why is it normal not to select full data through the limit?

A
Alj, 2018-03-27
@Alj

There is no ORDER BY in the query - the database should not return rows in order.
SELECT * FROM tbl
ORDER BY id LIMIT X,100000
X = 0 , 100000 , 200000 , ...

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question