Answer the question
In order to leave comments, you need to log in
How to sort responses when using multithreading?
Good day.
In a large file, requests to a certain API service are line by line. It is necessary to take these requests from the file, make an API call and add the answers to one file, and the order of the saved answers must correspond to the order of the requests (if the request is on the seventh line, then the answer to it must be in the new file on the seventh line). When executed sequentially, it takes quite a long time. When parallelized into several threads using a queue, it turns out that due to the uneven response time of the service, the answer to the 7th request can be received later than on the 8th one and there will be a discrepancy in the resulting file. It is not an option to form a list or dictionary from the answers for subsequent sorting. the source file can contain up to a million queries. Suggest an algorithm of actions in this case.
Answer the question
In order to leave comments, you need to log in
Add to the base with the number. Then, if necessary, you will unload the sorted data into a file.
If the response size for all requests is the same, then you can write directly to the address (as in an array). And so, you can make a save (index, result) function, through which to save the results, it will hold the answers in order, using the same dictionary, for example, in the end, even if you have a billion requests, and only 1000 requests are processed in parallel , then the size of the dictionary will not exceed 1000.
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question