K
K
Konstantin Malyarov2019-06-20 12:51:55
Django
Konstantin Malyarov, 2019-06-20 12:51:55

How to properly process a large amount of data received by SOAP?

There is a Django project that receives SOAP data (zeep library), translates it into JSON (for convenient processing in VueJS).
But bad luck, SOAP sends 212020 records page by page, which turns from one request into 425 requests.
As a result, this turns into a whole torment:
Axios sends a request to the Raspberry Pi 3 server, which redirects the request to nsi.rosminzdrav.ru , which sends the data back to the Raspberry Pi 3, where it is converted into JSON format and sent back to the client.
And the whole procedure takes from 6 to 12 hours. How to speed up the process (software and hardware, if possible with examples or links to documentation)?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
A
alternativshik, 2019-06-20
@alternativshik

425 queries and 212k records is not much. Why get them all the time? Why can't they be obtained only once and kept for yourself, and then only receive the latest entries - 1-2 pages.
Well, you can also send parallel requests, get different pages and save them.
Based on a meager description, it is quite difficult to offer something specific. In some place, at least it tupit. Getting all the data or processing it are completely different situations that require different solutions.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question