Answer the question
In order to leave comments, you need to log in
How to load huge json api via pagination?
There is an API for almost all cities in the world in .json format.
[
{
"id": 707860,
"name": "Hurzuf",
"country": "UA",
"coord": {
"lon": 34.283333,
"lat": 44.549999
}
},
{
"id": 519188,
"name": "Novinki",
"country": "RU",
"coord": {
"lon": 37.666668,
"lat": 55.683334
}
},
....
]
Answer the question
In order to leave comments, you need to log in
If the file is not very large (up to tens of megabytes) - you can read it into memory, you get a regular array, you can give it to the usual slice.
If the file is very large (hundreds of megabytes or more) - I think the best thing would be to read it through some lib like https://github.com/dominictarr/JSONStream and put it in the database. And contact her.
Please note that the file may not fit in RAM - then you need to do it in a threaded manner (and not only then).
Those. do not read the whole file piece by piece -> glue -> put it in the database,
but read the piece -> put the piece in the database (and make sure that the memory does not leak)
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question