A
A
Alexey2019-10-17 15:36:09
Node.js
Alexey, 2019-10-17 15:36:09

Hashmap in Node.JS for 350M+ objects, what to choose?

I am writing a node.js service that processes large amounts of geodata. In a particular case, there is an array of points, each point has a numeric id and coordinates (lat, lng) and I need to quickly get the coordinates by id. The first thing that came to mind was to simply create an Object in memory and add values ​​​​where the key is id and the value is the [lat, lng] array. The problem is that when the number of elements in an object exceeds 9 million, then performance drops sharply, the addition operation becomes longer and longer until eventually everything stops. I googled that node objects, and even simple index arrays, are not designed for such an amount of data. I found a module in npm that implements a hashmap in c ++, the author promised super speed and super volumes, in real life it also started to slow down already at 13 million, and the farther the worse. There is enough RAM on the machine. I need to store about 350 million records. Point id's are integers already sorted in ascending order.
What tool should I use to save such an array of data and subsequently quickly get it by id? Preferably without using "heavy artillery" like postgress or mongodb.

Answer the question

In order to leave comments, you need to log in

3 answer(s)
E
Eugene, 2019-10-17
@Espritto

redis, memcache

L
longclaps, 2019-10-17
@longclaps

If it is possible to reindex the geodata so that the id is in the range [0..N-1] - do so. And store your [lat, lng] in array[0..N-1], preferably in two: lat[0..N-1], lng[0..N-1]
If not, store all sorted id in array, and when referring to id by binary search, you find its position in the array, and using it you select coordinates from lat[0..N-1] and lng[0..N-1].

A
Alexander Filippenko, 2019-10-19
@alexfilus

Wouldn't it be necessary to search for them? Maybe it's better to take, for example, monga with its geoindexes?

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question