K
K
Kamal Khafizov2018-12-04 22:05:34
JavaScript
Kamal Khafizov, 2018-12-04 22:05:34

Need to read n LOBs (~4 billion properties in total) in the backend, how to do this on a Node.js site in the fastest way?

Update 1. Changed the question, now I'm only interested in loading the object in the backend.
Update 2. Added information about the essence of the task, posted an approximate object, changed the object to several objects and some other small changes.
Update 3. Replaced the word "download" with "read" for the correct understanding of me and small changes. Bold highlighted a question that has not yet been answered.
Or an array. The object has references to 10 variables (not cyclic). Ignoring nested objects/arrays, the final elements of an object/array are strings and numbers (although this is not essential). Nesting of properties - ten-level or single-level, has not yet been determined. Properties - options for decomposing 10 items into 9 cells. You can see more details and an example of an object in the comments to the question .. Since the size of the object that JS can work with is limited, including the power of the hardware on which it works, instead of one object there will be several objects of an acceptable size, but the essence of the issue remains the same.
Bonus questions. Are databases like Mongo DB faster in terms of loading objects than reading from a JSON file or not? Is reading an object faster in Node.js than in front-end JS if the hardware is the same? Is it an option to just insert the contents of the JSON file into the JS files (which will contain only those large objects) and connect them as a module in Node.js instead of reading from the database or the JSON file? Will this option of reading objects be faster, because parsing will not occur (do I understand correctly that parsing occurs when reading from the database?)?
What is the fastest way to read an object/array?

Answer the question

In order to leave comments, you need to log in

4 answer(s)
A
Alexander Aksentiev, 2018-12-04
@Sanasol

Let's say the properties do not contain any structure and overhead for data format, keys, nesting, etc.
And let's say they consist of one character, i.e. 1 byte.
Total 4kk bytes = 4GB of data.
So in any way at the current statement of a question/task.

V
Vitaly, 2018-12-04
@vshvydky

delusional question. I tried to minus 5 minutes. The
node is not recommended for anyone for heavy calculations, the case described in the question smells like a dump from some kind of database, it’s the way to it, jsonb (postgres) mongo elastic, whatever, the main thing is to read this file in a stream and decompose it into the database. ... trying to use JSON.parse is some rubbish....

A
Alexey Stupenkov, 2018-12-05
@alexstup

Remake AI

C
chibitko, 2018-12-05
@chibitko

Js is not suitable for such a data structure, js-object is an associative array, not a structure, it will consume more memory at times, you can try to load data into one key-value object, where the key and value are strings, the value needs to be parsed in order to get a parent, or use a database, any database will do, you will need to create an index on the parent node, to analyze the situation no more than 10 queries of 4 billion records - partitions to help

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question