Where is the best place to store large numbers of digital records for later analysis?

Question

NoSQL

There are many records (2 billion), moreover, of the form "almost_key-value":

1 - 2
1 - 33
1 - 256
1 - 103214
2 - 2321
123 - 432432
, etc.

There are 2 billion such pairs in total. And later a graph will be built, where the first index is a node, the second is another node with which this one is connected, and each of the records will have to be compared somewhere with 2-5 million other similar records . I want to choose the right database for these purposes. What do you advise?

Answer 1

In theory, with such a volume and with such a data structure, mongodb should normally cope.

Answer 2

The question is a bit broad. it is not clear what you will then do with these records?
There are at least 4 possible solutions. Their choice depends on what you then do with it.
1. Plain SQL. For example, Postgres + partitioning.
2. Look towards NoSQL solutions. Since you have graph solutions, that is, HyperGraphDB, Neo4j
, I myself did not use these databases, but it’s worth trying as a possible solution.
3. Look towards distributed solutions. For example, apache hadoop.
data processing is carried out using map-reduce technology.
4. If you need to search for something, then you can use some kind of search engine that indexes local files and searches for it quickly. there are no links at hand, maybe later I will throw off links to a couple of examples.
The 1st option was enough for me. The database is about 30GB. The selection comes from a specific section according to the condition, and so on ...
I know people who use the 3rd option, they don’t complain either. They have a larger database up to 64Tb

Answer 3

One of the fastest key/values ​​in existence is Oracle's BerkleyDB. You should have enough.