Which database will work with a 1 petabyte table and 3,000 billion records?

Question

MongoDB

There is a task - storage and search in logs:
timestamp - client IP - destination IP - destination URL
There are 10 billion of such records per day or 3000 billion per year. At the same time, there are no serious requirements for the load - several times a month you need to find the necessary records.
A simple web interface is supposed to create a request, but you need to decide on the base. What base will cope with this task?

Answer 1

SQLite

Answer 2

It might help. Recently, at a local conference, they told how Avito stores logs:
They don’t just store there, but monetize all this data through ad targeting, etc.
The report was interesting, it’s a pity the video recording was not preserved.
Here is another link to the presentation with this report.

Answer 3

mongo + sharding.

Answer 4

Have you heard about Big Data? On such a scale, forget about the classic database.

Answer 5

Firstly, you need to decide what exactly you need to look for in this heap, that is, what to dance from, it is one thing if for a given IP to find all destination URLs, it is quite another to find all the IPs that came to this destination URL. The storage architecture in both cases should be different. In any case, tasks of this magnitude are solved by professionals with the appropriate qualifications. "According to the recommendation on the toaster" such things are not done, at best, the self-propelled gun will slow down hellishly and do a "search" for years, at worst, at some point you will lose data.

Answer 6

You can read it yourself kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis
I would recommend looking towards Hadoop kids like HBase or Cassandra.
But MongoDB will do just fine.