How to store 500 million documents in Mysql or Postgresql?

M

miteigi nemoto2020-11-26 12:36:25

MySQL

miteigi nemoto, 2020-11-26 12:36:25

500 million text documents (html + text) need to be saved. The volume of the database will reach 700 GB. I believe that the storage will be on different VPS 60-80 GB each. What are the solutions for Mysql or Postgres?

Reply

Answer the question

In order to leave comments, you need to log in

2 answer(s)

M

Mikhail Vasilyev, 2020-11-26
@vasilyevmn

It is not necessary to store text documents in the database, especially 700Gb.
Use S3 compatible storage like Minio .

T

ThunderCat, 2020-11-26
@ThunderCat

500 million documents in Mysql or Postgresql... I need to index them through Sphinx

However, Sphinx does not depend on nor require any specific database to function.

If you do not have the task of working with an ALREADY EXISTING database, the sphinx does not need a base as such, it has its own indexes and its own engine, although it can also work with databases.
PS: sphinxsearch.com/forum/view.html?id=11964