Recommend a database

Question

Database

Given: 2 MySQL Servers with 24M data units. SOLR sits on top and indexes several fields. Re-indexing occurs once a day.
1M lines are updated daily with the help of several dozen crawlers. Those. MySQL is hammered from two sides: Frontend, which makes about 2 M queries (by ID) per day and crawlers with a million UPDATE / INSERT.
So far, it's all working pretty fast, with the exception of the backup, which lasts more than an hour at night.
In the near future, it is planned to increase the database by 10-20 times.
What I don't like at the moment is that
we implement "sharding" for MySQL ourselves.
Sometimes there are "too many connections".
We run SOLR ourselves. Indexing lasts ~40min. At this time, we have "2" SOLR - one working, the second indexing. Then we switch to the second, and the first is cut down.

What you want: 1 interface that indexes, selects, writes and takes care of shards.

What database can it? Or so: what database that you worked with in a project of this size were you satisfied with?

Answer 1

> Sometimes there are "too many connections".
So this is due to the fact that you have a restriction in the MySQL config, and not because of any problems.

Answer 2

What you want is implemented not by databases, but by the application.
Here are some useful links in my opinion. I can't share my personal experience yet
:( .slideshare.net/oemebamo/database-sharding-at-netlog

Answer 3

look towards mongodb

Answer 4

Alternatively, you can take everything to the cloud (Amazon DynamoDB or Microsoft Azure). There are no problems with scalability, speed and reliability.