Is it worth using hadup?

Question

elasticsearch

Fight everyone
There is a task, to absorb a lot of logs (about a terik per hour, but this is the border). We tried a cluster of 24 elastics with well-tuned rsyslogs, ingests, LVM tuned exactly to the task and other goodies, but as a result it rests on la and falls. If it didn’t rest against la, then it rests against memory and still falls. Of course, everything can still be tuned, but since the load can be even greater, then in general we are looking in the direction of the hadup. I decided to ask here, maybe someone will advise some practical advice on operation.
The task in the end is this: to take in a lot of logs, and through the api (self-written) give the requested and already distributed logs to some kind of json.
configs right now, to be honest, I won’t throw off, vpn is not configured
by resources, 4 hosters in wz, I don’t remember right now, but it seems like the 730s

Answer 1

gee gee, Elastic ...
well, Khadup is the easiest, but what does it have to do with "give away"?
look at Clickhouse, if in one move, it suddenly comes in
if not in one - keep in mind that the option "keep only aggregated on the sale", that is, give it away, and "raw data separately and slowly" does not leave
the solution
, the record is different - from that or Postgres / Percona to plain text files (like the same Hadup),
to right in Click, depends on ... both sampling and pre-prepared processing requests
- Click, (there are fantasies with the Sphinx, but Click made them unnecessary) to some Tarantula
(no experience here, sorry) and Aerospike
just look at the bottlenecks,

Answer 2

Hadoop will pull of course. I have a 2 PB cluster in the admin and development with logs of 3-4 TB / day, which after ETL turn into 5-10 billion rows in Hbase. The question here is rather in the gland and its quantity. Also, fast writing requires skills, and this is Java Api MR, Pig or NiFi.
In elastic everything is easier.