A
A
Artem Kileev2019-07-23 13:55:34
Java
Artem Kileev, 2019-07-23 13:55:34

How to analyze the logs for the correctness of the execution of the list of actions in tasks?

There is a large amount of logs (about 2 TB). The logs contain records about the execution of certain processes. Each process has its own log file. Each process periodically performs some kind of task, which has activity 1, activity 2, and so on. Sometimes these processes crash. It is necessary to analyze the logs and find out that the actions in the tasks were performed in the correct sequence. By what means can this be done?
I tried Apache Spark, but I can't figure out how to group actions by task in it. The matter is that until some time in logs identifiers of tasks in actions were not written down or were written down only for some.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
V
Vadim Shatalov, 2019-07-23
@netpastor

For processing logs, ELK is very good.

D
Dimonchik, 2019-07-23
@dimonchik2013

Serious guys use Clickhouse, where the type of storage for logs has already been drank
ELK not even yesterday

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question