Perfect logging in .NET application. How to achieve? What to use?

Question

C++ / C#

What is the task: I want to write logs in such a way that later it would be convenient to read them, filter them, and throw out unnecessary things from them. A simple example, there is a class that checks something on a timer 1 time per second and writes something to the logs 1 time per second. All this is activated if you go to a certain page in the application (navigation in a UWP application), which is not so rarely visited. When you look at such logs, it's a pain, since there is also logging of all HTTP requests and responses. It is very difficult to see the picture in them if you are looking for the cause of the error. For example, I want to see in the logs what has happened since the launch of the application, namely page transitions, and what happened on the last page only inside certain services (to see the logs of only these services).

I am convinced that the logs should not be in the form of ordinary strings, but written in the form of some data structures, for example, in a SQLite database. You will probably have to write your own application that retrieves only what you need from the database with filters. Maybe there is already something ready or there are some ideas, tips?

Answer 1

The mechanism is as follows:
1. Wherever there are logs, we write to the standard logger from Microsoft.Extensions.Logging, there ILogger, where T is the type of the class from which we write the logs.
2. We raise the ELK stack nearby or on a neighboring machine
3. In the application configuration, we connect the implementation of the logger that can send logs to Logstash (for example, Serilog
) what you can feed with your hands)
5. If necessary, filter the logs at the Logstash level (at least you need to ensure that the logs go to the desired index in ElastickSearch)
6. Set up dashboards in Kibana and monitor the application.
PS maybe this is of course from a cannon to sparrows, but as an option, you can use part of the ELK stack if everything is not needed - the same logstesh can not be written to elastic, but to a file, as far as I remember - i.e. it is possible to make sets of files by filters.

Answer 2

I can’t say about specific libraries, because I use self-written one, but the elastic stack allows you to analyze and visualize logs on the fly.
Also, some logging tips:
1) divide your program into logical multi-level nested subsystems, i.e. when logging, for example, a button, the system=client.ui.form attribute is added to the message
2) prioritize the logs, i.e. when an error occurs, the attribute is added level=error, on crash =critical, on debugging =trace...
3) add other useful attributes for analysis, e.g. date and time, user id
4) use two-way human-readable serialization, i.e. not stupid text, but some csv or json (necessarily utf8)
5) use a database or rotation by time and size
6) centralize the storage and analysis of logs
7) use a debug logging preprocessor only in a debug build, or rather, use a normal debugger with event tracking and a conditional stop
PS
The main logging rule is to reproduce the problem on the developer's machine, and not document it on the user's machine.

Answer 3

We take a logger that can do structural logging: Microsoft.Extensions.Logging or Serilog.
We take a database that can store documents.
There are several options:
- Seq (free for development, but paid for business)
- ApplicationInsights (needs Azure, there is a tiny free limit of logs per day, then paid)
- ElasticSearch + Kibana
- Grafana + Loki
There are, of course, other databases for storage docks, but what I indicated is closest to the task