Web application logging: syslog and stdout/stderr?

Question

linux

When developing web projects, I usually use logging in two ways:

1. For simple projects - logs are written to a file
2. For large projects - business logs are written to the queue (rabbitmq), and system logs are also written to a file. Then these logs are taken by logstash and uploaded to elasticsearch.

Also, more and more often I come across two other methods (from all sorts of reports and cases) that are of interest to me:

1. Write logs to syslog (rsyslog, nxlog and other implementations).
One of the benefits I know of is performance.

2. Write logs to stdout/stder according to 12 factor architecture .
As I understand it, the advantage is that if we have a lot of services, then they all write the log according to the same principle, and because of this, you can easily manage the logs of all services.
Those. the application itself transfers the responsibility for managing the logs to a higher level.

If I misunderstood something, please correct me.

I have a few questions:

1. In what cases are logs usually written to syslog? did you have experience?
What are the pros and cons?

2. To be honest, I don't quite understand the principle of sending logs to stdout.
First, what does it mean in stdout ? if this is a web application (website), then the log will be immediately shown to the user? ..
Secondly, how to collect these logs later if the project does not use containerization, for example?
What are the pros and cons of this approach? What was your experience?

Answer 1

1. The standard syslog service can parse logs by facility, so you can set up logging of different components to different files at the syslog level, and manage their rotation.
Also, syslog can work with other syslogs, so you can aggregate logs from different machines and manage them centrally.
2. logs sent to stdout are usually redirected somewhere, to the same file, or immediately robbed into some kind of aggregator.
In modern times, stdout is most often talked about when you run something in a container, and the container is spinning in an orchestrator like kubernetes / openshift.
In this case, an external collector is configured - the same filebeat, fluentd, logstash or syslog, which collects logs from the entire kubernetes / openshift cluster, parses them and throws them into the aggregator.
We just deployed a new component and by its name you can filter the logs in the same Kibana, while you don’t need to configure something for the new component anywhere in the logging system, everything is tagged automatically.

Answer 2

stdout is standard output, usually the console. So you launched the application in the console - and saw what it wrote to you, because it outputs messages to stdout. And it's not for nothing that std in the name is "standard". He can work with a lot of things. Let's take systemd for example: it starts the service, its stdout is caught and gets into journald. You can wrap it up somewhere else. Container management systems do this too.
How to collect? Well, you know how to do everything, you are talking about queues, write logstash - here, this is it. You take filebeat, well, or whatever other *beats it has, catch the output of your applications with them and shove it into the elastic. Well, or let's say you take fluentbit and send them logs to some thread in the queue, and from there to the storage.

1. In what cases are logs usually written to syslog?

well, syslog has been around for a very long time, it was before all these containers and 12-factor applications ;) So almost everyone used it at one time.