Why do logs overwrite each other in multi-threaded usage?

Question

Python

import logging
import logging.handlers

_log = logging.getLogger('DEBUG')
logfile = 'logs/debug.log'
formatter = logging.Formatter('%(asctime)s %(message)s')

handler = logging.handlers.TimedRotatingFileHandler(logfile,
                                                    when='midnight',
                                                    backupCount=10)
handler.setLevel(logging.DEBUG)
handler.setFormatter(formatter)

_log.setLevel(logging.DEBUG)
_log.addHandler(handler)

I'm running a script in 10 threads. In it inside I connect this logging.
The script in 10 threads works constantly and once a day it writes a log for a new day, but a bug gets out on a new day.
Each thread creates a file when it needs to write something to the log, and thus the file is re-created 10 times and the logs are not saved (
Perhaps everything happens a little differently, but the next day some of the logs are either missing or in a torn form. For example, I know what is written 3 logs in a row, but I see only 1 or 2 last logs in the file, but they are not in the file for the previous day

How can I fix this or use something else for multi-threaded logging

Answer 1

Strangely, the Python logger flush-it every entry
is possibly a problem with log rotation, it is not designed for multi-user recording
. For multi-threaded logs, it is better to raise some kind of server and send logs to it via sockets. The rotation must be done by someone alone.

Answer 2

The script in 10 threads works constantly and once a day it writes a log for a new day, but a bug gets out on a new day.
Each thread creates a file when it needs to write something to the log, and thus the file is recreated 10 times and the logs are not saved (

Remove the rotation from the script (Of course - each script rotated the same log) and use logrotate, use the copytruncate parameter there so that the scripts continue to write to the file with the same handler.

Answer 3

https://github.com/delgan/loguru