Analysis of server logs in Python. The script hangs during logrotate, how to deal with it?

Question

linux

I use a self-written script to analyze some logs. The script reads the log in a loop constantly and when logrotate is processed (the old log goes to the archive and a new, clean file appears) it freezes. How to deal with it?
I read the log like this

def monitor(file):
    file.seek(0,2)
    while True:
        line = file.readline()
        if not line:
            time.sleep(0.1)
            continue
        yield line

loglines = monitor(logfile)
for line in loglines
    ....

Answer 1

In your case, it is necessary not to keep the log files continuously open. Opened, counted available lines or their n-th number into the buffer, closed. Pause, repeat the cycle.
The problem is that after rotation, the open file descriptor continues to point to the old file.

Answer 2

The most obvious solution is to copy it to a tempo place, or if the logs are not large, then read the entire file not line by line

Answer 3

You should use the default package logging and TimedRotatingFileHandler - this is already correctly implemented and immediately with log rotation.
config example:

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "rotate": {"format": "%(levelname)s %(asctime)s %(pathname)s %(lineno)d %(message)s"}
    },
    "handlers": {
        "file_rotate": {
            "level": "DEBUG",
            "class": "logging.handlers.TimedRotatingFileHandler",
            "formatter": "rotate",
            "filename": "/var/log/my_logger.log",
            "when": "midnight",
        }
    },
    "loggers": {
        "my_logger": {
            "level": "DEBUG", 
            "handlers": ["file_rotate"]
        }
    },
}

write logs like this:

import logging

my_logger = logging.getLogger("my_logger")
my_logger.debug('debug message')
my_logger.info('info message')
my_logger.error('error message')
my_logger.critical('critical message')