What could be the reasons for such a stepwise growth of nginx access logs?

Question

linux

We noticed a very strange change in the size of the nginx access log file. Below is a screenshot from Zabbix, which shows that the nginx log file is growing in steps (the graph shows free space in /var, but there are only nginx logs, besides, we checked the change in the size of the logs in the console). It can be seen that the measurement of "steps" on the graph increases evenly and proportionally.
The strangest thing is that around 22:00 the log file began to weigh 8GB, and then immediately 4GB, while we did not carry out any actions and there were no "special log rotation daemons" on the host.
A few clarifications:

• The traffic is uniform, there were no jumps in the flow.
  
• The logs in the nginx config are configured by default:
  

access_log /var/log/nginx/access.log main;
Maybe there are some features of logging in nginx that we didn’t guess about from the dock?

Answer 1

In general, the answer turned out to be slightly unexpected for us, and the point was the specifics of XFS. It turned out that as the file grows, space is preallocated on the disk, which explains the jumpy growth of indicators in the du command, which does not correlate with traffic dynamics. Below are links for those who are interested, as well as a screenshot of the graph from Zabbix, on which we decided to separately monitor the log file (du and stat outputs). And as we can see, du grows in steps, and stat dynamically corresponds to traffic:
git.kernel.org/cgit/linux/kernel/git/torvalds/linu...
serverfault.com/questions/406069/why-are-my- xfs fi...

Answer 2

And who told you that there are no rotations?
Are you sure logrotate didn't work for you?

Answer 3

nginx does not know how to clean logs and does it right. This works out something like logrotate.
This is definitely not nginx itself, it only writes logs. I would do it first grep -r /etc -e nginx | grep -v ^/etc/nginx.
If you couldn't find anything in this way, you can connect heavy artillery: install auditd, write in the rules something like
And look in the auditd logs for everything that worked with this folder. Just keep in mind that there will be a lot of these logs, for each syscall.

Answer 4

Firstly, vfs.file.size on access.log
Secondly, cat /etc/logrotate.d/nginx
+ you can look at /var/log with lsof and see who and what writes there
at one time a 4GB log could not jump, how often do you check the free space in /var/log?
By the way, at the end of the krafik, the stair-likeness ends, did you change the metric sampling time?

Answer 5

I don't believe that there is nothing in /var except nginx logs...