Apache - send logs over the network

Question

Apache HTTP Server

About 40 virtual machines are used on the server, some of which are quite visited sites (20-50 thousand people a day).

Now i / o wait is starting to become a problem, as one of the ways to reduce the load on the screws turned off the Apache logs, but after a while it became clear that the logs are a very good thing and I need them.

Who-thread has an experience of operation of record of dens in a pipe and the further sending of dens on other server? How does this design behave? are there pitfalls?

CustomLog "| nc -u localhost 514" combined


Answer 1

Raise a "strong" host with syslog and send logs there via tcpip

Answer 2

1. for logging over the network, you should use udp and not tcp, because a small plug can add up logging in general.
2. perhaps it is worth considering the option of collecting logs locally, but not inside each vds, but in dom0, transmitting them over the network (but not releasing traffic outside the machine). This will help to more efficiently cache logging using fs dom0 and reduce io.
3. a significant part of the logs is generated by small requests for images, they can be selectively disabled, leaving important logs for requests to scripts

Answer 3

The simplest thing is to turn off logs for static js, css, pic, video content.
And so the syslog over the network is probably used by all serious implementations in production on virtual machines. It is convenient to watch all the logs in one place, the place from the logs does not end, and so on, it flies well and is available in all Linux distributions.

Answer 4

I saw this more than once on large production systems. In principle, this is how it should be (especially when the logs are also kept by F5 and the firewall, and you need to store them for six months), but there is a nuance - ddos ​​can easily put a grid on you.

Answer 5

Well, firstly, iowait is not only a disk, but also a network. By transferring the logs from disk to the network, you will not reduce iowait much.
secondly, you can send it to the syslog right away, Apache can do this, though for some reason only for error_log.
third, buy an SSD for logging.

Answer 6

It's hard to believe that the disk is sinking because of the logs.
I have about 1000 user requests per minute on one of the machines, the logs are written to disk + from the disk the utility is sent in real time via udp to another host, this does not create a noticeable load.
I think the problem with io is somewhere else, see what
iostat -xm shows 1 - disk
load iotop - disk load by process.
Perhaps the picture of the problem will become clearer.

Answer 7

Maybe a little offtopic? Why are 40 virtual machines with websites running on 1 server? o_o

Answer 8

I understand, thanks. I thought that usually this is done simply through fakeroot.

Answer 9

papertrailapp.com/