How can I collect traffic (traffic sniffer) from remote linux servers?

Question

linux

Advise the best solution to the problem.
There is a group of linux servers with a running application, incoming traffic is evenly distributed between them by an external tcp load balancer (LB passes traffic through itself, while the source IP is preserved).
Periodically, for diagnostics, it becomes necessary to collect traffic dumps of a specific client (by a known IP address), ideally, to look at this traffic in real time through wireshark on the windows of the workstation.
Unfortunately, I do not have the ability to collect traffic through SPAN on network equipment. The task needs to be solved only by software on servers.
The standard solution (running wireshark on all hosts, collecting the dump, merging it onto one machine, merging it into one file and opening it in wireshark) is extremely labor-intensive and allows you to view data only offline.
The only thing that was found to collect traffic from remote servers is rpcapd, but this is not at all the same, it will not work to collect from a dozen servers at once.
As a temporary crutch solution, I wrote a simple utility that runs on servers, collects the necessary traffic (based on libpcap), encapsulates it in TZSP / ERSPAN and sends it to a given host via UDP.
This solved the problem, but there is a feeling that there is something ready and correct solution to my problem.
Who faced a similar task? How did you solve it?

Answer 1

I am not special in this matter at all, but I would try the following:
- I would sharpen your utility so that it writes everything necessary to a text file
- we put logstash on each target server, set it on this very text file
- we raise Elasticsearch somewhere + Kibana
- we speak logstash on servers, send data to our elasticsearch
- we set up beautiful charts / graphs / tables in kibana
- ???
- PROFIT
So, for example, I collect statistics of http requests to my application from the customer's servers.
How to set up / install this whole thing is wonderfully described at the office. Elasticsearch website

Answer 2

Look towards the NetFlow protocol, it should help you

Answer 3

As for rpcapd...
Is it sure that when using rpcapd, you can't dump from multiple sources?
man tshark has this:

-i | -
Set the name of the network interface or pipe to use for live packet capture.
...
What if we try to do this?
tshark -i rpcap://host1/eth0 -i rpcap://host2/eth0 -f "some kind of filter"