How to collect traffic statistics?

Question

linux

With Cisco I take a copy of the traffic with RSPAN. It is required to collect statistics from this stream, in which it will be: who downloaded where, when and how much. Where - with a domain name, not just an IP address. Squid doesn't fit.

Answer 1

Ntop-ng .
If possible, it is better to export traffic through Netflow, there will be less load on the cisco.

Answer 2

Option 1: It's not that simple. It all depends on how accurate the information you expect. If you use Netflow, then you can also run DNS on the server itself and get confused by creating a script that will record the IP-Domain and compare it with the IP that came via Netflow. If you mess with it, then in the end you can get a good product))
Option 2: If your router supports Nbar2, then you need to find a netflow server that supports it. Scrutinizer seems to be able to. NBAR 2 can retrieve HTTP: URL, host, user-agent, referrer.But whether Scrutinizer can process these attributes is another matter. didn't try it myself. I don’t know if there will be problems with HTTPS, but the latest versions of NBAR2 can read the Server name (I checked it myself and it works).
PS I had option 1 before, it seemed to work somehow. Option 2 did not try at all, so I only know the theory, but if the docks do not lie, everything should work. Well, with a proxy it will be much easier, it's a pity that you are not satisfied with this option.