How to diagnose Linux server?

Question

linux

Well, for example, something is written to the disk, how to find out what, the algorithm of actions?
Or is the server slowing down, what will you do?
What do you watch and monitor for hacker activity?
When do you understand that there are loads and where?

Answer 1

To find out what is writing to the disk, you need to run iotop.
If the server "slows down", you need to see what the processor is taking up and whether the processor is busy - the disk subsystem may be busy. The top command usually handles this task.
All this has nothing to do with information security or hackers.
"Hacker activity" - what kind of activity is this? In what way does it usually manifest itself? If it is that someone took and broke a leaky wordpress, then this is easily searched for by cron by searching the .php files for the eval command. It's generally a good idea to look at files for changes, as Tripwire and AIDE do. The second good idea is to delimit the permissions so that "hacker activity" is not observed on the server - you cannot break the service if it is not publicly available. The third good idea is to monitor service logs and watch anomalies.

Answer 2

We have already said about iotop and top, it is also worth adding about iftop, which shows network activity and through which you can see with what intensity traffic goes to which ports.
As for "hacker activity," this concept is very broad and, if not a book, then a decent article for sure. As the main actions in case of suspicion of hacking, we can recommend viewing the logs (in particular, the output of the last and lastlog commands), viewing the list of running daemons for "left" processes (often on hacked systems, hackers run some kind of perl script or binary that connects host controlled by the villain to receive commands). Important: those pearl bots that I came across often use a tricky function that replaces the name of a running script with some harmless one (for example, httpd), while if you look at the list of processes from root with the ps auxwww command, this malware will appear in the list like this:perl /tmp/bot.pl --name "httpd -DSSL"At the same time, the /tmp/bot.pl file itself may not exist, which means that it was deleted after launch, so that it would not accidentally catch the eye of the admin. Before killing the bot process, it's worth looking at which server it connects to, i.e. where it has a "command center," this can be done with the command netstat -anp | grep PID_процесса_бота
Well, then to taste, or simply blacklist this address on the firewall and go dig web server, php, ftp and ssh logs for traces of penetration, or, if your hands are itching, you can see what kind of traffic the bot is exchanging with the command center. This can be done with the command

tcpdump -Nn -i имя_сетевого_интерфейса -x host ip_командного_центра

Answer 3

Rather not an answer, but an addition to the previous answers. This picture helps me
And the article is also recommended www.brendangregg.com/blog/2014-08-23/linux-perf-to...