Virus infection in a VERY large network?

Question

Malware

There is a VERY large LAN network of a very large medical center in Asia (China). The network has a distributed optical network, a domain network, peer-to-peer, without division into VLANs.
The network was designed a long time ago, when there were no normal switches, after the upgrade, the amount of work was so great that the peer-to-peer network was left.
~6000 computers, 4 buildings attached.
The situation is as follows:
1. All computers access the Internet through a very strict proxy server, with strict filtering, with authorization by plain login-password, most computers never work on the Internet.
2. Most computers run Windows XP, and there are a lot of workstations running Windows 98
3. The operating system should not be changed in any case, since most of the laboratory / diagnostic programs are very old, the drivers are old, they do not want to be friends under new systems.
4. Not only computers are on the network, various medical devices are connected as members of a peer-to-peer network, sending diagnostic information via the simplest SMB protocol. There are about 500 devices, starting with X-ray machines and ending with DNA analyzers.
5. Some workstations are not domain members.
6. The IT department of the medical center is 40 people, 15 of them are administrators.
7. More than 600 different operations on people are performed around the clock in the medical center, the network cannot be interrupted in any case, since all diagnostic information (MRI, X-ray, test results, accounting) is transmitted continuously.
8. Downtime of the network for a day threatens with losses of millions of dollars
Problem:
Some asshole (he has already been kicked out) set up a game server on the network, shared the Internet via mobile LTE Internet and launched a cryptolocker virus into the network.
The network turned out to be infected, and the encoder virus walks around the network, encrypting data on local computers and on shared SMB resources.
About 20-40 computers are infected every day.
The medical center is happy to pay the hackers a ransom, but through negotiations with them, they came to the conclusion that it is impossible to decrypt file systems separately every day, the hackers honestly said that they could not help with this.
On most computers, anti-virus protection is not available as such, since most workstations have Windows 98 & XP
. Is there a specific fakap or is there a way out?

Answer 1

Whatever the organization, I do not believe that ALL computers need to communicate with ALL computers.
Install adequate routers, break the network into VLANs, treat them separately.
Older operating systems can be run in virtual machines with the network turned off. And on the hardware itself - a normal, modern OS.
In many cases, the computer with diagnostic equipment can be completely disconnected from the network.

Answer 2

1. Shoot admins, on the topic
2.
If downtime costs millions of dollars, then what prevents you from paying a couple of million to solve this problem?
Here we shoot managers, well, those who were not shot in paragraph 1.
3.
An ancient axis, without updates (on most I'm sure no updates have been installed since the installation), without a banal antivirus.
What else did you want?
Virus epidemics, as in your situation, are not treated by "let's run that computer with an antivirus", they are treated by "chopping off the fuck everything and reinstalling the systems, because the hell understand what signature this rubbish has."
You have flown in full and there is only one solution - hardware / software upgrades, a competent approach to the network.

Answer 3

We have already written above that you need to segment the network. This task is both strategic (you will decide after the problem is fixed) and tactical (limiting the spread of malware).
You need to segment by replacing switches with managed ones with vlan, acl, dhcp snooping functionality (for the future), loopback detection. You can buy switches for ~ 6000 ports in one day (this is about 120 access switches, not counting the aggregation and the router). It will still be cheaper than financial and image losses from downtime.
How exactly you need to segment - can only be determined by location. It may be possible to immediately cut off the segments that are not involved in the operational activities of the company, sending 1-2 people there to clean up the antivirus, and throw the main forces into putting the network in order. Maybe it makes sense to "block everything" and open access to certain resources as problems arise. Of the 15 admins, you can leave 5 in place, and send the rest (35) to figure out who doesn’t work. All communication should be reduced to:
- Volodya, this is Semyon, IP address 10.156.2.25, MAC address ends with 13-AC-F4 need access to 10.43.1.67, this is an x-ray in
room
523 same for XXX, YYY, ZZZ...

Answer 4

Write a virus that extinguishes the previous one. There is contact with hackers, there is an encryption algorithm, the vulnerability is known. Buy the source code of their creation and edit, or contact those who can write something similar. This is if you can not turn off the network. The access points that were so familiar were returned under control, it was unrealistic to run around them - they were hung all over the city. In a couple of days, they piled a virus-cleaner. And then - according to the points described above: segmentation, traffic control, etc.

Answer 5

I'm very curious, how did it end?

Answer 6

You are using the SMB protocol, the ransomware uses it.
If you try to wrap all SMB traffic on the router, and either block infected ones or try to parse the headers of infected packets. (maybe they stand out against the general background)
I read the comments and we can say that it all comes down to one thing,
a real modernization awaits you, painful but justified. Since now the authorities are faced with the fact that it is impossible to save on IT!

Answer 7

Of course, such a network is a complete nightmare and it is difficult to advise something here. A comprehensive approach is needed here. I recommend contacting specialists in one of the anti-virus companies, as well as companies specializing in software and hardware firewalls (Fortinet, Check Point Software).
And if you take the problem locally, then you need to look at what crypto-locker you got infected with, for example, WannaCry or Petya (Not.Petya) have a stop file, the presence of which in certain places stops the crypto-locker from working. Of course, running through 6 thousand PCs is almost impossible, but ....

Answer 8

Basically, all decisions come down to what needed to be done before the infection. KMK, now the first thing to do is to cut down all the computers in general, turn on one by one and determine the status - infected or not, based on this, take further actions - give access to the network or the C format. To determine whether it is infected or not, involve specialists from anti-virus laboratories. Segment the network along the way (we involve specialists), it still needs to be done, because, in fact, according to this scheme, it will be configured well, almost from scratch. Naturally, with the obligatory removal of disk images in the current state, whatever it may be. Can't turn it off? Financial losses? Well, excuse me - the previous economy got out sideways now, it's just a fact.

Answer 9

And what about the computer park itself, how old are the computers themselves?
The idea is as follows: we take a new clean computer, put a modern axis and a virtual machine with Win98 on it, to which we transfer all the necessary software. The software itself and the drivers can be checked for antiviruses before installation.

Answer 10

1. Segment the network.
2. If the company has a multi-million dollar income, you can order an antivirus from the same Kaspersky.

Answer 11

We have been writing software capable of solving such problems for 2 years, we have not entered the market, since we are still using it for internal needs, but this is a hospital, is there a possibility of communication?
Contact email :
[email protected]

Answer 12

Try to download which LiveCD thread and treat after the fact. I do not think that what thread DrWeb Cureit! run on win98.
As I understand it, you still don’t have any options, so it seems to me that the best option is to understand which computers have a virus and turn them off the network, and then diagnose and treat them, then driving them back into the network.

Answer 13

Plus previous speakers. You got stuck specifically, and you can’t manage it on your own, attract serious integrators to solve it. And prepare money, of course - to pay for their services, for new equipment.
The network must be segmented. Into as small segments as possible, so that it would be easier to cut everything to hell without waiting for peritonitis. How exactly to cut - let the integrator tell you after analyzing the map of information flows in your network. There is hardware that builds such maps based on data from network equipment - who, where and what protocols go, based on this map, it will be clear how to segment the network and how to cut ACLs between networks.
You may have to implement some kind of traffic analyzer that, according to certain signatures in packets, will block ports - this is probably the only way to keep your old OS and at the same time have at least some semblance of protection against network worms. Well, all other complex measures too - setting up firewalls, the most severe control over the appearance of new devices on the network and new types of traffic (such as, for example, game servers ;-) ).
Here are good translations of two CIS regulations, start implementing:
https://habrahabr.ru/company/pentestit/blog/338532/
https://habrahabr.ru/company/pentestit/blog/339206/

Answer 14

Of course, I am not a sysadmin with experience, but I would do so. In parallel with the old network, I would quickly deploy a new one, with a new cable, new computers, new network hardware, properly organized so that it does not intersect with the old one at all. I would prepare it: I would put a computer with the necessary OS and drivers in each office, test everything in advance, break it into segments. And then on the night of X, having gathered all the admins, switch everything to the new network at once, and, along the way, solve the problems that arise. And then slowly disassemble the old network.

Answer 15

and riveting usb or dvd with Kaspersky Rescue Disk 10 is not fate?

Answer 16

Here people correctly wrote. The software is old ... we remove it on the server and we hammer on the rdp (application repair or analogues). We cut the weeds and begin to upgrade to modern axes. You can even put terminals.

Answer 17

Затушить то, что можно затушить, причем как можно быстрее. Может быть вы и потеряете какую-то выгоду, но упущенная прибыль - это не прямые убытки. И с каждым часом у вас добавляются прямые убытки.
Заплатить хакерам, пусть выпустят тулзу для лечения своего творения. Или как минимум, пусть документируют работу своего творения, укажут куда оно прописывается и как можно задетектить "зараженность". Собрать РХЕ-сервер с образом линупса, который бы детектил зараженные машины и слал бы отчет на головной сервер. Ребутнуть все что можно, загружаясь с РХЕ где можно, а где нельзя - грузимся с флешек и болванок. Если квалификации достаточно, то можно за день уложиться, зато будет карта зараженных машин и можно будет продолжить работу без инфицированных.
Infected machines, where infa is needed on them, should be treated individually, having previously disconnected from the network, and if not needed, upload fresh images to them from the same PXE server. Again, write to hackers, let them make a normal tool for encryption management.
In principle, there was already a correct opinion that it was necessary to resort to the services of a fat integrator, but ... But thick integrators collect cheap labor for a penny, so there will be nothing magical in their services. At best, the same specialists as here will come running, who can only roll out antiviruses and all the updates they see, shake the tambourine and hope that the latest version of the patches will save them from something there. I especially liked the installation of Win98 in a virtual machine on a modern OS, just the top of the local expertise. Do you need such experts?
But in any case, you have already got the money. So run to the switch and turn everything off. ALL!!

Answer 18

Такая сеть - кошмар для админов. О безопасности нужно было думать раньше. Такую сеть практически нереально полностью вычистить. Постоянно будет вылазить что-то с флех, архивов и т.д.
Рекомендации:
1) Анализ потоков и разбивка на сегменты. В нерабочее время для минимизации ущерба.
2) Межсетевые экраны на границе сегментов. Желательно с обнаружением аномального трафика.
3) Все шары для обмена - нормальный сервер с антивирусом.
4) Постепенный обход машин с накатыванием заплат. Для закрытия хотя-бы базовых уязвимостей.

Answer 19

напишите мне, пожалуйста, [email protected] . обсудим, чем я могу помочь.
для начала -- не нужно ничего покупать, не подумав. особенно у антивирусных и примкнувших к ним компаний.

Answer 20

Тут надо писать свои руткиты под 98 и ХРюшу, с основной идеей в пресечении любой активности кроме узкого профиля приложений необходимых для работы, плюс обнаружение аномального поведения.
У меня так, куча лабораторного старья здравствует и ныне, уж конденсаторы по второму кругу меняют, а оно всё работает.