How to diagnose (LAN) a local area network?

Question

Computer networks

We have been using the Cisco Catalyst 6513 for more than 5 years, there were no such problems with it, below is the show version listing and further a description of the situation.

Catalyst-6513#sh ver
Cisco Internetwork Operating System Software
IOS (tm) c6sup2_rp Software (c6sup2_rp-PSV-M), Version 12.1(26)E6, RELEASE SOFTW
ARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2006 by cisco Systems, Inc.
Compiled Mon 23-Jan-06 10:13 by hqluong
Image text-base: 0x40008F90, data-base: 0x41854000

ROM: System Bootstrap, Version 12.1(11r)E1, RELEASE SOFTWARE (fc1)
BOOTLDR: c6sup2_rp Software (c6sup2_rp-PSV-M), Version 12.1(26)E6, RELEASE SOFTW
ARE (fc1)

Catalyst-6513 uptime is 5 days, 22 hours, 23 minutes
Time since Catalyst-6513 switched to active is 5 days, 22 hours, 22 minutes
System returned to ROM by power-on (SP by power-on)
System image file is "sup-bootflash:c6sup22-psv-mz.121-26.E6.bin"

cisco WS-C6513 (R7000) processor (revision 1.0) with 112640K/18432K bytes of mem
ory.
Processor board ID SAL0802SHTV
R7000 CPU at 300Mhz, Implementation 39, Rev 3.3, 256KB L2, 1024KB L3 Cache
Last reset from power-on
X.25 software, Version 3.0.0.
Bridging software.
9 Virtual Ethernet/IEEE 802.3  interface(s)
338 Gigabit Ethernet/IEEE 802.3 interface(s)
381K bytes of non-volatile configuration memory.

32768K bytes of Flash internal SIMM (Sector size 512K).
Configuration register is 0x2102

For the second week already, we have been observing the following picture:
The load on the CPU is increasing up to 80%, the network starts to lag terribly, the ping to printers increases to 300 ms, the company's network services are unavailable (print server, 1C, IP telephony (Cisco UCM) drops), network on computers). In normal mode, the load on the CPU does not even reach 5%, the temperature of the switch components does not rise above 28 degrees Celsius.
The strangest thing in all this is that it started after the company purchased HP ProOne 600 monoblocks. The first problem that arose was that applications could not be launched via the network, switching on and off UAC did not lead to anything. Only copying distributions to a PC and subsequent installation helped (the reason was never revealed).
When the load increased, we found that one of these monoblocks was to blame, rebooted the PC, changed the patch cord and port to cisco (to be sure).
A day later, the situation repeated itself, but with another user - however, having the same candy bar at his disposal. On the one hand, they have been working with us for 3 months, on the other hand, they were the source of the problem in both cases, and their subsequent reboot and changing the port on the switch dropped the load from 80% to 2% within 10 minutes after the manipulations.
The problem is that I can not identify the culprit in this situation, I do not know where else to dig.
Googling about network diagnostics gave links to copied articles where Win 95 was used as a study.

Answer 1

Increases CPU load up to 80%

Which process consumes the most resources? The command will help:
on interfaces connected to end hosts. The threshold value may need to be adjusted.
A couple of off-topic notes:

Version 12.1(26)E6

Of course, I understand that it works - do not touch it, but you can think about updating the software.
In addition, it is difficult to talk about intelligible diagnostics without taking appropriate readings - memory utilization, CPU resources, interfaces (including the level of broadcast and multicast traffic), the presence of errors / dropped packets, traffic to the processor (mirrored using SPAN). Also consider setting up a control plane protection policy ( link ).

Answer 2

Enable traffic mirroring on the switch to another port and use wireshark to see what is special about this monoblock.
Another option is to take a dump directly with the switch and save it to a flash, but this is if possible.

Answer 3

According to the symptoms, it looks like a ring, remove the broadcast counters at the input from the ports and see how fast they grow, in general, this is usually done with a monitoring system, but you can also use handles. If you find 2 ports on which the counters grow stronger than the others at the entrance, you have a ring, if they grow strongly on one port, then either the ring is further away or some piece of iron is flooding. You can guess for a long time, without a monitoring system for the entire controlled hardware, you need experienced psychics in the state.