The hard drive falls off during backup, where to dig?

Question

System administration

Good afternoon, faced such problem. The disk serving for backups of machines began to fall off.
xfs filesystem.
Judging by the logs, this happens just during backup, judging by the logs

INFO:  75% (22.6 GiB of 30.0 GiB) in 5m 24s, read: 48.0 MiB/s, write: 29.6 MiB/s
INFO:  76% (22.9 GiB of 30.0 GiB) in 5m 27s, read: 122.0 MiB/s, write: 57.9 MiB/s
zstd: error 25 : Write error : Input/output error (cannot write compressed block) 
INFO:  76% (23.1 GiB of 30.0 GiB) in 6m 45s, read: 2.0 MiB/s, write: 1.7 MiB/s
ERROR: vma_queue_write: write error - Broken pipe
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - vma_queue_write: write error - Broken pipe
INFO: Failed at 2021-10-06 00:36:47
cp: cannot create regular file '/mnt/pve/backups/dump/vzdump-qemu-100-2021_10_06-00_30_02.log': Input/output error
command 'df -P -T -B 1 /mnt/pve/backups/dump' failed: exit code 1
ERROR: Backup of VM 101 failed - unable to create temporary directory '/mnt/pve/backups/dump/vzdump-qemu-101-2021_10_06-00_36_47.tmp' at /usr/share/perl5/PVE/VZDump.pm line 840.
INFO: Failed at 2021-10-06 00:36:47

Physically the disk is determined

lshw -class disk -class storage
  *-sata
       description: SATA controller
       product: Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode]
       vendor: Intel Corporation
       physical id: 17
       bus info: [email protected]:00:17.0
       logical name: scsi1
       logical name: scsi2
       logical name: scsi3
       version: 31
       width: 32 bits
       clock: 66MHz
       capabilities: sata msi pm ahci_1.0 bus_master cap_list emulated
       configuration: driver=ahci latency=0
       resources: irq:127 memory:f7428000-f7429fff memory:f742c000-f742c0ff ioport:f090(size=8) ioport:f080(size=4) ioport:f060(size=32) memory:f742b000-f742b7ff
     *-disk:0
          description: SCSI Disk
          physical id: 0
          bus info: [email protected]:0.0.0
          logical name: /dev/sda
          size: 3726GiB (4TB)
          configuration: logicalsectorsize=512 sectorsize=4096
     *-disk:1
          description: ATA Disk
          product: TOSHIBA DT02ABA4
          vendor: Toshiba
          physical id: 1
          bus info: [email protected]:0.0.0
          logical name: /dev/sdb
          version: 0A
          serial: 3131S3K2S75H
          size: 3726GiB (4TB)
          capabilities: gpt-1.00 partitioned partitioned:gpt
          configuration: ansiversion=5 guid=0a8127b4-6df3-4ba1-be3d-769a54d4c143 logicalsectorsize=512 sectorsize=4096

Mount point present but not detachable

ls -lah /mnt/pve/backups
ls: cannot access '/mnt/pve/backups': Input/output error

mount | grep sd
/dev/sdb1 on /mnt/pve/VMs type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sda1 on /mnt/pve/backups type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)

Smart doesn't work

smartctl -a /dev/sda
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.11.22-1-pve] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Short INQUIRY response, skip product id
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

xfs_check doesn't work either.

xfs_repair /dev/sda
Phase 1 - find and verify superblock...
superblock read failed, offset 0, size 524288, ag 0, rval -1
fatal error -- Input/output error

After a reboot everything is fine

Answer 1

5 Reallocated_Sector_Ct 485
...
197 Current_Pending_Sector 16304
Well, here's the answer. :( Kirdyk to the disk itself, you can not look for other reasons.

Answer 2

>cp: cannot create regular file
check the FS for an error, make sure that there are rights to create files in this folder.
>fatal error -- Input/output error
southern heater, cable? what's in dmesg?
you have a strange smart, as if it were a virtual machine

Answer 3

It is not visible what size the backup files are, their number, how much space is left on the disk. If there are a lot of small files and it is closer to 50%, then the answer is most likely:
Toshiba - 3.5 "DT02ABA400 4TB DT02(-V) DM- SMR
And this turns out to be
5 Reallocated_Sector_Ct 485
...
197 Current_Pending_Sector 16304
Due to reboots on the go, t .e He does not answer, but reset him.