Why did mariadb start to slow down?

Question

MySQL

There is a physical ubuntu server:

Linux pc 5.11.0-46-generic #51~20.04.1-Ubuntu SMP Fri Jan 7 06:51:40 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

with mariadb:

mariadb  Ver 15.1 Distrib 10.3.32-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

to which an external hard drive is connected via USB 2.0 (in ext4), which in turn contains the files of the database itself.
This database is constantly updated with scripts from a large number of 'sources', I'll list the directories below:

db_folder:

drwxr-xr-x 5 mysql mysql 4,0K janv. 17 20:06 .
drwxr-xr-x 5 root  root  4,0K janv. 17 19:54 ..
-rwxrwxrwx 1 root  root   16K janv. 17 20:05 aria_log.00000001
-rwxrwxrwx 1 root  root    52 janv. 17 20:05 aria_log_control
-rwxrwxrwx 1 root  root     0 janv.  5 17:05 debian-10.3.flag
-rw-rw---- 1 mysql mysql  17K janv. 17 20:05 ib_buffer_pool
-rwxrwxrwx 1 root  root   76M janv. 18 15:49 ibdata1
-rwxrwxrwx 1 root  root   48M janv. 18 15:49 ib_logfile0
-rwxrwxrwx 1 root  root   48M janv. 18 15:49 ib_logfile1
-rw-rw---- 1 mysql mysql  12M janv. 17 20:05 ibtmp1
-rwxrwxrwx 1 root  root     0 janv.  5 17:05 multi-master.info
drwxrwxrwx 2 root  root  4,0K janv.  5 17:05 mysql
-rwxrwxrwx 1 root  root    16 janv.  5 17:05 mysql_upgrade_info
drwxrwxrwx 2 root  root  4,0K janv.  5 17:05 performance_schema
drwx------ 2 mysql mysql 4,0K janv. 17 20:18 mydatabase
-rw-rw---- 1 mysql mysql  24K janv. 17 20:05 tc.log

cat ./mydatabase

-rw-rw---- 1 mysql mysql   67 janv. 17 20:06 db.opt
-rw-rw---- 1 mysql mysql 1,5K janv. 17 20:07 data.frm
-rw-rw---- 1 mysql mysql 828M janv. 18 15:50 data.ibd
-rw-rw---- 1 mysql mysql 1,1K janv. 17 20:11 photos.frm
-rw-rw---- 1 mysql mysql 748M janv. 18 15:50 photos.ibd
-rw-rw---- 1 mysql mysql 2,7K janv. 17 20:18 users.frm
-rw-rw---- 1 mysql mysql 2,1G janv. 18 15:50 users.ibd

config: cat /etc/mysql/mariadb.conf.d/50-server.cnf

#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
#
# See the examples of server my.cnf files in /usr/share/mysql

# this is read by the standalone daemon and embedded servers
[server]

# this is only for the mysqld standalone daemon
[mysqld]

#
# * Basic Settings
#
user                    = mysql
pid-file                = /run/mysqld/mysqld.pid
socket                  = /run/mysqld/mysqld.sock
#port                   = 3306
basedir                 = /usr
#datadir                = /var/lib/mysql
datadir                 = /media/tester/11b21335-f921-1e11-1ffr-6760202c7c0ab/db
tmpdir                  = /tmp
lc-messages-dir         = /usr/share/mysql
#skip-external-locking

# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address            = 0.0.0.0

#
# * Fine Tuning
#
#key_buffer_size        = 16M
#max_allowed_packet     = 16M
#thread_stack           = 192K
#thread_cache_size      = 8
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
#myisam_recover_options = BACKUP
max_connections        = 100
#table_cache            = 64
#thread_concurrency     = 10

#
# * Query Cache Configuration
#
#query_cache_limit      = 1M
query_cache_size        = 64M

#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file       = /var/log/mysql/mysql.log
#general_log            = 1
#
# Error log - should be very few entries.
#
log_error = /var/log/mysql/error.log
#
# Enable the slow query log to see queries with especially long duration
slow_query_log_file    = /var/log/mysql/mariadb-slow.log
#long_query_time        = 10
#log_slow_rate_limit    = 1000
#log_slow_verbosity     = query_plan
#log-queries-not-using-indexes
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
#server-id              = 1
#log_bin                = /var/log/mysql/mysql-bin.log
expire_logs_days       = 5
max_binlog_size        = 50M
binlog_do_db           = include_database_name
binlog_ignore_db       = exclude_database_name

#
# * Security Features
#
# Read the manual, too, if you want chroot!
#chroot = /var/lib/mysql/
#
# For generating SSL certificates you can use for example the GUI tool "tinyca".
#
#ssl-ca = /etc/mysql/cacert.pem
#ssl-cert = /etc/mysql/server-cert.pem
#ssl-key = /etc/mysql/server-key.pem
#
# Accept only connections using the latest and most secure TLS protocol version.
# ..when MariaDB is compiled with OpenSSL:
#ssl-cipher = TLSv1.2
# ..when MariaDB is compiled with YaSSL (default in Debian):
#ssl = on

#
# * Character sets
#
# MySQL/MariaDB default is Latin1, but in Debian we rather default to the full
# utf8 4-byte character set. See also client.cnf
#
character-set-server  = utf8mb4
collation-server      = utf8mb4_general_ci

#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!

#
# * Unix socket authentication plugin is built-in since 10.0.22-6
#
# Needed so the root database user can authenticate without a password but
# only when running as the unix root user.
#
# Also available for other users if required.
# See https://mariadb.com/kb/en/unix_socket-authentication-plugin/

# this is only for embedded server
[embedded]

# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]

# This group is only read by MariaDB-10.3 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don't understand

This database will be about 300GB in size, at the moment, with a database size of about 3GB, I noticed that its filling is much slower than at the beginning.
First of all, I checked the resources (CPU, RAM) of the server, they are used only by 10-15%, then I checked the speed at which the current read / write operations are taking place on the disk (maybe USB 2.0 is a bottleneck ..)
I checked the read speed using 'iotop' 304.13 K / s write speed is not more than 604.13 K / s ... That is, this option is also most likely discarded.
Actually, what other test can be performed to understand why the database slows down (it works many times slower than at the beginning)?

PS: I will try to provide the entire output of commands as quickly as possible.

Answer 1

via USB 2.0 external hard drive

strange setup
but

read speed 304.13 K/s write speed no more than 604.13 K/s...

says that your usb is running at 1mb speed, which is very similar to the usb1 version
to make sure, turn off the database (remove the load) and check the disk for linear reading in any way, for example by reading that 2gb file to null

dd if=/path/to/file of=/dev/null bs=1024k status=progress

the reason for the degradation of usb2 speed to version 1 is banal, on the same controller (neighboring usb output or, for example, the same usb hub), a mouse / keyboard or any other usb1 version device hangs. This is a feature of usb hubs, they lower the speed to the minimum supported (even if you have usb3)
The solution is to connect a mouse / keyboard or disk to another controller (just experimentally sort out the ports)