Why does mysql slow down on insert and update?

N

nikosid2013-01-02 13:08:33

MySQL

nikosid, 2013-01-02 13:08:33

Hetzner has a rather strong server, on which only mysql is running and everything works well, except for one table - this is a news table, in which there are 250,000 records and two indexes - primary by id and news date.
So just recently mysql began to hang, or rather this particular table. When I look at the list of requests at the moment (echo 'show full processlist;' | mysql | less), I see that some request was made to the Locked table and a huge queue of requests to the same table lined up behind it.
Insert / Update per minute a couple of pieces is done, the selection is much more frequent.
Now they removed the kidney of old news temporarily, there are about 80,000 left and it works very quickly.
I suspect that the problem is in setting up mysql itself, but there is not enough knowledge to understand what exactly is wrong.
Spreadconfig .
Can someone share their config or suggest what to correct in mine?

Reply

Answer the question

In order to leave comments, you need to log in

4 answer(s)

M

Melkij, 2013-01-02
@melkij

First things first: innoDB?

K

KEKSOV, 2013-01-03
@KEKSOV

Perhaps, here is a good article about the problems of large tables.
And why, all of a sudden, such a “fear” of partitioning? Here is a live example from the statistics subsystem on the site of one of my customers. Over the year, about 5,000,000 records have accumulated there, everything works fine and nothing slows down:

SET NAMES cp1251;
DROP TABLE IF EXISTS stat_event;
CREATE TABLE stat_event
(
-- Общие поля
    `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'Ключ записи',
    `oid` BIGINT UNSIGNED NOT NULL DEFAULT '0' COMMENT 'ID Объекта',
    `created` DATETIME NULL COMMENT 'Время создания записи',
    `changed` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP NOT NULL COMMENT 'Время последнего изменения записи',
    `uid` BIGINT UNSIGNED NOT NULL DEFAULT '0' COMMENT 'ID пользователя, сделавшего последние изменения',
    `comments` BLOB COMMENT 'Примечание',

-- Поля объекта
    `event_time` DATETIME NOT NULL COMMENT 'UNIX timestamp события (может отличаться от значения в поле created)',
    `event_type` ENUM (
          'UNKNOWN'     -- тип события не определен
        , 'INDEX'       -- загрузка заглавной страницы
        , 'PRODUCT'     -- загрузка страницы с описанием товара
        , 'CATEGORY'    -- загрузка страницы с описанием категории
        , 'GROUP'       -- загрузка страницы с описанием группы категорий
        , 'SITEMAP'     -- загрузка роботом карты сайта
        , 'ORDER'       -- оформление заказа через сайт
        , 'OTHER'       -- прочее событие
        , '404'         -- страница не найдена
        , '301'         -- редирект страницы
        , 'VIEW_CART'   -- просмотр корзины заказов
        , 'OFFLINE_MSG' -- Отправка сообщения через Форму пожаловаться
        , 'ONLINE_MSG'  -- Отправка сообщения через Форма помощь онлайн
        , 'PAGE_LOAD'   -- Страница загружена в браузер
    ) NOT NULL DEFAULT 'UNKNOWN' COMMENT 'Тип события',
    `server_name` ENUM('0','foo.ru','boo.ru','goo.ru') NOT NULL DEFAULT '0' COMMENT 'Сервер, на котором произошло событие',
    `shop_object_id` BIGINT UNSIGNED DEFAULT 0 COMMENT 'идентификатор объекта магазина, если событие связано с ним',
    `request_uri` TEXT NULL COMMENT 'строка запроса',
    `query_string` TEXT NULL COMMENT 'параметры запроса',
    `bot` ENUM('0','1') NOT NULL DEFAULT '0' COMMENT 'В строке USER_AGENT встречается bot',
    `remote_addr` VARCHAR(255) NOT NULL DEFAULT '' COMMENT 'IP-адрес клиента',
    `referer` TEXT NULL COMMENT 'Адрес страницы, с которой пришел клиент',
    `user_agent` VARCHAR(255) NOT NULL DEFAULT '' COMMENT 'название браузера, переданного серверу',
    `server` TEXT NULL COMMENT 'php-переменная $_SERVER',
    `session` TEXT NULL COMMENT 'php-переменная $_SESSION',
    `cookie` TEXT NULL COMMENT 'php-переменная $_COOKIE',
    `request` TEXT NULL COMMENT 'php-переменная $_REQUEST',
    `event_params` TEXT NULL COMMENT 'Дополнительные параметры события, как правило это JSON',
    `profiler` TEXT NULL COMMENT 'json счетчиков производительности',
    `profiler_total` DECIMAL(8,5) NOT NULL DEFAULT 0 COMMENT 'время, за которое был выполнен запрос',
    `db` TEXT NULL COMMENT 'json счетчиков SQL-запросов',
    `tephlon` TEXT NULL COMMENT 'json статистики кеша SQL-запросов',
    
    PRIMARY KEY (`id`, bot, event_time )
) ENGINE=MYISAM DEFAULT CHARSET=cp1251 COMMENT 'События, происходящие на сайте'
PARTITION BY RANGE COLUMNS( event_time ) 
SUBPARTITION BY LINEAR KEY( bot )
SUBPARTITIONS 2
(
      PARTITION p2011_09 VALUES LESS THAN ( '2011-10-01 00:00:00' )
    , PARTITION p2011_10 VALUES LESS THAN ( '2011-11-01 00:00:00' )
    , PARTITION p2011_11 VALUES LESS THAN ( '2011-12-01 00:00:00' )
    , PARTITION p2011_12 VALUES LESS THAN ( '2012-01-01 00:00:00' )

    , PARTITION p2012_01 VALUES LESS THAN ( '2012-02-01 00:00:00' )
    , PARTITION p2012_02 VALUES LESS THAN ( '2012-03-01 00:00:00' )
    , PARTITION p2012_03 VALUES LESS THAN ( '2012-04-01 00:00:00' )
    , PARTITION p2012_04 VALUES LESS THAN ( '2012-05-01 00:00:00' )
    , PARTITION p2012_05 VALUES LESS THAN ( '2012-06-01 00:00:00' )
    , PARTITION p2012_06 VALUES LESS THAN ( '2012-07-01 00:00:00' )
    , PARTITION p2012_07 VALUES LESS THAN ( '2012-08-01 00:00:00' )
    , PARTITION p2012_08 VALUES LESS THAN ( '2012-09-01 00:00:00' )
    , PARTITION p2012_09 VALUES LESS THAN ( '2012-10-01 00:00:00' )
    , PARTITION p2012_10 VALUES LESS THAN ( '2012-11-01 00:00:00' )
    , PARTITION p2012_11 VALUES LESS THAN ( '2012-12-01 00:00:00' )
    , PARTITION p2012_12 VALUES LESS THAN ( '2013-01-01 00:00:00' )

    , PARTITION p2013_01 VALUES LESS THAN ( '2013-02-01 00:00:00' )
    , PARTITION p2013_02 VALUES LESS THAN ( '2013-03-01 00:00:00' )
    , PARTITION p2013_03 VALUES LESS THAN ( '2013-04-01 00:00:00' )
    , PARTITION p2013_04 VALUES LESS THAN ( '2013-05-01 00:00:00' )
    , PARTITION p2013_05 VALUES LESS THAN ( '2013-06-01 00:00:00' )
    , PARTITION p2013_06 VALUES LESS THAN ( '2013-07-01 00:00:00' )
    , PARTITION p2013_07 VALUES LESS THAN ( '2013-08-01 00:00:00' )
    , PARTITION p2013_08 VALUES LESS THAN ( '2013-09-01 00:00:00' )
    , PARTITION p2013_09 VALUES LESS THAN ( '2013-10-01 00:00:00' )
    , PARTITION p2013_10 VALUES LESS THAN ( '2013-11-01 00:00:00' )
    , PARTITION p2013_11 VALUES LESS THAN ( '2013-12-01 00:00:00' )
    , PARTITION p2013_12 VALUES LESS THAN ( '2014-01-01 00:00:00' )

    , PARTITION p2014_01 VALUES LESS THAN ( '2014-02-01 00:00:00' )
    , PARTITION p2014_02 VALUES LESS THAN ( '2014-03-01 00:00:00' )
    , PARTITION p2014_03 VALUES LESS THAN ( '2014-04-01 00:00:00' )
    , PARTITION p2014_04 VALUES LESS THAN ( '2014-05-01 00:00:00' )
    , PARTITION p2014_05 VALUES LESS THAN ( '2014-06-01 00:00:00' )
    , PARTITION p2014_06 VALUES LESS THAN ( '2014-07-01 00:00:00' )
    , PARTITION p2014_07 VALUES LESS THAN ( '2014-08-01 00:00:00' )
    , PARTITION p2014_08 VALUES LESS THAN ( '2014-09-01 00:00:00' )
    , PARTITION p2014_09 VALUES LESS THAN ( '2014-10-01 00:00:00' )
    , PARTITION p2014_10 VALUES LESS THAN ( '2014-11-01 00:00:00' )
    , PARTITION p2014_11 VALUES LESS THAN ( '2014-12-01 00:00:00' )
    , PARTITION p2014_12 VALUES LESS THAN ( '2015-01-01 00:00:00' )
    
    , PARTITION p2015_01 VALUES LESS THAN ( '2015-02-01 00:00:00' )
    , PARTITION p2015_02 VALUES LESS THAN ( '2015-03-01 00:00:00' )
    , PARTITION p2015_03 VALUES LESS THAN ( '2015-04-01 00:00:00' )
    , PARTITION p2015_04 VALUES LESS THAN ( '2015-05-01 00:00:00' )
    , PARTITION p2015_05 VALUES LESS THAN ( '2015-06-01 00:00:00' )
    , PARTITION p2015_06 VALUES LESS THAN ( '2015-07-01 00:00:00' )
    , PARTITION p2015_07 VALUES LESS THAN ( '2015-08-01 00:00:00' )
    , PARTITION p2015_08 VALUES LESS THAN ( '2015-09-01 00:00:00' )
    , PARTITION p2015_09 VALUES LESS THAN ( '2015-10-01 00:00:00' )
    , PARTITION p2015_10 VALUES LESS THAN ( '2015-11-01 00:00:00' )
    , PARTITION p2015_11 VALUES LESS THAN ( '2015-12-01 00:00:00' )
    , PARTITION p2015_12 VALUES LESS THAN ( '2016-01-01 00:00:00' )
    
    , PARTITION p2016_01 VALUES LESS THAN ( '2016-02-01 00:00:00' )
    , PARTITION p2016_02 VALUES LESS THAN ( '2016-03-01 00:00:00' )
    , PARTITION p2016_03 VALUES LESS THAN ( '2016-04-01 00:00:00' )
    , PARTITION p2016_04 VALUES LESS THAN ( '2016-05-01 00:00:00' )
    , PARTITION p2016_05 VALUES LESS THAN ( '2016-06-01 00:00:00' )
    , PARTITION p2016_06 VALUES LESS THAN ( '2016-07-01 00:00:00' )
    , PARTITION p2016_07 VALUES LESS THAN ( '2016-08-01 00:00:00' )
    , PARTITION p2016_08 VALUES LESS THAN ( '2016-09-01 00:00:00' )
    , PARTITION p2016_09 VALUES LESS THAN ( '2016-10-01 00:00:00' )
    , PARTITION p2016_10 VALUES LESS THAN ( '2016-11-01 00:00:00' )
    , PARTITION p2016_11 VALUES LESS THAN ( '2016-12-01 00:00:00' )
    , PARTITION p2016_12 VALUES LESS THAN ( '2017-01-01 00:00:00' )
    
    , PARTITION p2017_01 VALUES LESS THAN ( '2017-02-01 00:00:00' )
    , PARTITION p2017_02 VALUES LESS THAN ( '2017-03-01 00:00:00' )
    , PARTITION p2017_03 VALUES LESS THAN ( '2017-04-01 00:00:00' )
    , PARTITION p2017_04 VALUES LESS THAN ( '2017-05-01 00:00:00' )
    , PARTITION p2017_05 VALUES LESS THAN ( '2017-06-01 00:00:00' )
    , PARTITION p2017_06 VALUES LESS THAN ( '2017-07-01 00:00:00' )
    , PARTITION p2017_07 VALUES LESS THAN ( '2017-08-01 00:00:00' )
    , PARTITION p2017_08 VALUES LESS THAN ( '2017-09-01 00:00:00' )
    , PARTITION p2017_09 VALUES LESS THAN ( '2017-10-01 00:00:00' )
    , PARTITION p2017_10 VALUES LESS THAN ( '2017-11-01 00:00:00' )
    , PARTITION p2017_11 VALUES LESS THAN ( '2017-12-01 00:00:00' )
    , PARTITION p2017_12 VALUES LESS THAN ( '2018-01-01 00:00:00' )
    
    , PARTITION p2018_01 VALUES LESS THAN ( '2018-02-01 00:00:00' )
    , PARTITION p2018_02 VALUES LESS THAN ( '2018-03-01 00:00:00' )
    , PARTITION p2018_03 VALUES LESS THAN ( '2018-04-01 00:00:00' )
    , PARTITION p2018_04 VALUES LESS THAN ( '2018-05-01 00:00:00' )
    , PARTITION p2018_05 VALUES LESS THAN ( '2018-06-01 00:00:00' )
    , PARTITION p2018_06 VALUES LESS THAN ( '2018-07-01 00:00:00' )
    , PARTITION p2018_07 VALUES LESS THAN ( '2018-08-01 00:00:00' )
    , PARTITION p2018_08 VALUES LESS THAN ( '2018-09-01 00:00:00' )
    , PARTITION p2018_09 VALUES LESS THAN ( '2018-10-01 00:00:00' )
    , PARTITION p2018_10 VALUES LESS THAN ( '2018-11-01 00:00:00' )
    , PARTITION p2018_11 VALUES LESS THAN ( '2018-12-01 00:00:00' )
    , PARTITION p2018_12 VALUES LESS THAN ( '2019-01-01 00:00:00' )
    
    , PARTITION p2019_01 VALUES LESS THAN ( '2019-02-01 00:00:00' )
    , PARTITION p2019_02 VALUES LESS THAN ( '2019-03-01 00:00:00' )
    , PARTITION p2019_03 VALUES LESS THAN ( '2019-04-01 00:00:00' )
    , PARTITION p2019_04 VALUES LESS THAN ( '2019-05-01 00:00:00' )
    , PARTITION p2019_05 VALUES LESS THAN ( '2019-06-01 00:00:00' )
    , PARTITION p2019_06 VALUES LESS THAN ( '2019-07-01 00:00:00' )
    , PARTITION p2019_07 VALUES LESS THAN ( '2019-08-01 00:00:00' )
    , PARTITION p2019_08 VALUES LESS THAN ( '2019-09-01 00:00:00' )
    , PARTITION p2019_09 VALUES LESS THAN ( '2019-10-01 00:00:00' )
    , PARTITION p2019_10 VALUES LESS THAN ( '2019-11-01 00:00:00' )
    , PARTITION p2019_11 VALUES LESS THAN ( '2019-12-01 00:00:00' )
    , PARTITION p2019_12 VALUES LESS THAN ( '2020-01-01 00:00:00' )
    
    , PARTITION p2020_01 VALUES LESS THAN ( '2020-02-01 00:00:00' )
    , PARTITION p2020_02 VALUES LESS THAN ( '2020-03-01 00:00:00' )
    , PARTITION p2020_03 VALUES LESS THAN ( '2020-04-01 00:00:00' )
    , PARTITION p2020_04 VALUES LESS THAN ( '2020-05-01 00:00:00' )
    , PARTITION p2020_05 VALUES LESS THAN ( '2020-06-01 00:00:00' )
    , PARTITION p2020_06 VALUES LESS THAN ( '2020-07-01 00:00:00' )
    , PARTITION p2020_07 VALUES LESS THAN ( '2020-08-01 00:00:00' )
    , PARTITION p2020_08 VALUES LESS THAN ( '2020-09-01 00:00:00' )
    , PARTITION p2020_09 VALUES LESS THAN ( '2020-10-01 00:00:00' )
    , PARTITION p2020_10 VALUES LESS THAN ( '2020-11-01 00:00:00' )
    , PARTITION p2020_11 VALUES LESS THAN ( '2020-12-01 00:00:00' )
    , PARTITION p2020_12 VALUES LESS THAN ( '2021-01-01 00:00:00' )
    
    , PARTITION p2021_01 VALUES LESS THAN ( '2021-02-01 00:00:00' )
    , PARTITION p2021_02 VALUES LESS THAN ( '2021-03-01 00:00:00' )
    , PARTITION p2021_03 VALUES LESS THAN ( '2021-04-01 00:00:00' )
    , PARTITION p2021_04 VALUES LESS THAN ( '2021-05-01 00:00:00' )
    , PARTITION p2021_05 VALUES LESS THAN ( '2021-06-01 00:00:00' )
    , PARTITION p2021_06 VALUES LESS THAN ( '2021-07-01 00:00:00' )
    , PARTITION p2021_07 VALUES LESS THAN ( '2021-08-01 00:00:00' )
    , PARTITION p2021_08 VALUES LESS THAN ( '2021-09-01 00:00:00' )
    , PARTITION p2021_09 VALUES LESS THAN ( '2021-10-01 00:00:00' )
    , PARTITION p2021_10 VALUES LESS THAN ( '2021-11-01 00:00:00' )
    , PARTITION p2021_11 VALUES LESS THAN ( '2021-12-01 00:00:00' )
    , PARTITION p2021_12 VALUES LESS THAN ( '2022-01-01 00:00:00' )
)
;

-- ALTER TABLE `stat_event` ADD UNIQUE `oid` (`oid`);
ALTER TABLE `stat_event` ADD INDEX `shop_object_id` (`shop_object_id`);
ALTER TABLE `stat_event` ADD INDEX `event_type` (`event_type`);
ALTER TABLE `stat_event` ADD INDEX `time_bot` (`event_time`, `bot`);
ALTER TABLE `stat_event` ADD INDEX `time_bot_type` (`event_time`, `bot`, `event_type`);
ALTER TABLE `stat_event` ADD INDEX `shop_object_id` (`shop_object_id`);

N

nbutikov, 2013-01-02
@nbutikov

1. What type of index?
2. What field is being sampled, and what type is this field?
If there is phpmyadmin, see what it offers to optimize the table:
Table -> Structure -> Analysis of the table structure.
Might show something useful.

W

Wott, 2013-01-02
@Wott

The strange thing is that removing from 250K to 80K helped unload. It is possible that at 80K you have a cache hit and not at 250K, which should not happen with the right requests.
Look there is a limit for SELECT queries to this table and are all identical queries the same as rows?