MySQL crash/restart during mysqldump

General: Standalone MySQL-5.5.43 in OpenVZ on ProxMox-3.4-6/102d4547 (Debian, 2.6.32-39-pve). Container uses 18Gb of RAM and 100Gb of disk. This container runs this service more than a year.

There's also two tiny containers running on this node, with tiny I/O load, LAN load - etc.

Hardware: 2x Xeon E5649 2.53GHz, 24Gb RAM, Intel RAID controller

Software:

• some kind of web-site and expert system on PHP. About 10-20

connections in second (depends of situaiton - can be 40-50, and in
another time even 0). InnoDB engine, about 20 Gb base size, all in
one file (no innodb_file_per_table set)

• Apache-2.4.10

my.cnf:

[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

[mysqld_safe]
log_error = /var/log/mysql/error.log
nice    = 0
socket = /var/run/mysqld/mysqld.sock

[mysqld]
basedir = /usr
bind-address = 127.0.0.1

datadir = /var/lib/mysql

expire_logs_days = 10

innodb_buffer_pool_size = 14G
innodb_fast_shutdown = 0

innodb_log_file_size = 1024M

key_buffer = 16M

lc-messages-dir = /usr/share/mysql
log_error = /var/log/mysql/error.log

max_allowed_packet = 16M
max_binlog_size = 100M
max_connections = 384
max_heap_table_size = 32M
myisam-recover = BACKUP

port = 3306
pid-file = /var/run/mysqld/mysqld.pid

query_cache_limit   = 1M
query_cache_size = 32M

skip-external-locking
socket = /var/run/mysqld/mysqld.sock

table_open_cache = 512
tmpdir = /tmp
thread_stack = 192K
thread_cache_size = 16
tmp_table_size = 32M

user = mysql

[mysqldump]
max_allowed_packet = 16M
quick
quote-names

[mysql]

[isamchk]
key_buffer = 16M

!includedir /etc/mysql/conf.d/

Dump of base is made by command:

/usr/bin/mysqldump --single-transaction --skip-opt -uUSER -pPASS |/bin/gzip -c >dump.gz

During dump sometimes mysql crashes (or restarts - IDK) with next error messages:

error.log

```
161028 08:07:02 mysqld_safe Number of processes running now: 0
161028 08:07:02 mysqld_safe mysqld restarted
161028 8:07:03 [Note] /usr/sbin/mysqld (mysqld 5.5.43-0+deb8u1) sta

It was OOM.

The first thing I was suspect was memory. But:

-
VM's dmesg didn't output any messages until I reboot container. Only after that I saw messages, like

[0.000000] Out of memory in UB 101: OOM killed process 779 (mysqld) score 0 vm:16724384kB, rss:13130908kB, swap:2250952kB

-
MySQL error log (and another system logs of VM and node) didn't complain about not enough memory state;

-
error was completely "unstable".

So now, I re-calculated memory, used by applications in container, tuned innodb_buffer_pool_size and till now everything is OK.

StackExchange Database Administrators Q#153575, answer score: 2