HiveBrain v1.2.0
Get Started
← Back to all entries
debugsqlMinor

Mysql suddenly crashing after several months

Submitted by: @import:stackexchange-dba··
0
Viewed 0 times
aftercrashingsuddenlyseveralmysqlmonths

Problem

I'm away on leave and mysql on an EC2 (m3.medium general purpose) based server I have The displeasure of managing has suddenly started crashing. I have very intermittent internet connectivity through my phone to restart the whole server (which does resolve issue temporarily for a few hours).
Running mysql 5.5 Apache 2.2 PHP 5.3
Instance has 3.75 GB memory.

As a very short term bodge to see us through this situation until I can get back and sort things properly I ran up a cron script that restarted mysqld if it became unavailable this worked a little, MySQL won't restart at all without a server restart so that idea is defunct.

I am still pretty new to db administration and have optimised my.cnf as best as I could understand - We are running magento which is very demanding.

A side note is that Yesterday I noticed that there were hundreds of sleeping processes from the magento site and wrote another script that would kill these if they had been around for ages- again a temp fix while I am away.

Can anyone advise what the issue causing the crashes might be or how to gain more insight?
It looks to me that it is to do with innodb buffer pool but will reducing this help?

Error_log from the failed attempts to automatically restart.

```
150903 10:15:06 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150903 10:20:12 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150903 10:20:14 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead.
150903 10:20:14 [Note] Plugin 'FEDERATED' is disabled.
150903 10:20:15 InnoDB: The InnoDB memory heap is disabled
150903 10:20:15 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150903 10:20:15 InnoDB: Compressed tables use zlib 1.2.7
150903 10:20:15 InnoDB: Using Linux native AIO
150903 10:20:16 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
150

Solution

150903 10:20:16 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
150903 10:20:16 InnoDB: Completed initialization of buffer pool
150903 10:20:16 InnoDB: Fatal  error: cannot allocate memory for the buffer pool


Note:

OS error code  12:  Not enough space


You have 3.75GB of RAM in a VM? But something is cluttering it. What? Do you have other applications (aside from Apache, PHP) running?


query_cache_size = 128M

Too big, make it only 50M


max_connections = 1000
[OK] Highest usage of available connections: 1% (16/1000)

You are not really needing that many? Suggest changing to 100.


[innodb]

Notice that the buffer_pool_size below that line is ignored. Remove [innodb]; the info should be in [mysqld].


thread_cache_size = 256

10 should suffice in most cases.


tmp_table_size = 256M
[!!] Temporary tables created on disk: 64% (22K on disk / 35K total)

That is much too big for almost anyone. That much space can be allocated for a SELECT that needs a tmp table. A complex SELECT might need multiple tmp tables. And that is "per connection". Use 16M.

Looks like you need some better indexes.


[--] InnoDB is disabled.

Try the script again after fixing the memory problem

init_connect='SET collation_connection = utf8_unicode_ci'
init_connect='SET NAMES utf8mb4'


Those are not consistent. And, keep in mind that user root does not execute the init_connect.

What version of MySQL/MariaDB are you running?

Code Snippets

150903 10:20:16 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
150903 10:20:16 InnoDB: Completed initialization of buffer pool
150903 10:20:16 InnoDB: Fatal  error: cannot allocate memory for the buffer pool
OS error code  12:  Not enough space
init_connect='SET collation_connection = utf8_unicode_ci'
init_connect='SET NAMES utf8mb4'

Context

StackExchange Database Administrators Q#113077, answer score: 2

Revisions (0)

No revisions yet.