debugsqlMinor
Mysql suddenly crashing after several months
Viewed 0 times
aftercrashingsuddenlyseveralmysqlmonths
Problem
I'm away on leave and mysql on an EC2 (m3.medium general purpose) based server I have The displeasure of managing has suddenly started crashing. I have very intermittent internet connectivity through my phone to restart the whole server (which does resolve issue temporarily for a few hours).
Running mysql 5.5 Apache 2.2 PHP 5.3
Instance has 3.75 GB memory.
As a very short term bodge to see us through this situation until I can get back and sort things properly I ran up a cron script that restarted mysqld if it became unavailable this worked a little, MySQL won't restart at all without a server restart so that idea is defunct.
I am still pretty new to db administration and have optimised my.cnf as best as I could understand - We are running magento which is very demanding.
A side note is that Yesterday I noticed that there were hundreds of sleeping processes from the magento site and wrote another script that would kill these if they had been around for ages- again a temp fix while I am away.
Can anyone advise what the issue causing the crashes might be or how to gain more insight?
It looks to me that it is to do with innodb buffer pool but will reducing this help?
Error_log from the failed attempts to automatically restart.
```
150903 10:15:06 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150903 10:20:12 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150903 10:20:14 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead.
150903 10:20:14 [Note] Plugin 'FEDERATED' is disabled.
150903 10:20:15 InnoDB: The InnoDB memory heap is disabled
150903 10:20:15 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150903 10:20:15 InnoDB: Compressed tables use zlib 1.2.7
150903 10:20:15 InnoDB: Using Linux native AIO
150903 10:20:16 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
150
Running mysql 5.5 Apache 2.2 PHP 5.3
Instance has 3.75 GB memory.
As a very short term bodge to see us through this situation until I can get back and sort things properly I ran up a cron script that restarted mysqld if it became unavailable this worked a little, MySQL won't restart at all without a server restart so that idea is defunct.
I am still pretty new to db administration and have optimised my.cnf as best as I could understand - We are running magento which is very demanding.
A side note is that Yesterday I noticed that there were hundreds of sleeping processes from the magento site and wrote another script that would kill these if they had been around for ages- again a temp fix while I am away.
Can anyone advise what the issue causing the crashes might be or how to gain more insight?
It looks to me that it is to do with innodb buffer pool but will reducing this help?
Error_log from the failed attempts to automatically restart.
```
150903 10:15:06 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150903 10:20:12 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150903 10:20:14 [Warning] The syntax '--log-slow-queries' is deprecated and will be removed in a future release. Please use '--slow-query-log'/'--slow-query-log-file' instead.
150903 10:20:14 [Note] Plugin 'FEDERATED' is disabled.
150903 10:20:15 InnoDB: The InnoDB memory heap is disabled
150903 10:20:15 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150903 10:20:15 InnoDB: Compressed tables use zlib 1.2.7
150903 10:20:15 InnoDB: Using Linux native AIO
150903 10:20:16 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
150
Solution
150903 10:20:16 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
150903 10:20:16 InnoDB: Completed initialization of buffer pool
150903 10:20:16 InnoDB: Fatal error: cannot allocate memory for the buffer poolNote:
OS error code 12: Not enough spaceYou have 3.75GB of RAM in a VM? But something is cluttering it. What? Do you have other applications (aside from Apache, PHP) running?
query_cache_size = 128M
Too big, make it only 50M
max_connections = 1000
[OK] Highest usage of available connections: 1% (16/1000)
You are not really needing that many? Suggest changing to 100.
[innodb]
Notice that the buffer_pool_size below that line is ignored. Remove
[innodb]; the info should be in [mysqld].thread_cache_size = 256
10 should suffice in most cases.
tmp_table_size = 256M
[!!] Temporary tables created on disk: 64% (22K on disk / 35K total)
That is much too big for almost anyone. That much space can be allocated for a
SELECT that needs a tmp table. A complex SELECT might need multiple tmp tables. And that is "per connection". Use 16M.Looks like you need some better indexes.
[--] InnoDB is disabled.
Try the script again after fixing the memory problem
init_connect='SET collation_connection = utf8_unicode_ci'
init_connect='SET NAMES utf8mb4'Those are not consistent. And, keep in mind that user root does not execute the
init_connect.What version of MySQL/MariaDB are you running?
Code Snippets
150903 10:20:16 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
150903 10:20:16 InnoDB: Completed initialization of buffer pool
150903 10:20:16 InnoDB: Fatal error: cannot allocate memory for the buffer poolOS error code 12: Not enough spaceinit_connect='SET collation_connection = utf8_unicode_ci'
init_connect='SET NAMES utf8mb4'Context
StackExchange Database Administrators Q#113077, answer score: 2
Revisions (0)
No revisions yet.