HiveBrain v1.2.0
Get Started
← Back to all entries
debugsqlMinor

MySQL crashes randomly with "Database was not shut down normally" message in log

Submitted by: @import:stackexchange-dba··
0
Viewed 0 times
shutlogwithnormallycrashesmessagedatabaserandomlymysqldown

Problem

I am having an issue where, for no apparent reason, my MySQL server crashes and has to restart.

Essentially, the error log has entries like this:

```
131128 12:16:35 mysqld_safe Number of processes running now: 0
131128 12:16:35 mysqld_safe mysqld restarted
131128 12:16:36 [Note] Plugin 'FEDERATED' is disabled.
131128 12:16:37 InnoDB: The InnoDB memory heap is disabled
131128 12:16:37 InnoDB: Mutexes and rw_locks use GCC atomic builtins
131128 12:16:37 InnoDB: Compressed tables use zlib 1.2.3
131128 12:16:37 InnoDB: Using Linux native AIO
131128 12:16:37 InnoDB: Initializing buffer pool, size = 768.0M
131128 12:16:37 InnoDB: Completed initialization of buffer pool
131128 12:16:37 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 5922117901
131128 12:16:37 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 5922118503
131128 12:16:41 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
131128 12:16:41 InnoDB: Waiting for the background threads to start
131128 12:16:42 InnoDB: 5.5.32 started; log sequence number 5922118503
131128 12:16:42 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3306
131128 12:16:42 [Note] - '0.0.0.0' resolves to '0.0.0.0';
131128 12:16:42 [Note] Server socket created on IP: '0.0.0.0'.
131128 12:16:43 [Note] Event Scheduler: Loaded 0 events
131128 12:16:43 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.32-cll' socket: '/var/lib/mysql/mysql.sock' port: 3306 M

Solution

In the end, it was only after changing my hosting plan (and moving away from cPanel, which seemed to obscure some of the logging) that I was able to see the issue. Turns out that from time to time, memory usage would spike (I'm guessing that this was because of a dramatic spike in apache child processes) and this would cause memory pressure. The out-of-memory killer would choose mysql to close down, because it was (usually) the biggest memory user in the system.

So to fix, I

  • got more memory



  • fine tuned Apache to control the number of child processes that were spun up.

Context

StackExchange Database Administrators Q#54200, answer score: 5

Revisions (0)

No revisions yet.