HiveBrain v1.2.0
Get Started
← Back to all entries
debugsqlMinor

MySql: Frequent crashes on server with substantial resources

Submitted by: @import:stackexchange-dba··
0
Viewed 0 times
withcrashesmysqlsubstantialserverresourcesfrequent

Problem

My MySQL server running the latest version of MySql on Ubuntu 16.x keeps crashing once or twice a day. Sometimes it repairs itself fairly quickly (10 minutes). Sometimes I have to reboot and do an fsck to get things running again.

What would be causing this?

Things I've tried so far:

  • increased RAM from 1.5GB up to 5GB.



  • Hardware Upgrades: MotherBoard, processor, RAM (DDR4) but that didn't help (it was running a 7 year old processor, now running 7th


Gen Core I5).

  • Setup UFW firewall to ensure it wasn't being caused by bots attacking MySQL or other services.



  • In my.cnf, changed innodb_buffer_pool_size from 128MB to 500MB. didn't help but still in place



  • I've run: mysqlcheck -u root -p --auto-repair --optimize --all-databases multiple times. didn't help



  • In my.cnf, Decreased mysql max_connections from 151 to 80 and restarted mysql. didn't help



  • Decreased apache MaxRequestWorkers from 150 to 100. Didn't help. Still crashing.



  • I already had a 1GB Swap file. Left it.



  • Scoured through Apache2 logs, SysLog, any other log that seemed appropriate but haven't found anything that caught my eye.



  • Shutdown the server and tried to move the VM to another drive but it fails with File Error.



  • My latest suspicion is this is being caused by a bad block but running badblocks seemed to trigger a crash at 25% complete. During the fsck I see this: fsck critical medium error, dev sda, sector 147306432



Here is a typical mysql error log:


2017-04-20T18:43:46.958430Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 11791ms. The settings might not be optimal. (flu shed=92 and evicted=0, during the time.)

2017-04-20T18:44:11.989905Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 6822ms. The settings might not be optimal. (flushed=8 and evicted=0, during the time.)

2017-04-20T18:44:49.145162Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 5021ms. The settings might not be optimal. (flus hed=0 and evicted=0, during the time.)

2017

Solution


  • Shutdown the server and tried to move the VM to another drive but it


fails with File Error.

  • My latest suspicion is this is being caused by


a bad block but running badblocks seemed to trigger a crash at 25%
complete. During the fsck I see this: fsck critical medium error, dev
sda, sector 147306432


Don't waste your time on DB tuning & others things, your drive seem definitely guilty for me...

Try to make a dump of the DB with as much as you can, try a low level drive formatting and start a clean VM.

Context

StackExchange Database Administrators Q#171558, answer score: 4

Revisions (0)

No revisions yet.