MySQL buffer pool usage + high load

We've got a quite powerful server: HP ProLiant DL360p Gen8, 2xIntel Xeon E5-2630 2.30GHz, 128GB of RAM.

The whole data set should fit in memory:

mysql> SELECT FLOOR(SUM(DATA_LENGTH+INDEX_LENGTH)/POWER(1024,2)) 'Total Size (MB)', FLOOR(SUM(DATA_LENGTH)/POWER(1024,2)) 'Data Size (Data_length in MB)', FLOOR(SUM(INDEX_LENGTH)/POWER(1024,2)) 'Index Size (Index_length in MB)' FROM information_schema.TABLES;
+-----------------+-------------------------------+---------------------------------+
| Total Size (MB) | Data Size (Data_length in MB) | Index Size (Index_length in MB) |
+-----------------+-------------------------------+---------------------------------+
|          110559 |                         62464 |                           48095 |
+-----------------+-------------------------------+---------------------------------+
1 row in set (1.64 sec)

• innodb_buffer_pool_size is 100GB

mysql> SELECT (@@innodb_buffer_pool_size / POWER(1024,3)) AS "innodb_buffer_pool_size in GB";
+-------------------------------+
| innodb_buffer_pool_size in GB |
+-------------------------------+
|                           100 |
+-------------------------------+
1 row in set (0.00 sec)

• SHOW ENGINE INNODB STATUS

```
mysql> SHOW ENGINE INNODB STATUS\G
1. row
Type: InnoDB
Name:
Status:
=====================================
140806 21:38:13 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 8 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 19534 1_second, 19526 sleeps, 1952 10_second, 49 background, 49 flush
srv_master_thread log flush and writes: 19578
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 5952877, signal count 43935931
Mutex spin waits 179717547, rounds 1194842254, OS waits 3382912
RW-shared spins 13682200, rounds 38631505, OS waits 343596
RW-excl spins 1136695, rounds 48060134

You have a read/write ratio of 26:1 (based on 10869877 divided by 406010), which is 96% reads.

I think you might be suffering from what I call READ AHEAD FUTILITY. Sounds like I made that up. Well, you are right. I did. What do I mean ? According to the MySQL Documentation on the status variable Innodb_buffer_pool_read_ahead_evicted

The number of pages read into the InnoDB buffer pool by the read-ahead background thread that were subsequently evicted without having been accessed by queries.

Tons of SELECTs may read too many pages of data for the purposes of doing read ahead. Those pages may never be accessed and are simply evicted from the Buffer Pool. It's like blowing up a gigantic balloon without tying a knot. The air will eventually come out. This is true even for a full Buffer Pool. Pages are not allowed to lie dormant. They must be removed to make room for other moving parts of the InnoDB Architecture

For example, look at the diagram. Were you aware that up to 25% of the Buffer Pool is used as a scratch pad for non-unique index changes? It forms a conduit that migrates those changes into the system tablespace. This can crash InnoDB if there are heavy INSERTs and UPDATEs into tables that have many nonunique indexes. You would drop those indexes, do your INSERTs, and create the indexes again to circumvent this. I wrote about this before

• Jul 30, 2014 : InnoDB import performance

• May 12, 2014 : Buffer pool above 90% utilized causes MySQL to crash

Looking back at your chart, your miscellaneous is non existent. What does that indicate?

According to MySQL Documentation on Innodb_buffer_pool_pages_misc

The number of pages that are busy because they have been allocated for administrative overhead such as row locks or the adaptive hash index. This value can also be calculated as Innodb_buffer_pool_pages_total – Innodb_buffer_pool_pages_free – Innodb_buffer_pool_pages_data.

Apparently, you are not doing frequent searches for the same data. Your SELECTs must be doing

• large index range scans

• lots of index range scans

• lots of table scan

• I discussed this in MySQL status variable Handler_read_rnd_next is growing a lot

SUGGESTIONS

• You probably need to tune your queries so they do not request so much data.

• You should also avoid doing mysqldumps in the middle of the business day and this tends to load every data page and its grandmother into the Buffer Pool, only to have those pages quietly evicted. If you have to run mysqldumps, setup a MySQL Slave and o your dump from the Slave.

• You could experiment with disabling the change buffering (set innodb_change_buffering to 0).

UPDATE 2014-08-11 16:45 EDT

Please look back at the comment section. Derek Downey gave you a link to my 3 year old post. Based on the 2012 update in my post, you need to set innodb_buffer_pool_instances = 2. This will help curb mysqld's swap behavior.

From your SHOW ENGINE INNODB STATUS\G, you have too few read and write threads

--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)

You only have 4 read and 4 write I/O threads. Bump up your threads

innodb_read_io_threads = 16
innodb_write_io_threads = 16

If you had more that 2 cores, I would use 32 or 64. You can try 16 for now.

InnoDB is always at odds with the Query Cache. Please disable it.

query_cache_size = 0

You need a bigger log buffer for better InnoDB write performance

innodb_log_buffer_size = 256M

These are all the changes

[mysqld]
innodb_buffer_pool_instances = 2
innodb_log_buffer_size = 256M
innodb_read_io_threads = 16
innodb_write_io_threads = 16
query_cache_size = 0

StackExchange Database Administrators Q#73343, answer score: 4