patternsqlMinor
Tons and tons of relay logs on a master
Viewed 0 times
logsmastertonsandrelay
Problem
I have a master that has 298 relay bin files as recent as today, going back well 298 days.
There is no relay-log definitions in the .cnf
and
Reset slave clears them out, but then they just start getting regenerated.
Any idea what's causing this? How to stop it?
EDITS TO REQUESTS
General critiques of the cnf are welcome but let's keep the OP topic in mind.
```
---- cnf request
[mysqld]
character_set_server = utf8
max_connections=200
max_user_connections=160
max_connect_errors=10000
userstat_running = 1
log_warnings
slow_query_log=1
slow_query_log_file=/var/log/mysql/mysql-slow.log
long_query_time=2
innodb_file_per_table
innodb_open_files=2048
innodb_additional_mem_pool_size=1M
innodb_buffer_pool_size=512M
innodb_log_buffer_size=1M
innodb_log_file_size=128M
innodb_autoextend_increment=16
innodb_flush_method=O_DIRECT
datadir=/var/lib/mysql/
tmpdir=/var/lib/mysql_ramdisk
server-id=2
log-bin = /var/log/mysql/mysql-bin
log-bin-index = /var/log/mysql/mysql.index
key_buffer_size = 800M
preload_buffer_size = 256K
max_allowed_packet = 8M
table_cache = 512
sort_buffer_size = 8M
join_buffer_size = 8M
read_buffer_size = 2M
read_rnd_buffer_size = 2M
thread_cache_size = 32
query_cache_size = 32M
query_cache_limit = 16M
myisam_sort_buffer_size = 2000M
tmp_table_size = 64M
max_heap_table_size = 64M
---- now for the cli requests
There is no relay-log definitions in the .cnf
and
mysql> show variables like '%relay%';
+---------------------------------+----------------+
| Variable_name | Value |
+---------------------------------+----------------+
| innodb_overwrite_relay_log_info | OFF |
| max_relay_log_size | 0 |
| relay_log | |
| relay_log_index | |
| relay_log_info_file | relay-log.info |
| relay_log_purge | ON |
| relay_log_space_limit | 0 |
+---------------------------------+----------------+Reset slave clears them out, but then they just start getting regenerated.
Any idea what's causing this? How to stop it?
EDITS TO REQUESTS
General critiques of the cnf are welcome but let's keep the OP topic in mind.
```
---- cnf request
[mysqld]
character_set_server = utf8
max_connections=200
max_user_connections=160
max_connect_errors=10000
userstat_running = 1
log_warnings
slow_query_log=1
slow_query_log_file=/var/log/mysql/mysql-slow.log
long_query_time=2
innodb_file_per_table
innodb_open_files=2048
innodb_additional_mem_pool_size=1M
innodb_buffer_pool_size=512M
innodb_log_buffer_size=1M
innodb_log_file_size=128M
innodb_autoextend_increment=16
innodb_flush_method=O_DIRECT
datadir=/var/lib/mysql/
tmpdir=/var/lib/mysql_ramdisk
server-id=2
log-bin = /var/log/mysql/mysql-bin
log-bin-index = /var/log/mysql/mysql.index
key_buffer_size = 800M
preload_buffer_size = 256K
max_allowed_packet = 8M
table_cache = 512
sort_buffer_size = 8M
join_buffer_size = 8M
read_buffer_size = 2M
read_rnd_buffer_size = 2M
thread_cache_size = 32
query_cache_size = 32M
query_cache_limit = 16M
myisam_sort_buffer_size = 2000M
tmp_table_size = 64M
max_heap_table_size = 64M
---- now for the cli requests
Solution
If a Master has relay logs, then the Master must also be a Slave in the midst of some Replication topology (i.e., Master/Master, Daisy-Chained Replication)
What could cause relay logs to grow like this?
BROKEN REPLICATION
MySQL Replication is broken when the IO Thread or SQL thread dies under these SCENARIOS:
It is SITUATION #3 that's the problem. When the SQL thread dies due to a SQL error, there is no built-in mechanism in MySQL Replication that triggers the IO thread to disconnect.
RECOMMENDATION
The only decent way to control the growth of relay logs is to set the limit on it
Setting relay_log_space_limit places a cap of 4G.
When a relay log is completely processed
EPILOGUE
If the Master used to be a Slave and it does not need to be anymore, simple disable it.
If the Master is a Slave, go correct the SQL error.
I would suggest this if the SQL error is in the way:
then run
What could cause relay logs to grow like this?
BROKEN REPLICATION
MySQL Replication is broken when the IO Thread or SQL thread dies under these SCENARIOS:
- SCENARIO #1 : When the IO Thread and SQL thread are off, one of two things happended
- MySQL was started with
skip-slave-startenabled
- Someone/something ran
STOP SLAVE;
- SCENARIO #2 : When the IO thread dies
- nothing can pile up the relay logs
- The SQL thread processes all SQL commands in the relay logs or until a SQL error occurs
- SCENARIO #3 : When the SQL thread dies
- SQL error occurred processing a SQL command
- Running
SHOW SLAVE STATUS\Gshows you theLast_ErrnoandLast Error
- IO Thread continued collecting SQL commands from the Master, making relay logs grow
It is SITUATION #3 that's the problem. When the SQL thread dies due to a SQL error, there is no built-in mechanism in MySQL Replication that triggers the IO thread to disconnect.
RECOMMENDATION
The only decent way to control the growth of relay logs is to set the limit on it
[mysqld]
relay_log_space_limit=4GSetting relay_log_space_limit places a cap of 4G.
When a relay log is completely processed
- it is rotated out
- the SQL thread starts working on the next relay log
- the I/O thread starts loading SQL from the Master from the last place it left of from, as long as there is enough freespace on the disk
EPILOGUE
If the Master used to be a Slave and it does not need to be anymore, simple disable it.
mysql -e"STOP SLAVE; CHANGE MASTER TO MASTER_HOST='';"
rm -f /var/lib/mysql/master.infoIf the Master is a Slave, go correct the SQL error.
I would suggest this if the SQL error is in the way:
STOP SLAVE;
SET GLOBAL sql_slave_skip_counter = 1;
START SLAVE SQL_THREAD;then run
SHOW SLAVE STATUS\G every minute to see if the relay logs get processed and rotated.Code Snippets
[mysqld]
relay_log_space_limit=4Gmysql -e"STOP SLAVE; CHANGE MASTER TO MASTER_HOST='';"
rm -f /var/lib/mysql/master.infoSTOP SLAVE;
SET GLOBAL sql_slave_skip_counter = 1;
START SLAVE SQL_THREAD;Context
StackExchange Database Administrators Q#6983, answer score: 8
Revisions (0)
No revisions yet.