MySQL relay log corrupted, how do I fix it? Tried but failed

A MySQL v5.1.61 relay got corrupted when the machine suddenly shut down. I tried to fix it but it didn't work.

— How do I fix it? Did I do something wrong?

As far as I've read, corrupted MySQL relay logs are easily fixed:

change master to master_log_file='',
                 master_log_pos=;

where Relay_Master_Log_File and Exec_Master_Log_Pos are listed by:

mysql> show slave status;

However when I did change master status ..., I got a primary key violation error. How is that possible? Is the above procedure no correct, or is e.g. some +1 missing?

(For now I've simply re-imported a --master-data mysqldump from the master to the slave, and this solved the problem. However, in the future, doing that
might not be appropriate.)

Here follows details about my particular problem:

```
mysql> show slave status \G
1. row
Slave_IO_State: Waiting for master to send event
Master_Host: the-master-host
Master_User: replication
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000021
Read_Master_Log_Pos: 33639968
Relay_Log_File: mysql-relay-bin.000271
Relay_Log_Pos: 2031587
Relay_Master_Log_File: mysql-bin.000020
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB: the_database
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 1594
Last_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network prob

Error: Last_SQL_Errno: 1594
Last_SQL_Error: Relay log read failure: Could not parse relay log event entry.

This error message means that the copy of the MySQL replication log could not be read on the slave server. A common cause of this is where a slave server has not been cleanly rebooted. In some cases it may mean that the replication log contains errors on the master server, but this is less common.

• Before doing anything backup all your databases, logs, image servers, repeat, several times, and only continue at your own risk.

** If you have MASTER-MASTER replication with additional SLAVE servers read through the notes at the bottom before continuing.

First run "show slave status \G" on the slave and note:

Relay_Master_Log_File: mysql-bin.000026
Exec_Master_Log_Pos: 1097157

First we want to make sure that the master log file is intact, so jump onto the master server and find the Relay_Master_Log_File (check /var/log/mysql) and run the following command:

mysqlbinlog mysql-bin.000026

The log will be displayed but hopefully you won't see any error messages. If you see error messages then the master logs are corrupt and you'll likely have to re-image - stop following this answer now.

Next run the same command on the slave relay log (often in /var/lib/mysql)

mysqlbinlog mysqld-relay-bin.000056

You will likely see some errors showing the corruption that has stopped replication, like this:

ERROR: Error in Log_event::read_log_event(): 'read error', data_len: 336, event_type: 2
ERROR: Could not read entry at offset 1097414: Error in log format or read error.
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
root@db:/var/lib/mysql#

If you see any errors then the log is fine on the master and only the slave's relay log is corrupt. This is good news, we can reset the slave and tell it the masters details and where to continue from. If you don't see any errors then stop reading now, you have a different problem.

If the slave relay log has errors, run the following commands to reset the slave and corrupted logs reconnect to the master, get the ok logs and start slaving again. Note that MASTER_LOG_POS is the Exec_Master_Log_Pos, and MASTER_LOG_FILE is the Relay_Master_Log_File both from the first command.

mysql> stop slave;
Query OK, 0 rows affected (0.14 sec)

mysql> reset slave;
Query OK, 0 rows affected (0.43 sec)

mysql>  CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000026', MASTER_LOG_POS=1097157;
Query OK, 0 rows affected (0.93 sec)

mysql> start slave;
Query OK, 0 rows affected (0.00 sec)

The reset slave command clears the slave's local copy of the master logs and some configuration. The "CHANGE MASTER" command reconfigures the replication. When the slave is started it gets new logs from the master server and starts replicating again.

Your replication should now be fixed and your master servers in sync.

MASTER-MASTER-SLAVE Information:

Please note the following if you are using MASTER-MASTER with additional SLAVE servers (ie you are replicating data from the 2 Master servers to additional MySQL instances):

Setup example:

MASTER1 <> MASTER2 > SLAVE1

If you have master servers MASTER1 and MASTER2, with an additional slave server SLAVE1 replicating from MASTER2, and you have had to stop, reset and restart replication on MASTER2, the slave server SLAVE1 may now try and process records from the log file from MASTER1 that it has already processed.

This may happen if the slave server had processed records from the log files from MASTER1 before they were cleared and re-downloaded on MASTER2. Depending on replication latency and data volumes this may only be a few log entries, but only you know your data or the impact this may have on integrity.

If this happens you will likely see the slave servers stop replicating due to DUPLICATE KEY errors when they insert something they've already received.

You can either recreate the slave servers to ensure you have valid data integrity, or skip the errors caused by reprocessing updates until it starts processing new log entries. You could also have a look in the MySQL log files to see what updates it contains.

If you want to skip past the errors as the slave server processes the log to get new data you can run the following command on the slave:

stop slave ; SET GLOBAL sql_slave_skip_counter = 1; start slave;

In a recent case we had to skip past 10 duplicate entry errors on a slave server before it got to new data. After skipping the 10th duplicate key error the slave continued working and was in sync with both master servers.

If I were using a SLAVE server for backups I would consider that it's data integrity was critical and re-image from a master. Otherwise you can make a judgement based on the servers application as to whether skip the errors or run checks on the data.

StackExchange Database Administrators Q#53893, answer score: 57