patternsqlMinor
mysql bin logs expiry to not to fill up disk space
Viewed 0 times
spacelogsbindiskexpirymysqlnotfill
Problem
I am using 5.7.37-log MySQL Community Server. my mysqld.cnf looks below:
my expectation with expire_logs_days = 1 is that after 1 day the log will be purged. and i wont be able to see the older logs..
but when i check logs after a week, i can still see bin.000001, when the current log name is bin.000006. and I observed the old logs are getting purged when I restart mysql server and a new log is being created.
But I dont want to restart the server to get my old files deleted. they should be deleted automatically. and i fear about filling up disk space when i accumulate older logs (i dont need them for sure) and i dont want to restart sql server for this purpose.
I also had seen in some forms suggesting to run
But I dont understand the point in manually running above statement when we already set the config.
Please help me if my understanding and expectations were correct or not.
and how do i test or make sure that the logs are getting deleted automatically ? and no harm to disk space.
attaching screenshot of my generated bin logs.
Here is the screenshot after 24 hours (a day)
server_id = 11
log_bin = bin.log
log-bin-index = bin-log.index
binlog_format = row
max_binlog_size = 100M
socket = mysql.sock
expire_logs_days = 1my expectation with expire_logs_days = 1 is that after 1 day the log will be purged. and i wont be able to see the older logs..
but when i check logs after a week, i can still see bin.000001, when the current log name is bin.000006. and I observed the old logs are getting purged when I restart mysql server and a new log is being created.
But I dont want to restart the server to get my old files deleted. they should be deleted automatically. and i fear about filling up disk space when i accumulate older logs (i dont need them for sure) and i dont want to restart sql server for this purpose.
I also had seen in some forms suggesting to run
PURGE BINARY LOGS BEFORE NOW() - INTERVAL 1 DAY; But I dont understand the point in manually running above statement when we already set the config.
Please help me if my understanding and expectations were correct or not.
and how do i test or make sure that the logs are getting deleted automatically ? and no harm to disk space.
attaching screenshot of my generated bin logs.
Here is the screenshot after 24 hours (a day)
Solution
One tricky thing about binlog expiration you have to keep in mind is that it doesn't happen until the MySQL Server opens a new binlog file. So as long as it is currently writing to
So in theory, if your rate of writes to
You don't have to restart MySQL Server to force it to open a new binlog, and you don't have to wait until the binlog file fills up to its maximum size. You can run
It's also possible that you could have a surge of writes in one day, so many new binlog files are rapidly opened and filled, and yet the oldest binlog file is still less than the 1 day old, so it's not expired. Thus you can still fill up disk space before expiring older files.
What's missing from MySQL is an expiration based on total space of binlogs, instead of expiration based on time.
Percona Server is a branch of MySQL. In version 8.0, they introduced an option
bin.000006, the oldest files won't be expired. When it rolls over to bin.000007, then MySQL Server checks to see which (if any) older binlog files need to be expired.So in theory, if your rate of writes to
bin.000006 is slow enough, it takes many days to fill up and roll over to bin.000007, so your old binlog files would hang around for many days.You don't have to restart MySQL Server to force it to open a new binlog, and you don't have to wait until the binlog file fills up to its maximum size. You can run
FLUSH LOGS; any time. This will close open log files, including the binary log, and open them again. In the case of binary logs, it also opens a new file, even if the last file is not full yet.It's also possible that you could have a surge of writes in one day, so many new binlog files are rapidly opened and filled, and yet the oldest binlog file is still less than the 1 day old, so it's not expired. Thus you can still fill up disk space before expiring older files.
What's missing from MySQL is an expiration based on total space of binlogs, instead of expiration based on time.
Percona Server is a branch of MySQL. In version 8.0, they introduced an option
binlog_space_limit which sets an absolute cap on the total storage used by binlogs. See:- https://www.percona.com/blog/percona-server-for-mysql-highlights-binlog_space_limit/
- https://docs.percona.com/percona-server/8.0/flexibility/binlog_space.html
Context
StackExchange Database Administrators Q#326383, answer score: 4
Revisions (0)
No revisions yet.