auto delete aws S3 backups

I need to automate deletion of aws S3 backups ... evidently if I tag the backup with one of daily, weekly, monthly, yearly then aws will delete based on my desired retention counts per those periods however I see no easy way to determine which of those tags I give to my daily backup process which is

//   Upload backup file

aws s3 cp /tmp/backup_2007020644.20200703.1248_blobs.tar  s3://foo-bar-baz-someaccount-us-east-2/backup_2007020644.20200703.1248_blobs.tar --region  us-east-2 --only-show-errors 

aws s3api put-object-tagging --bucket foo-bar-baz-someaccount-us-east-2  --key backup_2007020644.20200703.1248_blobs.tar --tagging  --region  us-east-2 TagSet=[{Key=backuptype,Value=blobs}]

//   now lets list this s3 bucket

aws s3 ls s3://foo-bar-baz-someaccount-us-east-2 --region  us-east-2 

2020-07-01 22:55:57   31904428 backup_2007010938.20200701.2233_blobs.tar
2020-07-01 22:55:43     893239 backup_2007010938.20200701.2233_mongo.tar
2020-07-02 15:30:36   34343354 backup_2007010938.20200702.1508_blobs.tar
2020-07-02 15:30:22     893676 backup_2007010938.20200702.1508_mongo.tar
2020-07-03 01:20:04   30596405 backup_2007020644.20200703.0055_blobs.tar
2020-07-03 01:19:51     893741 backup_2007020644.20200703.0055_mongo.tar
2020-07-03 12:48:44   34658003 backup_2007020644.20200703.1226_blobs.tar
2020-07-03 12:48:30     895294 backup_2007020644.20200703.1226_mongo.tar
2020-07-03 15:05:00   34657972 backup_2007020644.20200703.1248_blobs.tar
2020-07-03 15:04:46     895279 backup_2007020644.20200703.1248_mongo.tar

Alternatively I can code up my own logic to parse above listing then issue the delete command per backup file after my code keeps track of my retention policy. There must be a better way, any advice?

I think you're on the right track. Just need a tiny bit more logic on the backup process, then S3 can handle the rest.

Consider this:

• A backup is uploaded every day.

• If today is January 1st (for example), the backup is given a "yearly" tag.

• Otherwise, if today is the 1st of the month, the backup is given a "monthly" tag.

• Otherwise, if today is a Sunday, the backup is given a "weekly" tag.

• Otherwise the backup is given a "daily" tag.

Next, set up S3 Bucket Lifecycle Policies with each tag as the filter.

• Transition "yearly" backups to Glacier after 365 days (kept indefinitely)

• Delete "monthly" backups after 365 days (1 year)

• Delete "weekly" backups after 183 days (6 months)

• Delete "daily" backups after 31 days (1 month)

References:

• https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html

• https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-configuration-examples.html

StackExchange DevOps Q#11957, answer score: 4