How to monitor AWS S3 cross-region replication process?

Amazon S3 has a cross-region replication which will handle copy of new/updated objects to additional region.

The problem is that solution does not provide visibility on state for replication process, for example at the moment there's no way to easily monitor missing objects on destination or any possible permission issues that can interfere with the process and can result with replication not doing it's job, as well missing feedback for replication lag.

We found several tools similar to CRR Monitor https://aws.amazon.com/answers/infrastructure-management/crr-monitor/
but we didn't have good experience with it as we manage tens of millions of objects which can cost several thousand of dollars per month for CloudWatch events, as well as additional management for services required by solution (DinamoDB, Kinesis etc.) another layer of "monitoring the monitor" required.

For now we are using daily comparison of S3 objects inventory between source and destination using AWS Athena query, which also feels like a hack and it's not optimal for problem detection as inventory generated with daily frequency.

I hope to see some feedback, suggestions, ideas or experience for alternatives. Ideal solution would be managed or "serverless" solution which will improve visibility on the process and will provide fast detection of errors.

Still no answers, and still no CRR monitoring in place on AWS, here's minimalistic solution, any feedback and suggestions are welcome. This monitor will only show which S3 objects are missing from destination, it will not provide visibility on versions of replicated objects.

Prerequisites

• Enable inventory reports on src (Source) and dst

(Destination) buckets

• Create Athena tables (src_inventory, dst_inventory) based on inventory reports for each bucket

Monitor

Run Athena query after both inventory reports generated e.g. on daily basis, can be executed from Lambda, Step Function, build server or any type of cron job.

Example of Athena query which should show number of missing objects on destination bucket at the time of generation of inventory reports:

FROM src_inventory src 
LEFT JOIN dst_inventory dst 
ON src.key = dst.key 
WHERE dst.key is NULL
AND src.dt = '2018-11-11-08-00';

The result can be reported as CloudWatch metric and monitored with CloudWatch alarm.

StackExchange DevOps Q#5115, answer score: 2